HTTP monitoring on different port

You can certainly do this. Your monitor would look something like this (replace 10.10.10.10 with your server IP address and put the correct host in the GET send string):

ltm monitor http http-test {
    defaults-from http
    destination 10.10.10.10:5100
    interval 5
    recv "200 OK"
    send "GET /f5monitor.html HTTP/1.1\\r\\nHost: host.domain.com\\r\\nConnection: Close\\r\\n"
    time-until-up 0
    timeout 16
}

If service is running on 5100 but you want to monitor 5101 because 5101 indicates the status of the service on 5100, create a health monitor and make sure to populate "Alias Service Port" as 5101, and bind the monitor to the pool.

Normally, monitor will check whatever port the pool members are running by default, but you can "force" it to check specific port by configuring the Alias Service Port.

Service is running on port 5101 but we want to monitor 5100. I have created the health check and added an alias for port 5100 then binded it to the pool. However, the pool is not coming up.

have you tried tcpdump?

 tcpdump -nni 0.0:nnn -s0 -w /var/tmp/output.pcap host x.x.x.x and host y.y.y.y and port 5100 -v

x.x.x.x is non floating self ip on server vlan
y.y.y.y is server ip