Forum Discussion

Ken_B_50116's avatar
Ken_B_50116
Icon for Cirrostratus rankCirrostratus
Mar 25, 2015
Solved

Servers using LTM as its gateway can ping but can't connect on 443

I have a situation where a group of servers that use LTM as their gateway can successfully ping and tracert to servers in a totally different subnet, but are unable to connect on port 443.

 

Specifically, servers in 10.54.13.x use gateway IP address 10.54.13.1 which is an LTM traffic group address. These servers are part of a load balanced group, no SNAT is in use for the virtual server, and this all works fine. A tracert from server 10.54.13.101 to 172.22.0.100 completes in 6 hops. Ping succeeds. I cannot, however, telnet to 172.22.0.100:443, or open https://172.22.0.100 from the 10.54.13.101 server (or any server in that subnet).

 

The successful tracert from server 10.54.13.101 to 172.22.0.100 shows the first hop as the LTM self-IP address 10.54.13.21. The second hop is 10.54.12.1 which is a core router.

 

If I do the same tracert from the LTM to 172.22.0.100, it succeeds in 1 hop. Also from the LTM, I can connect to https://172.22.0.100 without problem. So I do not believe this is a firewall issue. Because pings and tracerts are OK, I tend to think the routing is all correct.

 

https connections to 172.22.0.100 from anywhere else on the network all work fine. Only from these servers using LTM as their gateway have this problem.

 

I do not believe it's a problem with the 172.22.0.100 host because I have the same exact symptoms with other servers in the 172.22.0.x subnet.

 

On LTM, I do have the "RoutedForwarding_IP_Forwarding" iapp installed which performs IP forwarding on all sources, destinations, and VLANs (except the sync VLAN)

 

So, somehow the LTM is blocking this 443 connection from the server to the other server at 172.22.0.100. Why would tracerts and pings succeed, but https fail? Does the LTM allow ICMP by default somehow? Short of opening a case with F5 support, I thought I'd post here and see if anyone had any ideas about the cause, or what steps I can take to troubleshoot further? Probably a packet capture is needed here.

 

thanks!

 

application delivery
design
LTM
  • Jason_40733's avatar
    Jason_40733
    Mar 25, 2015

    I keep thinking asymmetric routing... but your ports match up. a tcpdump on the F5 may give you more info where you could see all the MAC addresses.

     

8 Replies

Sort By
  • Ken_B_50116's avatar
    Ken_B_50116
    Icon for Cirrostratus rankCirrostratus
    Mar 25, 2015

    I ran a tcpdump and got the following. The only thing that stands out to me is all of the incorrect checksums:

    [root@F5-LTM:Active:Changes Pending] config  tcpdump -vvv -i 0.0:nn host 172.22.0.100
tcpdump: listening on 0.0:nn, link-type EN10MB (Ethernet), capture size 96 bytes
09:08:23.873055 IP (tos 0x0, ttl 128, id 17365, offset 0, flags [DF], proto: TCP (6), length: 48) 10.54.13.101.14196 > 172.22.0.100.https: S, cksum 0x0da9 (correct), 1382886100:1382886100(0) win 64240 
09:08:23.873085 IP (tos 0x0, ttl 127, id 17365, offset 0, flags [DF], proto: TCP (6), length: 48) 10.54.13.101.14196 > 172.22.0.100.https: S, cksum 0xc437 (incorrect (-> 0x0da9), 1382886100:1382886100(0) win 64240 
09:08:23.874461 IP (tos 0x0, ttl 124, id 16901, offset 0, flags [none], proto: TCP (6), length: 48) 172.22.0.100.https > 10.54.13.101.14196: S, cksum 0x6fbf (correct), 3001936618:3001936618(0) ack 1382886101 win 64240 
09:08:23.874482 IP (tos 0x0, ttl 123, id 16901, offset 0, flags [none], proto: TCP (6), length: 48) 172.22.0.100.46740 > 10.54.13.101.14196: S, cksum 0xc437 (incorrect (-> 0xbae5), 3001936618:3001936618(0) ack 1382886101 win 64240 
09:08:23.874607 IP (tos 0x0, ttl 128, id 17366, offset 0, flags [none], proto: TCP (6), length: 40) 10.54.13.101.14196 > 172.22.0.100.46740: R, cksum 0x033e (correct), 1382886101:1382886101(0) win 0
09:08:23.874614 IP (tos 0x0, ttl 127, id 17366, offset 0, flags [none], proto: TCP (6), length: 40) 10.54.13.101.14196 > 172.22.0.100.https: R, cksum 0xc42f (incorrect (-> 0xb817), 1382886101:1382886101(0) win 0
09:08:26.851901 IP (tos 0x0, ttl 128, id 17376, offset 0, flags [DF], proto: TCP (6), length: 48) 10.54.13.101.14196 > 172.22.0.100.https: S, cksum 0x0da9 (correct), 1382886100:1382886100(0) win 64240 
09:08:26.851911 IP (tos 0x0, ttl 127, id 17376, offset 0, flags [DF], proto: TCP (6), length: 48) 10.54.13.101.14196 > 172.22.0.100.https: S, cksum 0xc437 (incorrect (-> 0x0da9), 1382886100:1382886100(0) win 64240 
09:08:26.852979 IP (tos 0x0, ttl 124, id 16902, offset 0, flags [none], proto: TCP (6), length: 48) 172.22.0.100.https > 10.54.13.101.14196: S, cksum 0xc0eb (correct), 3002702258:3002702258(0) ack 1382886101 win 64240 
09:08:26.853039 IP (tos 0x0, ttl 123, id 16902, offset 0, flags [none], proto: TCP (6), length: 48) 172.22.0.100.34221 > 10.54.13.101.14196: S, cksum 0xc437 (incorrect (-> 0x3cf9), 3002702258:3002702258(0) ack 1382886101 win 64240 
09:08:26.853180 IP (tos 0x0, ttl 128, id 17377, offset 0, flags [none], proto: TCP (6), length: 40) 10.54.13.101.14196 > 172.22.0.100.34221: R, cksum 0x3425 (correct), 1382886101:1382886101(0) win 0
09:08:26.853186 IP (tos 0x0, ttl 127, id 17377, offset 0, flags [none], proto: TCP (6), length: 40) 10.54.13.101.14196 > 172.22.0.100.https: R, cksum 0xc42f (incorrect (-> 0xb817), 1382886101:1382886101(0) win 0
  • Jason_40733's avatar
    Jason_40733
    Icon for Cirrocumulus rankCirrocumulus
    Mar 25, 2015

    You mentioned the F5 completes the traceroute in 1 hop, but the server directly behind the F5 completes in 6 hops. Does the F5 have an interface on the 172.22.0.x subnet?

     

    -- Jason

     

    • Ken_B_50116's avatar
      Ken_B_50116
      Icon for Cirrostratus rankCirrostratus
      Mar 25, 2015
      Great question. The LTM does not have an interface on the 172.22.0.x subnet.
  • Jason_40733's avatar
    Jason_40733
    Icon for Cirrocumulus rankCirrocumulus
    Mar 25, 2015

    I keep thinking asymmetric routing... but your ports match up. a tcpdump on the F5 may give you more info where you could see all the MAC addresses.

     

  • DEJ's avatar
    DEJ
    Icon for Nimbostratus rankNimbostratus
    Mar 25, 2015

    Things to check: - Software firewall on server - Packet Filters on F5

     

  • gsharri's avatar
    gsharri
    Icon for Altostratus rankAltostratus
    Mar 25, 2015

    I think your tcpdump is showing some problems. I see two syn packets with the same seq number initiated from the 10 host: 10.54.13.101.14196 > 172.22.0.100.https : S the first one with a correct cksum the second is incorrect and the TTL has been decremented compared to the first.

     

    Then I see the 172 host respond with a syn-ack 172.22.0.100.https > 10.54.13.101.14196: S,...ack. But then the 172 host initiates another connection to the 10 host 172.22.0.100.46740 > 10.54.13.101.14196: S and the cksum is incorect.

     

    Then the 10. host sends tcp resets to 172.22.0.100.https and 172.22.0.100.46740. Again one cksum is correct the other incorrect.

     

    There is something strange happening here, not sure what it is...

     

  • Ken_B_50116's avatar
    Ken_B_50116
    Icon for Cirrostratus rankCirrostratus
    Mar 26, 2015

    Thanks everyone for the replies. Based on information from the firewall showing incomplete sessions but no ACL/policy issues, I think this a routing or asymmetric routing problem. This was mentioned earlier. How can I check to determine this? Get simultaneous packet captures on the LTM and the servers on 172.22.0.100 and 10.54.13.101 while trying to open https://172.22.0.100 from 10.54.13.101 ?

     

  • Ken_B_50116's avatar
    Ken_B_50116
    Icon for Cirrostratus rankCirrostratus
    Jan 25, 2016

    The problem here was a static route that was pointing to a different gateway than what was acceptable. I don't know how why the static route was instituted, but once the static route was deleted, the traffic took the default route and the problem was resolved.