Forum Discussion

Jon_43169's avatar
Jon_43169
Icon for Nimbostratus rankNimbostratus
Jun 05, 2014

Logs say no route to host. Routing table and tcptrraceroute say otherwise. Reset packets being sent as a result.. halp

I've not seen this before and am honestly stumped.

From the logs:

Thu Jun 5 12:00:34 CDT 2014     err     F5  tmm[9502]   01230140    RST sent from 10.180.48.237:80 to 10.180.48.3:58937, [0x174f304:2855] No route to host
Thu Jun 5 12:02:08 CDT 2014     err     F5  tmm[9502]   01230140    RST sent from 10.180.48.237:80 to 10.180.48.3:59023, [0x174f304:2855] No route to host
Thu Jun 5 12:33:39 CDT 2014     err     F5  tmm1[9502]  01230140    RST sent from 10.180.48.237:80 to 10.180.48.3:62140, [0x174f304:2855] No route to host
Thu Jun 5 12:34:36 CDT 2014     err     F5  tmm2[9503]  01230140    RST sent from 10.180.48.237:80 to 10.180.48.3:62230, [0x174f304:2855] No route to host

From the cli:

[root@F5:Active:Changes Pending] config  tcptraceroute 10.180.62.121 -p 80
traceroute to 10.180.62.121 (10.180.62.121), 30 hops max, 40 byte packets
 1   (192.168.193.2)  0.669 ms  0.883 ms  0.880 ms
 2   (10.180.62.121)  2.888 ms  2.867 ms  2.853 ms

[root@F5:Active:Changes Pending] config  telnet 10.180.62.121 80
Trying 10.180.62.121...
Connected to 10.180.62.121.
Escape character is '^]'.
GET /r\n\


...
[output truncated]

Packet capture:

tcpdump -nni any host 10.180.48.237 or host 10.180.62.121 or host 10.180.62.122
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type EN10MB (Ethernet), capture size 96 bytes
13:18:41.546914 IP 10.180.48.3.49224 > 10.180.48.237.80: S 2908648920:2908648920(0) win 65535 
13:18:41.546950 IP 10.180.48.237.80 > 10.180.48.3.49224: S 2185775265:2185775265(0) ack 2908648921 win 4140 
13:18:41.559960 IP 10.180.48.3.49224 > 10.180.48.237.80: . ack 1 win 53248
13:18:41.560068 IP 10.180.48.237.80 > 10.180.48.3.49224: R 1:1(0) ack 1 win 4140

From the client testing:

a041235@SATD-L-PB01KVKP ~
$ curl -ivvv http://10.180.48.237
* STATE: INIT => CONNECT handle 0x60002e1c0; line 1026 (connection -5000)
* Rebuilt URL to: http://10.180.48.237/
* Hostname was NOT found in DNS cache
*   Trying 10.180.48.237...
* STATE: CONNECT => WAITCONNECT handle 0x60002e1c0; line 1073 (connection 0)
* Connected to 10.180.48.237 (10.180.48.237) port 80 (0)
* STATE: WAITCONNECT => DO handle 0x60002e1c0; line 1192 (connection 0)
> GET / HTTP/1.1
> User-Agent: curl/7.36.0
> Host: 10.180.48.237
> Accept: */*
>
* STATE: DO => DO_DONE handle 0x60002e1c0; line 1278 (connection 0)
* STATE: DO_DONE => WAITPERFORM handle 0x60002e1c0; line 1404 (connection 0)
* STATE: WAITPERFORM => PERFORM handle 0x60002e1c0; line 1417 (connection 0)
* Recv failure: Connection reset by peer
* Closing connection 0
* The cache now contains 0 members
* Expire cleared
curl: (56) Recv failure: Connection reset by peer

And finally config bits:

root@(F5)(cfg-sync Changes Pending)(Active)(/NONPROD)(tmos) list ltm virtual crmqa
ltm virtual crmqa {
    destination 10.180.48.237:http
    ip-protocol tcp
    mask 255.255.255.255
    partition NONPROD
    pool crmqa
    profiles {
        /Common/tcp { }
    }
    source 0.0.0.0/0
    vlans-disabled
}
root@(F5)(cfg-sync Changes Pending)(Active)(/NONPROD)(tmos) list ltm pool crmqa
ltm pool crmqa {
    members {
        SA1W-PIVWEB-Q1:http {
            address 10.180.62.121
            session monitor-enabled
            state up
        }
        SA1W-PIVWEB-Q2:http {
            address 10.180.62.122
            session monitor-enabled
            state up
        }
    }
    monitor MON-HTTP
    partition NONPROD
}
root@(F5)(cfg-sync Changes Pending)(Active)(/NONPROD)(tmos) list ltm monitor http MON-HTTP       
ltm monitor http MON-HTTP {
    defaults-from /Common/http                
    destination *:*                
    interval 5
    partition NONPROD
    send "GET /\\r\\n"
    time-until-up 0
    timeout 16
}

When I try to hit the virtual server, there's a three way handshake followed by an immediate reset. The F5 is reporting there's no route to host, which I presume its referencing it's pool members in that statement? I dunno.. any help would be appreciated.

5 Replies

  • root@(F5)(cfg-sync Changes Pending)(Active)(/NONPROD)(tmos) show /net route

    -------------------------------------------------------------------------------
    Net::Routes
    Name                Destination         Type       NextHop            Origin
    -------------------------------------------------------------------------------
    fe80::/64           fe80::/64           interface  tmm0               connected
    ff02::/64           ff02::/64           interface  tmm0               connected
    fe80::%vlan4095/64  fe80::%vlan4095/64  interface  tmm_bp             connected
    ff02:fff::/64       ff02:fff::/64       interface  tmm_bp             connected
    fe80::%vlan52/64    fe80::%vlan52/64    interface  /Common/NONPROD    connected
    ff02:34::/64        ff02:34::/64        interface  /Common/NONPROD    connected
    fe80::%vlan12/64    fe80::%vlan12/64    interface  /Common/PCI        connected
    ff02:c::/64         ff02:c::/64         interface  /Common/PCI        connected
    fe80::%vlan4094/64  fe80::%vlan4094/64  interface  /Common/HA         connected
    ff02:ffe::/64       ff02:ffe::/64       interface  /Common/HA         connected
    fe80::%vlan32/64    fe80::%vlan32/64    interface  /Common/PROD       connected
    ff02:20::/64        ff02:20::/64        interface  /Common/PROD       connected
    fe80::%vlan48/64    fe80::%vlan48/64    interface  /Common/nonprod48  connected
    ff02:30::/64        ff02:30::/64        interface  /Common/nonprod48  connected
    fe80::%vlan28/64    fe80::%vlan28/64    interface  /PROD/Prod_vip28   connected
    ff02:1c::/64        ff02:1c::/64        interface  /PROD/Prod_vip28   connected
    127.1.1.0/24        127.1.1.0/24        interface  tmm0               connected
    127.20.0.0/16       127.20.0.0/16       interface  tmm_bp             connected
    10.170.32.0/22      10.170.32.0/22      interface  /Common/PROD       connected
    10.160.12.0/22      10.160.12.0/22      interface  /Common/PCI        connected
    10.180.52.0/22      10.180.52.0/22      interface  /Common/NONPROD    connected
    192.168.25.0/30     192.168.25.0/30     interface  /Common/HA         connected
    10.170.28.0/22      10.170.28.0/22      interface  /PROD/Prod_vip28   connected
    

    [root@F5:Active:Changes Pending] config route -n

    Kernel IP routing table
    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
    192.168.192.53  192.168.193.1   255.255.255.255 UGH   9      0        0 eth0
    192.168.192.52  192.168.193.1   255.255.255.255 UGH   9      0        0 eth0
    192.168.25.0    0.0.0.0         255.255.255.252 U     0      0        0 HA
    127.1.1.0       0.0.0.0         255.255.255.0   U     0      0        0 tmm0
    192.168.193.0   0.0.0.0         255.255.255.0   U     0      0        0 eth0
    127.3.0.0       0.0.0.0         255.255.255.0   U     0      0        0 mgmt_bp
    127.2.0.0       0.0.0.0         255.255.255.0   U     0      0        0 eth0.1
    10.101.120.0    192.168.193.1   255.255.255.0   UG    9      0        0 eth0
    10.170.32.0     0.0.0.0         255.255.252.0   U     0      0        0 PROD
    10.180.52.0     0.0.0.0         255.255.252.0   U     0      0        0 NONPROD
    10.170.28.0     0.0.0.0         255.255.252.0   U     0      0        0 Prod_vip28
    10.160.12.0     0.0.0.0         255.255.252.0   U     0      0        0 PCI
    0.0.0.0         192.168.193.1   0.0.0.0         UG    9      0        0 eth0
    

    So it's in the kernel routing table, but not LTMs. I suppose a static route should resolve that?

    If that's the case, I'm curious why the pool is passing the health monitor:

    [root@F5:Active:Changes Pending] config  tmsh show ltm pool /NONPROD/crmqa members | egrep 'Ltm::Pool|Avail|State|Monit'
    Ltm::Pool: /NONPROD/crmqa
      Availability : available
      State        : enabled
      Monitor      : /NONPROD/MON-HTTP
      | Ltm::Pool Member: /NONPROD/SA1W-PIVWEB-Q1:80
      |   Availability   : available
      |   State          : enabled
      |   Monitor        : /NONPROD/MON-HTTP (pool monitor)
      |   Monitor Status : up
      | Ltm::Pool Member: /NONPROD/SA1W-PIVWEB-Q2:80
      |   Availability   : available
      |   State          : enabled
      |   Monitor        : /NONPROD/MON-HTTP (pool monitor)
      |   Monitor Status : up
    
  • can you add tmm route for 10.180.62.121 and 10.180.62.122?

     

    i think tcptraceroute, telnet and health monitor work because it goes through management route. application traffic has to use tmm route.

     

  • I appreciate the help everyone, got it sorted. Hooked up another interface and trunked a VLAN up for the .62 net. Problem solved.

     

  • Hi,

     

    We had the same issue.

     

    Adding a static route to to the LTM routing table actually solved this issue for us.

     

    Regards,