Forum Discussion

Jeff_Siegel_152's avatar
Jeff_Siegel_152
Icon for Nimbostratus rankNimbostratus
Jun 03, 2014

Timeout tracking

We had a customer saying they were getting occasional timeout issues when hitting our internally hosted web service, but we didn't see anything in the logs on the Linux box inside our firewall.

 

Since the customer does an SSL connection through our F5 to get to that box, is there a way we could track this down at the network or F5 level? Was our web service not responding quickly, was our F5 not responding quickly, was it even getting to our F5 or any part of our network at all, or was this perhaps an issue with the customer's network?

 

For example if the customer told us we end-to-end responded at 10, timed out at 10:01 and responded again at 10:05, can we see that on our side? Did the timeout-packet even get to inside our firewall?

 

Thanks,

 

Jeff

 

6 Replies

  • OK, so, let's put this into perspective. You'd like complete end-to-end visibility of every packet or connection (or lack thereof) between you and your client, at every hop under your control, between your server and the 'client'. Forgive my negativity but people have built careers and companies around such things.

     

    On a more positive note, what can you do?

     

    • You could perform a tcpdump on the F5 over a suitable period of time, dumping it to a file and then use Wireshark or a similar commercial application to perform some analysis (manual or otherwise).
    • Check relevant switch ports and network interfaces for errors and drops.
    • Take a close look at the Traffic Summary statistics (of course, the downside is these stats are global, not VS specific)
    • Take a close look at the Profiles Summary > TCP statistics (of course, the downside is these stats are global, not VS specific)
    • Take a close look at the Network statistics (of course, the downside is these stats are global, not VS specific)
    • Enable TCP reset logging and monitor statistics accordingly

    There are also other things you could consider that may help you track things down (or simply improve things in general);

     

    • Can you monitor server connection/CPU/memory usage?
    • Consider modifying TCP profile settings based on the traffic type
    • Consider increasing default idle-timeout settings
    • You could perhaps use an iRule to log relevant VS specific statistics?
    • You could use increased SNMP monitoring and monitor accordingly
    • You could setup sFlow and monitor accordingly

    If you can provide some more info on the VS type and setup, traffic flow and behaviour (constant, small flows, large bursts etc.) I may be able to offer further advice.

     

  • Thanks for your response. I realize the problem is a hard one, and I’m fishing for ideas on how to tackle in the future.

     

    More details:

     

    The sending App runs on SAP’s WebMethods. The receiving App runs on Oracle’s WebLogic/Fusion but in the end they are just web servers sending small XML messages wrapped in SOAP, with relatively low volume message burst of a few messages a minute. Our F5 handles many different customers making it hard to isolate data, however no one else complained during the timeout issues, so while we think this was isolated to that one service we aren’t sure. Does that help?

     

    Before we increase default idea-timeout setting, or some such thing, is there something you described above that indicates timeouts are really happening on my network at all, and more specifically with the service in question?

     

    Thanks,

     

    Jeff

     

    • What_Lies_Bene1's avatar
      What_Lies_Bene1
      Icon for Cirrostratus rankCirrostratus
      Understood. There's generally a few changes to the TCP profile that help if its a standard VS using SSL termination. Not so much if its SSL passthrough. Is it a standard VS? What TCP profiles are you using client and server-side? Are you terminating the SSL on the F5? Standard timeouts? Is the real server dedicated to this app? Just one real server? Is the clients-side network a WAN? Low bandwidth? High latency? Reliable? Is it a Linux server? If so, what Kernel version?
    • Jeff_Siegel_152's avatar
      Jeff_Siegel_152
      Icon for Nimbostratus rankNimbostratus
      Standard HTTPS VS with SSL offload on the F5. The inbound network is the Internet, internally the network is a LAN. It should be high bandwidth and pretty reliable. We have just one real server. Client TCP timeout is the default, 5 minutes. Latency should not be more than a few seconds. Does any way to track this come to mind? Thanks, Jeff
  • We use Hyperspin, a service that monitors specific websites, from several places around the globe. This gives insight into whether the slowness is part of a greater Internet issue. With Hyperspin, you can specify IP addresses, which is helpful if you host at multiple geographic locations. Also, if your customer complaints are up-to-the-minute, you can use tools like www.internetpulse.net, to see where there may be trouble with the major carrier peering points.