Forum Discussion

cheezus_115419's avatar
cheezus_115419
Icon for Nimbostratus rankNimbostratus
May 07, 2013

Slow first outbound connection. Fast subsequent connections.

The problem is that the first outbound connection from a server to a host is very slow. The second and subsequent connections are very fast. We are testing wget 100m test files over http from various cdn providers.

 

For example if we download a large file to a server behind the viprions from linode. The first connection will download at approximately 100K/s. The second and subsequent connections will download at 5M/s (or sometimes much more).

 

Our test server is on a local subnet and vlan 10.10.220.0/24. The viprions have a floating ip address of 10.10.220.1. The router which both the viprions and the server physically connect has the vlan svi ip of 10.10.220.4. The server's gateway is the router of 10.10.220.4 We have policy based routing on this vlan that matches any IP traffic not destined for 10.10.220.0/24 and forwards it to the viprions floating ip of 10.10.220.1. The goal is to keep internal traffic off the viprions and only allow external traffic to go through them.

 

But this is when the first slow, second fast connection problems come into play. When the server's gateway is set to the floating ip on the viprions of 10.10.220.1 and the policy based routing is removed from the vlan, the first and subsequent connections are always fast. We can repeat this over and over, with any external host to download from.

 

I've done some tcpdumps and in the slow initial connection dump I see "continuation or non-http traffic" from the external host followed by test server sending RST. I'm not sure why that would be happening. But the viprion floating ip can directly reach the test server so there's probably some asymmetric routing where server -> router -> viprions -> external host. Then external host -> viprions -> server. Why it would only be slow on that initial connection is puzzling me...

 

I'm probably missing something simple and was wondering if it's jumping out at anyone reading this. Thanks for reading :)

 

config
design

14 Replies

Sort By
  • cheezus_115419's avatar
    cheezus_115419
    Icon for Nimbostratus rankNimbostratus
    May 07, 2013
    Ok so it appears to have been the "auto last hop option". Which was default and enabled. After disabling everything is fine.

     

     

    Another thing to note is that forwarding ip virtual server bypasses the tcpdump. That's why my tcpdumps looked way too small and threw me off the trail by leading me to think traffic was routing around the f5's.
  • cheezus_115419's avatar
    cheezus_115419
    Icon for Nimbostratus rankNimbostratus
    May 14, 2013
    Just wanted to update this thread that auto last hop does need to be disabled globally for me. Also CMP is fine. When checking the virtuals they show cmp is still enabled :)

     

     

    I'm on 11.3 HF5.
  • cheezus_115419's avatar
    cheezus_115419
    Icon for Nimbostratus rankNimbostratus
    May 14, 2013
    Not sure since we're still in development and don't have any traffic flowing through. We're also a webhosting provider so our traffic would be widely disbursed anyway.

     

     

    But this command shows cmp enable when run on all our virtuals where auto last hop is disabled:

     

     

    root@(2400-01)(cfg-sync Changes Pending)(/S1-green-P:Active)(/Common)(tmos) show ltm virtual VS_Outbound detail

     

     

    ------------------------------------------------------------------

     

    Ltm::Virtual Server: VS_Outbound

     

    ------------------------------------------------------------------

     

    Status

     

    Availability : unknown

     

    State : enabled

     

    Reason : The children pool member(s) either don't have service checking enabled, or service check results are not available yet

     

    CMP : enabled

     

    CMP Mode : all-cpus

     

  • What_Lies_Bene1's avatar
    What_Lies_Bene1
    Icon for Cirrostratus rankCirrostratus
    May 15, 2013
    I don't think the VS configs will be touched despite the fact CMP is disabled. Perhaps someone else can confirm by belief that disabling ALH globally will disable CMP globally?
  • cheezus_115419's avatar
    cheezus_115419
    Icon for Nimbostratus rankNimbostratus
    May 15, 2013
    I'm working with f5's consulting all week and they aren't 100% sure. They've searched and ping others but no response yet. From that they can see it doesn't appear to be disabled. Do you know the command to check the global status of cmp?

     

     

    As for why it helps us, no one's really sure about that either. The cisco router is the default gateway for the server. The f5 has local floating ip's on the same subnet as the server. f5 and server can talk directly. Cisco vlan (server gateway) has policy if non local traffic forward to f5. So Cisco sends redirect and server communicates directly with f5, f5 communicates directly with server. It would seem that in both cases the last hop would be the server and not actually Cisco router.
  • What_Lies_Bene1's avatar
    What_Lies_Bene1
    Icon for Cirrostratus rankCirrostratus
    May 16, 2013
    OK, I was wrong, it's PVA/ePVA processing that's disabled, not CMP, my apologies. Still, this can have quite an impact on performance.

     

     

    As noted in this (but no specific mention of ePVA in VIPRION): http://support.f5.com/kb/en-us/solutions/public/9000/400/sol9487.html

     

     

    "Note: The auto_lasthop setting is a global setting, and disabling it may affect system performance. Without it, the BIG-IP system must instead ARP for MAC addresses and perform routing table lookups rather than using last hop data from the connection table. In addition, Packet Velocity Accelerator (PVA) will also be disabled globally when the auto_lasthop setting is disabled on platforms with a PVA."
  • What_Lies_Bene1's avatar
    What_Lies_Bene1
    Icon for Cirrostratus rankCirrostratus
    May 16, 2013
    Also, your general design approach seems a bit back to front and over engineered (if you don't mind me saying so). Why not have the default pointing to the F5 and statics for your internal networks. No need for the PBR then.
  • cheezus_115419's avatar
    cheezus_115419
    Icon for Nimbostratus rankNimbostratus
    May 16, 2013
    Thanks for the info! I'll pass this along with our consultant today and see what he says.

     

     

    The design is because we have ~1000 servers, and rapidly expanding, behind the viprions (web hosting). We don't want to push backup server traffic through the viprions. Also there's a lot of remote logging / perf monitoring / etc.. traffic that we don't want to through them either. If they were the default then all this traffic would flow through them and we really don't want that happening.