Forum Discussion

smp_86112's avatar
smp_86112
Icon for Cirrostratus rankCirrostratus
Jan 15, 2010

Persistent Connection Forwarded to Wrong Pool Member

v9.3.1HF8

 

(I haven't really studied this behavior closely before, so please go light on me.)

 

 

It seems we have identified an issue where HTTP clients with a BigIP Cookie are being forwarded to the wrong pool member. Upon review of the logs, we see the same BigIP cookie header value on both pool members. The clients are behind a proxy server, so we suspect that multiple client sessions are being pipelined into a single TCP session between the LTM and the proxy. The VIP is pretty vanilla, and so is the persistence profile which I have included below for reference.

 

 

How do you troubleshoot something like this? The GUI doesn't show the persistence records, so I can't confirm from that perspective. Is there a different type of persistence I could use that would prevent this problem from happening at all? What other mitigation strategies are out there?

 

 

virtual app {

 

destination :80

 

ip protocol tcp

 

profile default_http tcp

 

persist cookie

 

pool app

 

 

}

 

 

profile persist cookie {

 

mode cookie

 

cookie mode insert

 

cookie expiration 0d

 

cookie hash offset 0

 

cookie hash length 0

 

cookie name none

 

rule none

 

mirror disable

 

timeout 0

 

}

 

 

pool app {

 

lb method member predictive

 

min active members 1

 

monitor all dtac.deere.com

 

member :

 

member :}

 

config
design

3 Replies

Sort By
  • Hamish's avatar
    Hamish
    Icon for Cirrocumulus rankCirrocumulus
    Jan 17, 2010
    For cookie persistence, there is no information stored on the F5 itself for the persistence (At least none that I've found, and F5 tell me this as well). The information for persistence is stored solely within the cookie. Because it's un-encrypted, you can decode the cookie and see what the value says.

     

    If the poolmember listed in the cookie isn't valid (i.e. isn't a valid and up poolmember), then a new poolmember will be chosen from the list of available poolmembers.

     

     

    How much traffic are we talking about here? If there too much to do a tcpdump out to a file (i.e.

     

     

    tcpdump -i 0.0 -n -p -s 0 -w /var/tmp/somefile.dump host and host and port 80

     

     

    )

     

     

    to view the traffic both between the clients & VS and between the F5 and the poolmembers?

     

     

    Then you can verify whether things are working as designed or not. (i.e. use wireshark or similar to look at the connections. Is the cookie being set by the F5 the same as the cookie being presented back to the F5 by the client? Is the proxy inserting an XForwarded-For header? Does that IP change in the header on the same connection?

     

     

    H

     

    H
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Jan 18, 2010
    Those are good troubleshooting steps, but I think this is an expected problem. The issue occurs when multiple clients access the VIP over the same TCP connection with different persistence records. You can add a OneConnect profile to fix this:

     

     

     

    http://devcentral.f5.com/wiki/default.aspx/AdvDesignConfig/oneconnect.html

     

     

    OneConnect with HTTP

     

     

    Without OneConnect enabled, persistence data is examined only in the first request of a Keep-Alive connection, so if multiple requests are sent on the same clientside Keep-Alive connection, LTM will persist them all to the same destination as the first unless a OneConnect profile is applied (even if logic contained in an iRule dictates otherwise).

     

     

     

     

    If the connections from LTM to the pool use SNAT, you can use a /0 source mask on the OC profile. Else, you should use a /32 mask to ensure the server logs show the correct client IP address.

     

     

    Aaron
  • smp_86112's avatar
    smp_86112
    Icon for Cirrostratus rankCirrostratus
    Jan 19, 2010
    I think we have verified the behavior with tcpdump and a review of the web server logs - what we were lacking is an explanation of what we were seeing. Ultimately I found SOL7964 which seems to be describing the symptoms, and also lays out the two potential F5 options we have. Thanks for the feedback.