Learn F5 Technologies, Get Answers & Share Community Solutions Join DevCentral

Filter by:
  • Solution
  • Technology
Answers

One-Way Persistence records problem

Hello All,

tl;dr - I have an iRule that is allowing HTTP_Request to insert invalid persistence records and unnecessary updates to existing persistence records and I would like to stop that by using 'persist lookup uie' instead of 'persist uie', but I think I'm doing it wrong :( Jump to the iRules to win a free headache, or not :D

Admittedly I'm still getting to grips with some of the more funky things you can do with iRules and my current challenge is that a customer of our rolled out a new client app version which has caused the VS that supports the app to chew up around 46% of the system resources where it used to use around 8%.

Thankfully everything is functioning just fine however if someone so much as farts the system could run out of resources and failover which would be a massive problem. Aside from the fact that we know the client side app has changed behavior in some way the symptoms we see on the F5, aside from the CPU load, is HUGE amounts of persistence record updates across the HA Sync interface, in the region of 5Gb/20 seconds.

What I am attempting to achieve is three fold, one, to reduce the number of updates to the persistence table, two, ditch the invalid JSESSIONIDs that are being inserted and finally, try save some cpu time.

The current universal persistence iRule that is currently in use allows, to my understanding at least, the HTTP Request to update or add persistence records by using the usual 'persist uie' command:

# iRule Name: JSESSIONID_persist
when HTTP_REQUEST {
   # Check if there is a JSESSIONID cookie
   if { [HTTP::cookie "JSESSIONID"] ne "" }{
      # Persist off of the cookie value with a timeout of 5 minutes (300 seconds)
      persist uie [string tolower [HTTP::cookie "JSESSIONID"]] 300
   }
}
when HTTP_RESPONSE {
   # Check if there is a jsessionid cookie in the response
   if { [HTTP::cookie "JSESSIONID"] ne "" }{
      # Persist off of the cookie value with a timeout of 5 minutes (300 seconds)
      persist add uie [string tolower [HTTP::cookie "JSESSIONID"]] 300
   }
}

The design doc for the solution says that only the app servers can supply a valid JSessionID and so this action by the user is not necessary and has two problems, 1. the HTTP_Requests are able to insert or update stale JSessionIDs in the persistence table and 2. the overhead caused by the lookup and add combo that 'persist uie' carries out as well as the addition and tracking of these stale records is completely unnecessary.

So my intention is to only allow lookups to be run by the HTTP_Requests which I was hoping would result in a successful direction of the connection to the correct pool member that had created the matching persistence record:

# iRule Name: JSESSIONID_persist_v2
when HTTP_REQUEST {
	# Check if there is a JSESSIONID cookie in the request header.
	if { [HTTP::cookie "JSESSIONID"] ne "" }{
		# Run a lookup for any existing records, return relevant values for any matches 
		# otherwise ignore and follow load balancing method to select destination server.
		persist lookup uie [string tolower [HTTP::cookie "JSESSIONID"]]
	}
}
when HTTP_RESPONSE {
	# Check if there is a jsessionid cookie in the response.
	if { [HTTP::cookie "JSESSIONID"] ne "" }{
		# Persist off of the cookie value with a timeout of 5m15s to ensure that 
		# the F5's persistence entry will not expire before the server token.
		persist add uie [string tolower [HTTP::cookie "JSESSIONID"]] 315
	}
}

This worked great in load and functions tests that we ran in the dev environment where the VS had two pools, but failed on the production test VS where there are 7 pool members and it looks like the persistence is not being followed.

I have a sneaky suspicion that my basic understanding of the 'persist lookup' function is wrong and that it does not make load balancing decisions for the connection based on the returned info which would mean that it was complete chance that the load and function tests worked perfectly.

If this is the case then I think I need to assign the returned node info to a variable and then use that to direct the connection to the pool member using 'node $returned_lookup' or some such. Is this right or is there a better way to do this?

The main thing is that this iRule has to be low cost because it has to handle a large volume of operations, I was even looking to see if I could use 'switch' to take over from the 'if' statement, but I got a bit stuck with the negative comparison.

What do you guys and gals reckon?

Regards, Al

0
Rate this Question

Answers to this Question

placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

persist lookup does not make load balancing decisions, it just looks to see if there is a persist record that matches what you are using for a key. In fact the none of the persist options make any load balancing decisions, they just lookup, add, delete a "record" of persistence.

I'm fairly sure that switch is no more efficient than a single if.

One possible issue is that you set the lifetime of the persistence record to 315 seconds. So after 315 seconds of no activity from the user, the persistence record will be deleted and the next time they come in the request will be re-load balanced.

Is it possible that there are instances where there is more than 315 seconds between requests from the user?

Is it possible that there are instance where it could take longer than 315 seconds for a response?

On responses servers normally only add cookies when the cookie does not appear on the request. Meaning that the cookie on the response will only appear once, the first time the server sees a request. If a response takes longer than 315 seconds, then the F5 will delete the persistence record, the response will NOT have the JESSIONID cookie, and so the persist command will not be executed.

0
Comments on this Answer
Comment made 04-Sep-2015 by AlanTen 75
Thanks giltjr, you are right and I have seen as much with the lookups so I have set the output to variables in the following way: <pre>set pnode [ persist lookup uie [string tolower [HTTP::cookie &quot;JSESSIONID&quot;]] node ] set pport [ persist lookup uie [string tolower [HTTP::cookie &quot;JSESSIONID&quot;]] port ] if { $pnode != &quot;&quot; }{ node $pnode $pport } </pre> As for the lifetime, the app servers in the background restrict the JSessionIDs that they issue to 300 seconds so the JSessionID will expire on the server before the F5 does. If the client takes too long they will just have to re-auth to get a new session ID. We just finished testing the iRule and everything seems to be working, but I suspect that it will use up a couple more cycles than the original, but we'll only know that once the dev's run the soak test. The current iteration of the script is: <pre># iRule Name: JSESSIONID_Persist_v4 # Last Mod: AT 2015/09/04 when HTTP_REQUEST { # Check if there is a JSESSIONID cookie in the request header. if { [HTTP::cookie &quot;JSESSIONID&quot;] ne &quot;&quot; }{ # Run a lookup for any existing records, return relevant values for any matches # otherwise ignore and follow load balancing method to select destination server. set pnode [ persist lookup uie [string tolower [HTTP::cookie &quot;JSESSIONID&quot;]] node ] set pport [ persist lookup uie [string tolower [HTTP::cookie &quot;JSESSIONID&quot;]] port ] if { $pnode != &quot;&quot; }{ node $pnode $pport } } } when HTTP_RESPONSE { # Check if there is a jsessionid cookie in the response. if { [HTTP::cookie &quot;JSESSIONID&quot;] ne &quot;&quot; }{ # Persist off of the cookie value with a timeout of 6 minutes to ensure that # the F5's persistence entry will not expire before the server token. persist add uie [string tolower [HTTP::cookie &quot;JSESSIONID&quot;]] 315 } } </pre>
0
Comment made 07-Sep-2015 by Opher Shachar 144
Anyways (referring to our other thread), did this iRule work out for you as expected?
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Had to do some more research because I started to remember some "weirdness" when I wrote my first iRule using persist uie dealing with timeout.

When you do a persist add with a timeout the persist record will be deleted after that number of seconds unless the timeout is "refreshed."

The timeout will be refreshed only if you do another "persist add" or just a plain "persists". A "persist lookup" will not refresh the timeout. Since the cookie will NOT exist on any response other than the 1st one, you have to refresh the persist record in HTTP_REQUEST. This is why the sample iRule use just plain "persist", this will look up to see if a record exists and if it does it resets the timeout to what you have coded.

So in your "new" iRule, 315 seconds after the 1st request comes in the persist record will be deleted, because you never refresh the timeout. You need to have just plain "persist" in HTTP_REQUEST, if the cookie JSESSIONID exists.

However my other notes about the time out still applies. If there is more than 315 seconds between users requests, then the persist record will be deleted and then next request will be re-load balanced.

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

I guess or posts crossed. If the server kills the JESSIONID after 300 seconds, then the original iRule was messing things up because it was refreshing the persist record on every request that came in with 300 seconds of the last request.

Doing what you changed to last might be the only way to do this.

0
Comments on this Answer
Comment made 07-Sep-2015 by AlanTen 75
The old script was definitely messing things up and wasting resources because there are a couple of tasks that the client applications are able to do that are 'sessionless' or don't need valid sessions to carry out the task and what we were finding is that if these 'sessionless' requests happened to have a JSessionID cookies, usually stale or junk, they would insert cookies into the persistence tables so this and the the fact that the clients aren't the ones dictating the cookie lifetime we definitely don't need the requests to do anything other than lookups. So yeah, it does seem that this is the only way, at least so far. I'll be thinking a little more on the script to see if I can get things running a little quicker since we successfully tested a version of the script on the prod test systems on Friday afternoon.
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Are you by any chance running Oracle Weblogic as your backend servers?
If so you can reduce the session table to 7 records flat.

0
Comments on this Answer
Comment made 07-Sep-2015 by AlanTen 75
Hello Opher, I have no idea what is in the back end because I just support the F5's in the front end, but I do know there are some Orcale DB servers somewhere in the back. Unfortunately each client needs a unique session ID to ensure that their requests are processed correctly.
0
Comment made 07-Sep-2015 by Opher Shachar 144
Oracle Weblogic employs an optimization to the JSESSIONID cookie it issues whereby it appends its JVMID (after an exclamation mark '!'). One can (that's what we do and is the intent of this feature) persist on the NLB based on the JVMID alone. That means one persistence record per Application Server instance.
0
Comment made 07-Sep-2015 by AlanTen 75
Interesting, I will have to look into this and discuss it wilt the client's dev/server teams to see if this is viable.
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

The iRule that we successfully tested on Friday had some minor adjustments made because somewhere in the back end something was wanting the pool name as well :/

# iRule Name: JSESSIONID_Persist_v4
# Last Mod: AT 2015/09/04

when HTTP_REQUEST {
	# Check if there is a JSESSIONID cookie in the request header.
	if { [HTTP::cookie "JSESSIONID"] ne "" }{
		# Run a lookup for any existing records, return relevant values for any matches 
		# otherwise ignore and follow load balancing method to select destination server.
		set ppool [ persist lookup uie [string tolower [HTTP::cookie "JSESSIONID"]] pool ]
		set pnode [ persist lookup uie [string tolower [HTTP::cookie "JSESSIONID"]] node ]
		set pport [ persist lookup uie [string tolower [HTTP::cookie "JSESSIONID"]] port ]
		if { $pnode != "" }{
			pool $ppool member $pnode $pport
		}
	}
}
when HTTP_RESPONSE {
	# Check if there is a jsessionid cookie in the response.
	if { [HTTP::cookie "JSESSIONID"] ne "" }{
		# Persist off of the cookie value with a timeout of 6 minutes to ensure that 
		# the F5's persistence entry will not expire before the server token.
		persist add uie [string tolower [HTTP::cookie "JSESSIONID"]] 315
	}
}
0
Comments on this Answer
Comment made 07-Sep-2015 by Opher Shachar 144
This iRule seems overly verbose in the request part and not quite clear why you *lower-case* the cookie in the response. Also, do you need to support User-Agents (i.e. browsers and such) that do not support cookies?
0
Comment made 07-Sep-2015 by AlanTen 75
I agree that it is a bloated and I need to find a better way to split out the info pulled back from the persistence records. Due to time constraints and the need to get it working in the prod environment I went the verbose route so I could test function rather than performance. What I would like to do is use a single 'persist lookup all' which contains all the information I need and then split it into two or three variables so that they can be used with the 'pool' command, but I'll only be able to look into this some time later today or tomorrow. I don't know why the cookie is being lower-cased since I did not write the original iRule, but because it's being inserted that way I need to reference it in the same way in the requests :( I suspect that whoever did just copy-pasta'd from someone else' generic iRule. Thankfully the client app is a fat client and is the only connecting client that we are concerned about and so we don't have to worry about supporting varied User-Agents :)
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Here is a cut down version of our persistence iRule.
It offers a more robust handling of persistence:
* stale session cookie
* no session cookie
* allows debug logs (on persistence or request level)

# Weblogic Persist iRule, Version 0.93
# Mars, 2012
# Last update: Aug, 2012
# Updated v11.6.0: 15/02/2015

# Created by Opher Shachar (contact me through devcentral.f5.com)

# Purpose:
# This iRule persists requests based on the JSESSIONID cookie issued by the
# Weblogic server.

when RULE_INIT {
   # Set debug level to: 0, 1 or 2
   set static::Weblogic_debug 1
   # Set a timeout of 5.25 minutes (315 seconds)
   set static::Weblogic_timeout    315
}

when CLIENT_ACCEPTED {
   set log_prefix "Client [IP::client_addr]:[TCP::client_port]"
}

when HTTP_REQUEST {
   # Log details for the request
   if {$static::Weblogic_debug > 1} {
      log local0. "$log_prefix: Request to [HTTP::uri] with cookie: [HTTP::cookie value $static::Weblogic_CookieName]"
   }

   # Check if there is a session cookie
   set pkey [HTTP::cookie $static::Weblogic_CookieName]

   # Did we find a session id?
   if {$pkey ne ""} {
      # Persist off of the session id value with the set timeout
      if {[persist lookup uie $pkey] ne ""} {
         persist uie $pkey $static::Weblogic_timeout
         if {$static::Weblogic_debug > 1} {
            log local0. "$log_prefix: Used persistence record from cookie or path: [persist lookup uie $pkey]"
         }
      } else {
         # Client gave stale session id  --  don't persist
         persist none
         if {$static::Weblogic_debug > 0} {
            log local0. "$log_prefix: No persistence record found for key: $pkey"
         }
      }
   } else {
      # Client gave no session id  --  don't persist
      persist none
      if {$static::Weblogic_debug > 1} {
         log local0. "$log_prefix: No session id given in cookie or path."
      }
   }
}

when HTTP_RESPONSE {
   # Check if there is a session cookie in the response
   set pkey [HTTP::cookie $static::Weblogic_CookieName]

   # Did we find a session id?
   if {$pkey ne ""} {
      # Persist the session id value with the set timeout
      persist add uie $pkey $static::Weblogic_timeout
      if {$static::Weblogic_debug > 0} {
         set val [persist lookup uie $pkey]
         log local0. "$log_prefix: Added persistence record from cookie or redirect: $pkey -> $val"
      }
   }
}

Cut out are:
* persistence on JVMID alone (reduce persistence table size)
* support for cookie-less user-agents
* workaround for bug in pre-v11.1.0 HF2

0
Comments on this Answer
Comment made 07-Sep-2015 by Opher Shachar 144
I had an omission but seem unable to edit my answer. The 'RULE_INIT' part should look like this: ~~~ when RULE_INIT { # Set debug level to: 0, 1 or 2 set static::Weblogic_debug 1 # Set the name of the Weblogic session cookie set static::Weblogic_CookieName "JSESSIONID" # Set a timeout of 5.25 minutes (315 seconds) set static::Weblogic_timeout 315 } ~~~
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

One thing you may want to look into and test with to make sure it does not upset your application server is to set the expires options on the JSESSIOINID cookie, or better yet see if the application server can set it.

That way the only time a inbound request will have the cookie is when the cookie is still valid in the application server. It would mean a bit more logic in your iRule.

In HTTP_RESPONSE you would do something like:

if { [HTTP::cookie "JSESSIONID"] ne "" }{
        # Set the cookie to expire 290 seconds from now.
        HTTP::cookie expires JSESSIONID 290 relative
        # Persist off of the cookie value with a timeout of 4m50s to ensure that 
        # the F5 will never send an expired cookie back to the application.
        persist add uie [HTTP::cookie "JSESSIONID"] 290
    }

You might have to experiment, I'm not 100% sure that the above example will add the expired information, you might have to save the value of the current JSESSSIONID cookie, delete the existing one and then insert the cookie.

I also took the "string tolower" out, this removes the overhead of converting the JSESSIONID to all lower case letters. So you will need to take it out of the HTTP_REQUEST section of code. It will also remove the possibility of having two JSESSIONID's that are different due to case (JSESSIOND=AbCd and JSESSIOND=aBcD) end up being used in a persistence record as the same because they are both converted to "abcd".

I used 290 seconds instead of 315. I personally think it is better for the browser and the F5 to timeout the cookie sooner than the back-end application. That way a request will never get back to the application with an expired JSESSIONID.

Again, if you can get the application server to set the expires date, that would be best then you only have to worry about the persistence record, but still set it just slightly less than the applications time.

0
Comments on this Answer
Comment made 07-Sep-2015 by Opher Shachar 144
This will probably wreck havoc on the application :=) HTTP Cookie expiration logic differs (fundamentally) to (Java) Application server session expiration. The former sets the cookie to expire at a fixed future time. The latter, otoh, expires the session (cookie) when it has been idle for a set number of seconds from last access.
0
Comment made 07-Sep-2015 by giltjr 942
AlanTen needs verify how the application works, then. Originally he said application set the JSESSIONID cookie to expire after 5 minutes period, not from the last request, but that could be his misunderstanding. If their application is using 5 minutes of inactivity then he does need to reset the uie timeout value on each http request, if the persistence record still exists. If the record does not exist, then he needs to remove the JSESSIONID cookie from the request header and let the F5 reload balance the request and add a new persistence record in HTTP_RESPONSE.
0
Comment made 08-Sep-2015 by Opher Shachar 144
Indeed he misunderstood if that's what he said. The Java Servlet Specification http://download.oracle.com/otn-pub/jcp/servlet-3_1-fr-eval-spec/servlet-3_1-final.pdf defines in section 7.5 the Session Timeout as a number of seconds of *inactive interval*.
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Quick question, is there a special reason you are using uie for persistence? Why not just setup a persistence profile that uses a cookie, set it to a non-session cookie have it's lifetime be 240 seconds, or 315, or whatever you choose.

0
Comments on this Answer
Comment made 07-Sep-2015 by Opher Shachar 144
If you're referring to *cookie insert mode* the the application can't control the lifetime of the F5's LTM cookie. If the user hits *logout*, say after 60 seconds, F5 will still be sending him to the same backend server for another 180 seconds.
0
Comment made 07-Sep-2015 by giltjr 942
We look for the "logout" URI and delete all persistence records that deal with that JSESSIONID. Wonder if he can do the same thing.
0