Forum Discussion

dragonflymr's avatar
dragonflymr
Icon for Cirrostratus rankCirrostratus
Dec 08, 2015

iRule killing BIG-IP

Hi,

 

I have huge issue with iRule obviously killing BIG-IP 2000. In tests iRule logic works OK, but under load BIG-IP is overloaded.

 

Scenario:

 

  • BIG-IP no iRule, 30 to 40k concurrent connections - CPU usage 40-50%
  • Same number of concurrent connections with iRule attached to VS - CPU usage skyrocketing to almost 100%

I am looking for directions what to check first. iRule is quite complicated and not optimized for sure - lack of knowledge and time on my side. Still it looks for me as not so much complicated - so I am puzzled.

 

What to look for first and how?

 

It's HTTP intensive iRule in first place - most of the logic in HTTP_REQUEST event.

 

I wonder what load could be caused by inactive debug statements like:

 

if {0} (log local0. "something"} - of course 0 is some global static variable like $static::debug

 

I have something around 40 of those

 

Then I have nested if - around 8, mostly using HTTP::cookie value $cookie name in if and subtable lookups.

 

Then of course some set/add keys to subtables - one main with around 5000 keys, other just single key in main table.

 

One switch without -global and five conditions resulting in HTTP::response - either 403 or 302 with Location and Set-Cookie

 

So what is first candidate for closer investigation? Should I use timing on (seems that above 11.5.0 it's enabled by default so no need to set in iRule?)

 

Any advises/ideas will be appreciated a lot, I am running out of options here :-(

 

Piotr

 

devops
irule optimization
iRules
LTM
performance

12 Replies

Sort By
Show More
  • dragonflymr's avatar
    dragonflymr
    Icon for Cirrostratus rankCirrostratus
    Dec 10, 2015

    I would like o confirm if my calculations are correct (based on iRule timing video😞

    • CPU budget 4 000M
    • Linux overhead 20% - available budget 3 200M
    • Average CPU usage without iRule - 30%
    • Available CPU budget without crossing 70% CPU usage - 1 280M
    • Expect iRule executions per second 2 500
    • Max iRule CPU usage to not cross 70% - 512K

    And now I am a bit surprised. Simple iRule like below

    when HTTP_REQUEST { 
    HTTP::header remove X-Forwarded-For     
    if { [HTTP::header value "X-Our-Custom-IP"] ne ""} {        
        HTTP::header insert X-Forwarded-For [HTTP::header value "X-Our-Custom-IP"]      
    } else {        
        HTTP::header insert X-Forwarded-For [IP::client_addr]       
    }   
}

    Is after 116 executions showing average 94.1K so it is around 18,38% of available max CPU cycle usage limit (512K)

    Seems that there is quite a limited option for heavy iRule here.

    Is above more or less correct?

    If so my last question is how to calculate theoretical available CPU budget based on known TMOS version (11.5.+) number of CPU, cores per CPU and CPU speed?

    I am especially interested how to do that for VE. Is that safe to use host machine CPU speed times number or vCPU for calculation?

    Piotr