Forum Discussion

johnniebegood_1's avatar
johnniebegood_1
Icon for Nimbostratus rankNimbostratus
Feb 27, 2014

Cpu load high ltm version BIG-IP 11.3.0 3144.0 Hotfix HF8

Hello,

 

Could someone please guide me in this situation. I manage 2 ltm's which i have upgraded from BIG-IP 10.2.4 655.0 Hotfix HF4 to BIG-IP 11.3.0 3144.0 Hotfix HF8

 

F5 Model:BIG-IP 6900

 

Now the cpu is sky rocketing high to 99.5 %.

 

I checked with every virtual server (125) what the cpu was doing at the time with :

 

  • tmsh show ltm virtual (name of virtual )

I counted every every last 5 and 1 minutes seperately but it's just about 48 procent of cpu while the total is flipping at 99.5 %

 

I tried to see with command :

 

  • tmctl -w 200 -a

But i got a lot of information which i find it difficult to read.

 

I looked everywhere in the gui as well to find something where i can see what is taking all the rest of the cpu. Like some kind of top 5 or 10 or something but i couldn't find any.

 

Due to the very high cpu time the ltm's often failed over and even reboots at some time. Also randomly pools drop and become active again later on.

 

I am in the proces of sending the quickviews and the coredumps to f5 but i wonder if someone can maybe tell me what i can do more to see what kind of process is taking up half of the cpu power.

 

Thanks in advance for the help.

 

application delivery
citrix
cloud
LTM
management
oracle
vmware

2 Replies

Sort By
  • Chris_Akker_129's avatar
    Chris_Akker_129
    Historic F5 Account
    Feb 27, 2014

    Hi Johnnie, you could try "bigtop" from the commandline, but if you are truly maxing out the CPU, it could roll the box, too. So I'd be careful with using that during production.

     

    Getting F5 support involved is probably the quickest route to resolution.

     

    I have seen SNAT pool exhaustion cause high cpu load, so if you are using one-armed load balanciing, or doing SNAT on a lot of virtual servers, you could be running out of the tcp ports used for SNAT. Check the /var/log/ltm log file for "Inet port exhaustion" type messages. If you see those, add more Self-IPs to a SNAT pool, and use the SNAT pool instead of "automap".

     

    Hopefully you installed V11.3 on a new volume. If so, I'd consider a rollback to 10.2.4 at your earliest convenience.

     

    Good Luck !!

     

    Chris

     

  • johnniebegood_1's avatar
    johnniebegood_1
    Icon for Nimbostratus rankNimbostratus
    Feb 27, 2014

    Thanks Chris for your support.

     

    I have indeed installed the new version on a new volume so i can do a rollback ;)

     

    I have checked both loadbalancers and only on 1 loadbalancer i found the "Inet port exhaustion" messages. 60 in total and it ended just before i upgraded the server. Is having those 60 in the logs before the upgrade a sign as you stated that i am runing into problems with SNAT pool exhaustion ?

     

    I did not find it on the other server because this server was running for months on one side. After the upgrade i can not find those messages anymore. Don't know if those are fixed or not.

     

    Thanks again.