Forum Discussion

hc_andy_35682's avatar
hc_andy_35682
Icon for Nimbostratus rankNimbostratus
Apr 01, 2010

High connection rate (millions conns/sec)

Hi All,

 

 

Last night at close to 10.30pm, our cacti graphs showed a hugh spike in the number of connections/sec hitting the majority of our inside_vlans (forwarding vips) and virtual servers. This lasted just a few mintues and I have been unable to track it down to what might have caused this.

 

 

The only thing I saw in the logs were:

 

 

Mar 31 22:33:15 local/f5-1-manage emerg logger: Re-starting tmm1

 

Mar 31 22:33:15 local/f5-1-manage emerg logger: Re-starting tmm2

 

Mar 31 22:33:15 local/f5-1-manage emerg logger: Re-starting tmm3

 

 

Does anyone have a clue what might have happened? At first I thought it was a ddos attack but this isn't evident by looking at our router graphs which shows normal traffic levels.

 

 

Here's an example of the cacti graphs I'm talking about. Propel is a dial up acceleration application that our dial up users use. Port 80 is barely used. We'd be lucky to get 1-2 people hitting that page a day, but last night it took a hammering. Not sure if this might have just been a cacti or LTM bug???

 

 

Thanks.

 

 

Andy
config
design

7 Replies

Sort By
  • hc_andy_35682's avatar
    hc_andy_35682
    Icon for Nimbostratus rankNimbostratus
    Apr 01, 2010
    The only other link I could find relating to the issue was that ICMP traffic for our wildcard VIP shot through the roof at the same time.
  • Cspillane_18296's avatar
    Cspillane_18296
    Icon for Nimbostratus rankNimbostratus
    Apr 01, 2010
    Hello hc_andy,

     

     

    it's possible that if there were a sudden burst of traffic this is what caused the tmm process to restart. However, you mention that associated routers show normal traffic levels....which doesn't seem to match.

     

     

    What version of TMOS are you running?

     

     

    Are there any other devices using the forwarding VS's that might have been involved (aside from the routers), maybe a proxy server for instance?
  • L4L7_53191's avatar
    L4L7_53191
    Icon for Nimbostratus rankNimbostratus
    Apr 01, 2010
    An aggressive port scanner could have caused this as well.

     

    -Matt
  • JRahm's avatar
    JRahm
    Icon for Admin rankAdmin
    Apr 01, 2010
    sounds like an infrastructure event, perhaps a mistimed recovery in l2/l3 convergence?
  • hc_andy_35682's avatar
    hc_andy_35682
    Icon for Nimbostratus rankNimbostratus
    Apr 05, 2010
    Thanks to all who chipped in with a reply.

     

     

    I've tracked the issue down to the active unit failing and the backup unit taking over (we have a HA setup). I can only assume that cacti goes a bit "crazy" when this sort of event happens hence why the connection rate is so high. Atleast the graphs pointed me to an issue.

     

     

    I went searching the ltm logs and found that the F5 interfaces had gone down (maybe due to duplex mismatch).

     

     

    Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160016:6: Interface 2.1, link admin status: enabled, link status: down, duplex mode: half, lacp operation state: down

     

    Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160010:6: Link 2.1 removed from aggregation

     

    Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160016:6: Interface 2.2, link admin status: enabled, link status: down, duplex mode: half, lacp operation state: down

     

    Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160010:6: Link 2.2 removed from aggregation

     

    Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160016:6: Interface 2.3, link admin status: enabled, link status: down, duplex mode: half, lacp operation state: down

     

    Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160010:6: Link 2.3 removed from aggregation

     

    Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160016:6: Interface 2.4, link admin status: enabled, link status: down, duplex mode: half, lacp operation state: down

     

    Apr 5 19:02:29 local/f5-1-manage info lacpd[10509]: 01160010:6: Link 2.4 removed from aggregation

     

     

    Cheers.

     

     

    Andy

     

     

     

     

     

  • hc_andy_35682's avatar
    hc_andy_35682
    Icon for Nimbostratus rankNimbostratus
    Apr 13, 2010
    We're running 10.1.0 so hopefully that bug you mentioned is no longer present.

     

     

    We were advised by F5 support that the version we were using had 2 bugs affecting SSL and FTP that can cause the LTM to restart. We applied an engineering hotfix so hopefully the problem has gone away.