Forum Discussion

Jgawrych_43121's avatar
Jgawrych_43121
Icon for Nimbostratus rankNimbostratus
Mar 18, 2008

Kill all active connections to a Vserver during maintenace....

 

I'm working through a process for placing our site into maintenace mode for application upgrades etc. This process is going to be handed of to 1st level NOC people who are smart but novices to the BigIP - so I want the act of taking a virtual server in and out of maintenace to be a simple and straight forward as possible.

 

 

My preffered method for the NOC is to log onto the LTM and disable all pool members for the pool associated with the VServer we want to do maintenace on.

 

Local Traffic / Pools / "PoolName" / Members Tab. Disable all members.

 

 

I have a simple Irule applied to the VServer that I previously found here on Dev central that redirects browsers to a maintenance page when all pool members are down (see bottom of this message for the iRule).

 

 

This works as expected for any NEW connections that are initiated to the Vserver. When all the Nodes in a pool are disable, the users browser goes to the maintenace page

 

 

My Problem and what I'm looking for help on is that when we go into maintenace, we want ALL sessions through the Vserver to end and go to maintenace page. Don't care if users are in the middle of something - we want all traffic to the web servers behind the BigIP to stop.

 

 

However, by default existing connections will remain connected when node/pool member or virtual servers are marked down. In other words, if a user is logged onto the application and has an open browser session, they will stay active on the application even though we've marked everything down . (I was actually pretty surprised to find that there was not a configuration parameter that would by default immediately stop all traffic to Vserver, Pool, Node if it was disabled)

 

 

Problem for us because our maintenace usually has the webservers up and functional but we are running DB updates where we do not want user traffic interacting with the DB.

 

 

F5 Support pointed me to some Icontrols to kill TCP connections but I'm not sure if iControls are the best route to take on this.

 

 

http://devcentral.f5.com/wiki/default.aspx/iControl/System__Connections__delete_active_connection.html

 

http://devcentral.f5.com/wiki/default.aspx/iControl/System__Connections__delete_all_active_connections.html

 

http://devcentral.f5.com/Wiki/default.aspx/iControl/System__Connections.html

 

 

Anyone have any suggestions on irules that might be able to help? (e.g. an iRule to identify all active connections to a vserver/pool and kill/reset them, etc) Any different approaches any one has used in their production enviroments for halting all traffic to a given VServer during a maintenace event?

 

 

Thanks!

 

-John G

 

 

PS - Running BIG-IP 9.3.1 Build 37.1

 

 

******************************************************************

 

 

Generic iRule to redirect all traffic when all nodes in a pool are down.

 

 

when LB_FAILED {

 

 

HTTP::redirect "http://maintenance.sitename.com/"

 

 

}

5 Replies

  • Hi John,

     

     

    It's my understanding that setting the action on service down to reject on the pool dictates that existing connections are reset when the pool member is down. You can check the config guide for your version or try testing it to confirm.

     

     

    Aaron
  • Hi,

     

     

    You don't have a specific command to kill all connection on a vs or a pool when a pool is down.

     

     

    The idea aaron is offering is surely the best one.

     

     

    Otherwise you may try to use the LB_FAILED event to do something like:

     

     

    Click here

     

     

    when LB_FAILED {

     

    reject

     

    }
  • Thanks to the previous posters for responding! I'm new to the LTMs and appreciate your repsonses. I've done some additional research/work on this.

     

     

    >>It's my understanding that setting the action on service down to reject

     

    >>on the pool dictates that existing connections are reset when the pool

     

    >>member is down.

     

     

    That was my expectation also and does seem to be case for any NEW HTTP/TCP sessions going to the Vserver/pool/nodes. However, CURRENT ESTABLISHED sessions running through the VServer do not appear to get reset/redirected and continue on if you disable the Vserver, pool, or node members. This behavior is most noticable on HTTP .NET/Java applications where the application has something going on in the background that keeps the browser from immediately closing a HTTP/TCP session after a trasmission is done. ( in my case, our appliation has sometype of Java applet that downloads and runs through the browser ).

     

     

    I actaully ticketed with f5 Support on this and:

     

    a) They confirmed that a manually disabled object (Vserver, Pool, node) will indeed continue to pass traffic for an established TCP session. New sessions will be rejected/reset.

     

    - In my own testing this is definitely the case if you disable all the nodes and there are no other irules to modify behavior - established TCP sessions do continue on to the nodes. I was able to continue using my webapp continuously for 5-6 minutes after shutting down all nodes before I gave up testing.

     

    - What was also interesting was that if you knew what pool member your traffic was going to and disabled just that one member, the LTM still continued to pass established sessions to that server. Interesting Implications if need to immediately pull a problem server out of the pool and did that by just disabling that one node. (e.g. - user traffic on a given webserver is causing that application server to send send problematic sql queries to a backend database)

     

    - Shutting the the Vserver seems to have a more immediate impact for terminiating all active traffic - but you loose the ability to process irules and redirect traffic to a maintenace page).

     

    b) they indicated there are no configuration options to change that behavior. They pointed me towards some icontrols about doing TCP resets to connections (see my original post).

     

     

    I was pretty surprised at that response. F5 has been doing this stuff since forever and that enoung people would have asked about this that it would have been added as a configurable option.

     

     

     

    My functional requirement was to was to be able to shutdown all nodes in a pool to enable a mainatenace window and have all traffic (new requests and existing sessions) redirect to a maintenance page. I found a differnt irule on Devcentral that used a HTTP_REQUEST with 'active_members rather than a LB_FAILED method that does work as expected. See irules below.

     

     

    this iRULES allows established TCP Connections to conitue on to the pool node members even if they are all manually disabled

     

    when LB_FAILED {

     

    HTTP::redirect "http://maintenance.sitename.com/"

     

    }

     

     

    this iRULES checks pool member status with each http request. If all pool members are disbaled,

     

    immediately sends all traffic (new requests as well as established TCP sessions) to the maintenace page.

     

    when HTTP_REQUEST {

     

    if {[active_members [LB::server pool]] == 0} {

     

    HTTP::redirect "http://maintenance.sitename.com/"

     

     

     

    }

     

    }

     

     

     

    Thanks!

     

    -John G

     

  • Hi,

     

     

    a)it's the expected behavior yes. When you disable a node it switches the state of this node to "down except for active and persisted connections" you can see that by clicking on the disabled member in the GUI

     

     

    I thought you wanted to handle case were all the pool member were really down not disabled

     

     

    Your iRule is definitely the best option !

     

  • I don't know if this can help, but I built a rule that allows specifically listed people to be able to enable or disable maintenance mode on a VIP. This rule is not dependent on node manipulation, but rather stores information in a global array:

    
    when RULE_INIT {
      array set ::maintmode { }
    }
    when HTTP_REQUEST {
      if { ([ info exists ::maintmode([virtual]) ] and ( $::maintmode([virtual]) == 1 )) or ( [HTTP::uri] equals "/enmaintmode" ) or ( [HTTP::uri] equals "/dismaintmode" ) } {
       maintenance mode is set or attempting to set or unset
      
        switch [HTTP::uri] {
          "/enmaintmode" {
        if { [matchclass $::maintenance_mode_users equals [lindex [session lookup ssl [SSL::sessionid]] 1]] } {
          set ::maintmode([virtual]) 1
      HTTP::respond 200 content "Maintenance Mode Settings"
    } else {
       send content and die
      HTTP::respond 200 content $::error_html Connection Close
      event HTTP_REQUEST disable
      SSL::session invalidate
    }
      }
      "/dismaintmode" {
        if { [matchclass $::maintenance_mode_users equals [lindex [session lookup ssl [SSL::sessionid]] 1]] } {
      set ::maintmode([virtual]) 0
      HTTP::respond 200 content "Maintenance Mode Settings"
    } else {
       send content and die
      HTTP::respond 200 content $::error_html Connection Close
      event HTTP_REQUEST disable
      SSL::session invalidate
    }
      }
      default {
         send content and die
        HTTP::respond 200 content $::maintmode_html Connection Close
        event HTTP_REQUEST disable
    SSL::session invalidate
      }
        }
      }
    }

    The basic premise is this: The site requires a global array and a data class "maintenance_mode_users" (a single string table) that lists the users that can make this work. We use smart cards, so on initial connect, I store some information from the user's certificate in an ssl session array. This could also be done with IP addresses or other unique values, I suppose, but I'd be careful to choose something that can't be spoofed. When the user goes to the site and enters "/enmaintmode" as the URI, the rule first checks to see if the user is listed in the data class. If they are, the name of the virtual server is added to the global array with the value of 1 (ex. "TEST_VS" 1). Next time through the request process, regardless of the URI, the maintmode flag for the virtual server is set and a static maintenance page is displayed. If "/dismaintmode" is entered and the user is valid in the data class, maintmode for the virtual server is set to 0 and the maintenance page is disabled. Resetting the iRule in the GUI or a "B LOAD" at the command line will also reset, or rather destroy the global array entries. By the way, there are also two other global variables, $::error_html and $::maintmode_html. These are the actually HTML pages displayed to the user. $::error_html is displayed if a non-authorized user tries to use the maintenance URIs. $::maintmode_html is the maintenance page users would see if maintenance was enabled for the virtual server. They don't have to be global variables, of course. I just did it that way for readability.

    HTH

    Kevin