lb_failed with lb mode rr does not seem to produce expeted results

The problem is that even though the rule advances things to the next member, other instances of the same rule might be doing the same thing and things are advanced globally, not just for that rule.

i.e.

pool with 2 members

member 2 goes down

LB_FAILED is triggered for connection 1

iRule instance for connection 2 advances to member 1

LB_FAILED is triggered for connection 2

iRule instance for connection 2 advances the pool again to member 2

iRule instance for connection 1 routes to member 2 (down)

This will only happen with high load when a high percentage of the pool is down.

You can avoid this by keeping track of where things are failing in a table and then making sure you don't select that same member again.

i.e.

when CLIENT_ACCEPTED {
    set retry count to 0 to start off
   set retries 0
}
when LB_SELECTED {
    set initial value for server_addr
   set server_addr [LB::server addr]
}
when LB_FAILED {             
    retry a limited number of times
   if { $retries < 3 }{
       only count as a try if tried a member not previously known to be down
      if { [table lookup -notouch -subtable dont_try $server_addr] != 1 }{
         incr retries
      }
       remember that this node failed
      table set -subtable dont_try $server_addr 1 10 20
      set loop_tries 0
       work around to bug where LB::server is not updated after LB::reselect
      set new_pick [LB::select]
      set server_addr [getfield $new_pick " " 4]
       keep looping until we get a server not in teh dont_try table, loop a maximum of 5 times
       if you have a small pool, you will likely hit the max loop tires quickly, with 2 nodes and 200 concurrent connections I saw 8 max
      while { ([table lookup -notouch -subtable dont_try $server_addr] == 1) and ($loop_tries < 10) }{
         incr loop_tries
         set new_pick [LB::select]
         set server_addr [getfield $new_pick " " 4]
          debug logging to see what is happening in the loop
         log local0. "set addr to $server_addr: loop try: $loop_tries"
      }
       select the new server based on values determined above
      eval $new_pick
      LB::reselect
   } else {
       if all else fails, send a 504 error back to the client and log
      log local0. "504 virtual: [virtual name], retries: $retries, last_server: $server_addr"
      HTTP::respond 504 content { reached max retries or all members of the pool failed } noserver Connection close
   }
}