Forum Discussion

JamesS_40157's avatar
JamesS_40157
Icon for Nimbostratus rankNimbostratus
Sep 17, 2009

404 re-load balance in pool

Hi all,

 

 

Firstly apologies if this has already been covered - I've tried searching the forums and google but can't find exactly what I'm looking for. I'm also very new to irules so please forgive my ignorance here!

 

 

We have a pool of servers that in theory should have the same content on each but in practice they do not - that is to say at any one time 1 or 2 servers could potentially serve a 404 on a piece of content.

 

 

What I'm after is a rule to say that if the load balancer receives a 404 error, to transparently re-balance to the next node, and then re-balance again to the next node until it has tried all servers in the pool. At this point I'd like it to serve a 404 to the client if it cannot find the content anywhere. To me this seems like a trivial thing a load balancer should do but I can't see to find info on it anywhere!

 

 

I've had a go at writing the rule however it does not work perfectly (using LB:reselect) as this relies I belive on the load balancer mechanism which is currently set to round-robin. Therefore it will not be foolproof to ensuring it tries each and every node in the pool for a single client request.

 

 

This is the code I have ( mainly copied from an older post on this forum)

 

 

when CLIENT_ACCEPTED {

 

set retries 0

 

}

 

 

when HTTP_REQUEST {

 

set request [HTTP::request]

 

}

 

 

when HTTP_RESPONSE {

 

if { [HTTP::status] eq "404"} {

 

if { $retries < [active_members mediapool] } {

 

incr retries

 

HTTP::retry $request

 

} else {

 

HTTP::respond 404

 

}

 

}

 

}

 

 

This does not guarantee to try each member of the pool per client request. I'm sure I'm close but can't see the final step...could someone please help me out? Many thanks!
application delivery
dev
devops
iRules

5 Replies

Sort By
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Sep 17, 2009
    Hi James,

    That's a nice rule to start with...

    You could try to track each server that's been requested and failed, but that's going to get expensive in terms of memory and CPU to process the used members for every request which generates a 404. If you wanted to try, you could use [active_members -list] (Click here) if you're running 9.4.2 or higher. You'd need to save the available members to a list and then move them to a tried/failed list.

    Also, HTTP::request only returns the HTTP headers from a request. And it would be very expensive in terms of memory to try to collect every request payload to potentially use if there is a 404 response. So you might be better off checking the request method using HTTP::method and only saving the request headers for GET requests. You could then add a check in HTTP_RESPONSE for a 404 status to a GET request: 

     
 when CLIENT_ACCEPTED { 
    set retries 0 
 } 
 when HTTP_REQUEST { 
  
    if {[HTTP::method] eq "GET"}{ 
  
        Save the request headers 
       set request [HTTP::request] 
    } else { 
  
        Not going to retry this request, so unset the variable 
       unset request 
    } 
 } 
 when HTTP_RESPONSE { 
  
     Check if the response is a 404 and it was a response to a GET request 
    if { [HTTP::status] == 404 && [info exists request]} { 
  
        Check if we've tried fewer times than number of pool members 
       if { $retries < [active_members mediapool] } { 
  
           Retry request 
          incr retries 
          HTTP::retry $request 
       } else { 
  
           Give up and send a 404 
            We could just do nothing and send back the application's response... 
          HTTP::respond 404 
       } 
    } 
 }

    Aaron

  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Sep 18, 2009
    What about resetting the retries to 0 in an else clause if this is false: ' {[HTTP::status] == 404 && [info exists request]}'?

     

     

    Aaron
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Sep 21, 2009
    Can you add logging to the iRule and post the log output when a failure occurs? Try to include the client IP:port in the log statements so you can track what requests were made over the same TCP connection.

     

     

    I'm not sure if there is another approach that would work.

     

     

    Aaron
  • JamesS_40157's avatar
    JamesS_40157
    Icon for Nimbostratus rankNimbostratus
    Sep 21, 2009
    Hi hoolio - yes I added logs however I know they were definitely different connections as I initiated my tests from 2 separate phsyical servers/IP's. If I started to request 2 different bits of content they would get 404's every now and then which we can't have. I'm going to start to look at a different way of doing this..if I get anywhere I'll post it as I'm sure this is a reasonable thing a load balancer should do!

     

     

    Cheers
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Sep 21, 2009
    Actually, if the servers don't all have the same content and the load balancing is done based on round robin it's possible that a request could be re-load balanced to servers that don't have the content. I think the only way to guarantee the request would be tried against all the servers until one is found that serves the content would be to explicitly check each server using [active_members -list].

     

     

    Another option for improving this might be to use RAM caching. Assuming you didn't cache 404 responses, you could reduce the number of requests to the pool by answering the requests from cache on LTM.

     

     

    Aaron