Forum Discussion

squeezebox_2829's avatar
squeezebox_2829
Icon for Nimbostratus rankNimbostratus
Aug 05, 2016

HTTP::retry doesn't work on BIGIP LTM 10.2.4

I am attempting to reference an external service which performs a lookup based on the request's ip address and returns a response either allowing or disallowing the traffic. I am using the solution posted here as my baseline: https://devcentral.f5.com/articles/conditioning-irule-logic-on-external-information-01-httpretry

 

I have found this code doesn't work to completion. It seems to receive the request, modify it, send it to the external service, receive the response back, and analyse it. At that point it does seem an HTTP retry is issued and the iRule is hit again. At this point the iRule successfully skips the external service call but the retry request doesn't seem to hit any member of the pool.

 

I have seen several other commenting with similar type rules that they are using, so I am hoping someone can point out what I am doing wrong. I am thinking perhaps the retry requests are originating from the LB address instead of the original client IP address and are not making their way back out to the proper pool or back to the client.

 

Here is my iRule

 

 when RULE_INIT {
    set string to match for valid connection
   set valid_string "Sure"
}
when CLIENT_ACCEPTED {
    set flag to control logical flow. 1 means lookup is pending.
   set lookup 1
    set string to match for valid connection
   set valid_string "Sure"
}
when HTTP_REQUEST {
   If each request on the same connection must force a lookup,
   re-initialize the value of the flag here 
   In this case, lookup result based on client IP is good for
   the life of the connection, so we'll leave the flag alone here.
   set lookup 1
  if {$lookup == 1} {

save the original request and original payload and payload length
set original_request [HTTP::request]
set original_payload [HTTP::payload]
set original_payload_length [HTTP::payload length]

 if client hasn't already been looked up, save the 
 request so we can replay it to the LB server later;
set LB_request [HTTP::request]
set LB_uri [HTTP::uri]

 inject lookup URI in place of original request;
HTTP::uri "/getAllowed?ip=[IP::client_addr]"
 and send the out-of-band validation query to the DB_pool.
pool geo_pool
  } else {

correct the request payload and content length
set retry_payload_length [HTTP::payload length]
HTTP::payload replace 0 $retry_payload_length $original_payload
if { [HTTP::header exists "Content-Length"] } {
HTTP::header replace "Content-Length" $original_payload_length
}


 otherwise, send the request to the LB pool
pool reg_pool
  }
}
when HTTP_RESPONSE {
   If lookup flag is still on in response event, this is the response
   to the lookup, so we collect entire payload (up to 1MB limit) 
   both to evaluate the DB server response and to prevent this response
   from being returned to the client.
   Already-validated connections will bypass the rest of the rule.
  if {$lookup == 1}{
if {[HTTP::header exists Content-Length] && \
 ([HTTP::header Content-Length] < 1048576)}{
   set clength [HTTP::header Content-Length]
} else {
   set clength 1048576
}
HTTP::collect $clength
  }
}
when HTTP_RESPONSE_DATA {
   HTTP_RESPONSE_DATA will only be triggered for a DB lookup.
   (All other requests have already been forwarded to the LB pool.) 
   If response from DB indicates connection is valid, reset the
   lookup flag & replay the request to the LB server.
   Otherwise, reject the connection

  if {[HTTP::payload] contains $valid_string}{
set lookup 0
pool reg_pool
HTTP::retry $LB_request
  } else {
log "rejected"
reject
  }
   depending on the app, reset the value of the flag here
   in case of new request on same connection
   (useful for situations where each URI requires a lookup)
   In this case, lookup result based on client IP is good for
   the life of the connection, so we'll leave the flag set to 0
  set lookup 0
}

3 Replies

  • Hi Squeezebox,

    using

    HTTP::retry
    to query external services is an very outdated approach and also with lots of limitations and performance implications.

    I'd like to strongly recommend to update your box to at least TMOS v11 and query the external service using a

    [SIDEBAND]
    connection feature and if somehow possible try to cache the results into LTMs session
    [table]
    .

    If this is not possible, then report back and I'll try my best to catch all the glitches in your existing iRule... 😞

    Cheers, Kai

  • Hi Squeezebox,

    take a look to the iRule below. Its a polished version of your iRule with an added

    [TABLE]
    functionality to cache previous verification results and an additional filter to
    HTTP::retry
    only request using a
    GET
    method (the verification of other methods would have lots of implications).

    when RULE_INIT {
         set string to match for valid connection
        set static::valid_string "Sure"
        set static::table_cache_duration "10" ; seconds
    }
    when CLIENT_ACCEPTED {
         set flag to control logical flow. 1 means lookup is pending.
        set lookup 1
    }
    when HTTP_REQUEST {
         If each request on the same connection must force a lookup,
         re-initialize the value of the flag here 
         In this case, lookup result based on client IP is good for
         the life of the connection, so we'll leave the flag alone here.
        if { $lookup } then {
             verify only GET request.
            if { [HTTP::method] eq "GET" } then {
                 try to offload the verification of the current request with data of previous lookup results
                if { [table lookup -notouch "is_allowed_[IP::client_addr]"] eq "" } then {
                     No previous request data is cached. Performing an online verification...
    
                     save the request so we can replay it to the LB server later;
                    set orig_request [HTTP::request]
    
                     inject lookup URI in place of original request;
                    HTTP::uri "/getAllowed?ip=[IP::client_addr]"
    
                     remove any cookie information for security purposes
                    HTTP::header remove "Cookie"
    
                     and send the out-of-band validation query to the DB_pool.
                    pool geo_pool
    
                } else {
                     The client is verified using cached results.
                    set lookup 0
                }   
            } else {
                 Current request is not a GET request. Do not verify the request to reduce complexity of collection and replay of HTTP::payload data.
                set lookup 0
            }
        }
    }
    when HTTP_RESPONSE {
         If lookup flag is still on in response event, this is the response
         to the lookup, so we collect entire payload (up to 1MB limit) 
         both to evaluate the DB server response and to prevent this response
         from being returned to the client.
         Already-validated connections will bypass the rest of the rule.
        if { $lookup } then {
            if { ( [HTTP::header value "Content-Length"] ne "" ) and 
                 ( [HTTP::header value "Content-Length"] < 1048576) } then {
                HTTP::collect [HTTP::header "Content-Length"]
            } else {
                HTTP::collect 1048576
            }
        }
    }
    when HTTP_RESPONSE_DATA {
         HTTP_RESPONSE_DATA will only be triggered for a DB lookup.
         (All other requests have already been forwarded to the LB pool.) 
         If response from DB indicates connection is valid, reset the
         lookup flag & replay the request to the LB server.
         Otherwise, reject the connection
    
        if { [HTTP::payload] contains $static::valid_string} then {
             Cache the OK response to offload subsequent requests
            table set "is_allowed_[IP::client_addr]" "1" indefinite $static::table_cache_duration
             Replay the origianl request
            set lookup 0
            pool reg_pool
            HTTP::retry $orig_request
        } else {
            log "rejected"
            reject
        }
    }
    

    Note: If written the iRule out of my mind. If the syntax isn't working correctly, then post back the error logs to see whats going wrong...

    Cheers, Kai

  • Hi Squeezebox,

    good to know that the iRules is (more or less) working for you and that you've found the remaining glitches by your own. I've also found a remaining bug, which is fixed in the version below...

    when RULE_INIT {
         set string to match for valid connection
        set static::valid_string "Sure"
        set static::table_cache_duration "10" ; seconds
    }
    when CLIENT_ACCEPTED {
         set flag to control logical flow. 1 means lookup is pending.
        set lookup 1
        set get_request 0
    }
    when HTTP_REQUEST {
         If each request on the same connection must force a lookup,
         re-initialize the value of the flag here 
         In this case, lookup result based on client IP is good for
         the life of the connection, so we'll leave the flag alone here.
        if { $lookup } then {
             verify only GET request.
            if { [HTTP::method] eq "GET" } then {
    
                 try to offload the verification of the current request with data of previous lookup results
                if { [table lookup -notouch "is_allowed_[IP::client_addr]"] eq "" } then {
                     No previous request data is cached. Performing an online verification...
    
                     save the request so we can replay it to the LB server later;
                    set orig_request [HTTP::request]
    
                     inject lookup URI in place of original request;
                    HTTP::uri "/getAllowed?ip=[IP::client_addr]"
    
                     remove any cookie information for security purposes
                    HTTP::header remove "Cookie"
    
                     and send the out-of-band validation query to the DB_pool.
                    pool geo_pool
    
                } else {
                     The client is verified using cached results.
                    set lookup 0
                    pool reg_pool
                } 
                set get_request 1
            } else {
                 Current request is not a GET request. Do not verify the request to reduce complexity of collection and replay of HTTP::payload data.
                set get_request 0
                pool reg_pool
            }
        }
    }
    when HTTP_RESPONSE {
         If lookup flag is still on in response event, this is the response
         to the lookup, so we collect entire payload (up to 1MB limit) 
         both to evaluate the DB server response and to prevent this response
         from being returned to the client.
         Already-validated connections will bypass the rest of the rule.
        if { $lookup and $get_request } then {
            if { ( [HTTP::header value "Content-Length"] ne "" ) and 
                 ( [HTTP::header value "Content-Length"] < 1048576) } then {
                HTTP::collect [HTTP::header "Content-Length"]
            } else {
                HTTP::collect 1048576
            }
        }
    }
    when HTTP_RESPONSE_DATA {
         HTTP_RESPONSE_DATA will only be triggered for a DB lookup.
         (All other requests have already been forwarded to the LB pool.) 
         If response from DB indicates connection is valid, reset the
         lookup flag & replay the request to the LB server.
         Otherwise, reject the connection
    
        if { [HTTP::payload] contains $static::valid_string} then {
             Cache the OK response to offload subsequent requests
            table set "is_allowed_[IP::client_addr]" "1" indefinite $static::table_cache_duration
             Replay the origianl request
            set lookup 0
            pool reg_pool
            HTTP::retry $orig_request
        } else {
            log "rejected"
            reject
        }
    }
    

    Note: The bugfix makes sure that just the non-GET requests can bypass the IP validation (added the variable

    $get_request
    and additional checks). In the previous version of my iRule it was possible, to unlock the full communication by using a single non-GET request.

    Cheers, Kai