Forum Discussion

Krzysztof_Kozlo's avatar
Krzysztof_Kozlo
Icon for Nimbostratus rankNimbostratus
Jul 20, 2006

Fetching multiple headers

We are attempting to use Universal Persistence to work around source IP addresses being obscured by intervening infrastructure (i.e. proxy servers). The proxy servers in our environment insert an X-Forwarded-For header, as does the BigIP. Normally, the value of the header we wish to base universal persistence on is that of the first X-Forwarded-For.

 

 

However, the BigIP does not appear to allow you to fetch the first header using HTTP::header if a subsequent header appears. (Please don't tell me that the HTTP RFC does not permit multiple headers with the same field-name, the BigIP itself will create a new header if "Insert X-Forwarded-For" is enabled instead of appending to the existing header.)

 

 

I can't even fetch headers by index, which would allow me to iterate the header-names, increment a variable, and retrieve the header -- HTTP::header at n returns only the header name, not the header value.

 

 

The iRules quick reference seems to suggest that HTTP::request and HTTP::collect will somehow return the entire HTTP request, which would also do the job, but I can't get these working.

 

 

HTTP::collect

 

set x [HTTP::payload]

 

 

this returns null

 

 

HTTP::request doesn't even compile.

 

 

The only way I've found to get what I want so far is to TCP::collect when CLIENT_ACCEPTED and then parse the TCP::payload. There has to be a better way.

 

 

Incidentally I notice several others asked about how to get at the nth Set-Cookie or Accept header, and when the initial responses proved unhelpful or incomplete, the questions seemed to die on the vine. As I am often encouraged to search through or post my questions in DevCentral, I harbor some hope these are the exception rather than the norm.
application delivery
dev
devops
iRules

9 Replies

Sort By
  • unRuleY_95363's avatar
    unRuleY_95363
    Historic F5 Account
    Jul 20, 2006
    Yes, we are aware of that limitation and no, we are not going to saying anything like the RFC doesn't allow that (because it quite specifically does). The problem is more of an implementation detail. We hope to eventually get it fixed. It might help to make a call into support so a Case file gets created and tracked. This will help raise the priority of such a requested feature enhancement. Thanks.
  • Krzysztof_Kozlo's avatar
    Krzysztof_Kozlo
    Icon for Nimbostratus rankNimbostratus
    Jul 20, 2006
    Thanks for the fast response. I suppose HTTP::request and HTTP:collect on HTTP_REQUEST events are not implemented in 9.1.1 which we run here.

     

     

    I find myself wishing there was an LTM guide for 9.1 available as there are definitely some features in 9.1 that don't exist in 9.0 and yet reading the 9.2 guide one has to simply experiment to see what works and what doesn't.

     

     

    Is there any way to specify a character class or a set of characters to terminate the findstr command? Even the TCP::collect kluge I proposed above won't work properly unless I can specify the findstr term character being either a comma or a space/CR, as I need to retrieve the first IP following the first XFF header and that can be one-per-line or a set of comma-separated values following a single field-name.

     

     

     

  • Joe_Pruitt's avatar
    Joe_Pruitt
    Jul 20, 2006
    I can't speak for the ltm guides, but the wiki documentation for the iRule commands contain an "Introduced" field that lists the product version they were first available. In the case of HTTP::request, it states BIGIP-9.2.0.

     

     

    -Joe
  • bl0ndie_127134's avatar
    bl0ndie_127134
    Historic F5 Account
    Jul 21, 2006
    Ok, we will give the people what they want. We are considering adding the follwing new rule.

     

     

    HTTP::header values

     

     

    This rule will return all header values as a TCL list (same order they appear in the header). This will most probably make it out on the next release of BigIP V9.4.
  • Krzysztof_Kozlo's avatar
    Krzysztof_Kozlo
    Icon for Nimbostratus rankNimbostratus
    Jul 21, 2006
    That would be great, as the current HTTP::header functions are simply inadequate for application in a high-volume heterogenous production environment. It's hard enough to persuade owners of client applications to adhere to RFCs without having to ask them to further limit their implementation to i.e. unique field-names.

     

     

    The irony here is that the BigIP steps on its own tail when you ask for the X-Forwarded-For header -- it supplies you with the one it just inserted!

     

  • bl0ndie_127134's avatar
    bl0ndie_127134
    Historic F5 Account
    Jul 21, 2006
    Then why don't you remove the X-Forwarded-for option from the http profile and insert it in the rule after you retreive the upstream header value. That might allow you to get around this for the time being.
  • Krzysztof_Kozlo's avatar
    Krzysztof_Kozlo
    Icon for Nimbostratus rankNimbostratus
    Jul 21, 2006
    Good idea, but I can't guarantee that there's not an upstream BigIP (i.e. in front of the web proxies) or another device that already added an XFF header, obscuring the original one to the BigIP in question.

     

     

  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    May 02, 2007
    I ran into this issue of getting the values for headers with identical names (mainly worrying about Set-Cookie). It looks like the 'HTTP::header values ' command Bl0ndie mentioned has been added in 9.4:

    
when HTTP_RESPONSE {
    insert some test headers with the same name
   HTTP::header insert header_1 value_1
   HTTP::header insert header_1 value_2
   HTTP::header insert header_1 value_3
    test a few options for parsing the headers
   log local0. "\[HTTP::header header_1\]: [HTTP::header header_1]"
   log local0. "\[HTTP::header value header_1\]: [HTTP::header value header_1]"
   log local0. "\[HTTP::header values header_1\]: [HTTP::header values header_1]"
}

    Log output:

    
[HTTP::header header_1]: value_3
[HTTP::header value header_1]: value_3
[HTTP::header values header_1]: value_1 value_2 value_3

    But I'm wondering if multiple headers with the same name could be handled better.

    With the 'values' command, I can actually get the values of headers that exist multiple times (which is a good improvement over past versions). But I have to check to see if a header name exists more than once and then parse the list of values.

    Could a method for getting the values of the headers by their index be added? Before I saw this post, I assumed that the 'HTTP::header at $index' command would return the value of a header at $index. I'm not sure what use there is in getting the header name at a specific index. Maybe 'HTTP::header at $index' could be changed to return the value of the header? Else, maybe a new command could be added, like 'HTTP::header value at $index'.

    I'll submit an RFE on this, but I figured I'd check here first to find out more on the logic for the existing functionality.

    Thanks,

    Aaron
  • Krzysztof_Kozlo's avatar
    Krzysztof_Kozlo
    Icon for Nimbostratus rankNimbostratus
    May 02, 2007
    Agreed that this would be a useful addition, particularly if you could get the size of the index for a given header and calculate that to grab the nth one.