Forum Discussion

Movva_110216's avatar
Movva_110216
Icon for Nimbostratus rankNimbostratus
Apr 07, 2011

http_request payload logging is not full

Hi Experts,

 

 

I have the below simple irule

 

when HTTP_RESPONSE {

 

if {[HTTP::header value Content-Type] contains "application/vnd.syncml+xml"}

 

{ log local0. " [clock format $secs -format {%H:%M:%S}].$fract HTTP::payload: [HTTP::payload]" }

 

}

 

 

but when i see the logged data the payload is icomplete. Attached the output:

 

 

in the file we can see the xml tag at the end is incomplete.

 

 

 

How can i generate/log the complete payload ?

 

 

Thanks,

 

Ravi

 

6 Replies

  • Hi Ravi,

    The HTTP_RESPOND event is triggered once TMM has collected enough packets to get the last HTTP header. If you call [HTTP::payload] in HTTP_RESPONSE, it will only return the HTTP payload available in the packet(s) required to parse all of the HTTP headers. If you want to collect the full HTTP payload, you'll need to use HTTP::collect:

     http://devcentral.f5.com/wiki/default.aspx/iRules/http__collect
    when HTTP_RESPONSE {
    
       Trigger collection for up to 1MB of data
      if {[HTTP::header exists "Content-Length"] && [HTTP::header "Content-Length"] <= 1048576}{
        set content_length [HTTP::header "Content-Length"]
      } else {
          set content_length 1048576
      }
       Check if $content_length has been set and is not set to 0
      if { $content_length > 0} {
        HTTP::collect $content_length
      }
    }
    when HTTP_RESPONSE_DATA {
       do stuff with the payload
      set payload [HTTP::payload]
    }
    

    Aaron

  • Hi Aaron,

     

     

    Thanks for your response. I tried executing your above code but it do not do the required.

     

     

    Still the entire request is not being logged. I read the link on HTTP::collect. It saya that if we use it with specifying any content_length arguement to it it will print the entire payload. but that seems to hang my client.

     

     

     

     

    what i observere is the entire payload is chunked and sent. the above code prints the first chunk and exits of.

     

     

     

    thanks,

     

    Ravi

     

     

     

     

  • Hi Ravi,

    You can set the request HTTP version to 1.0 to prevent the server from sending chunked responses:

    
    when HTTP_REQUEST {
    
        Prevent the server from sending a compressed response
        remove the compression offerings from the client
       HTTP::header remove "Accept-Encoding"
    
        Don't allow response data to be chunked
       if { [HTTP::version] eq "1.1" } {
    
           Force downgrade to HTTP 1.0, but still allow keep-alive connections.
           Since HTTP 1.1 is keep-alive by default, and 1.0 isn't,
           we need make sure the headers reflect the keep-alive status.
    
           Check if this is a keep alive connection
          if { [HTTP::header is_keepalive] } {
    
              Replace the connection header value with "Keep-Alive"
             HTTP::header replace "Connection" "Keep-Alive"
          }
    
           Set server side request version to 1.0
           This forces the server to respond without chunking
          HTTP::version "1.0"
       }
    }
    

    Aaron
  • Hi Aaron,

     

     

    Below is the complete irule i have with your suggestions:

     

     

    when HTTP_REQUEST {

     

    if {[HTTP::header value Content-Type] contains "application/vnd.syncml+xml"}{

     

     

    Prevent the server from sending a compressed response

     

    remove the compression offerings from the client

     

    HTTP::header remove "Accept-Encoding"

     

     

    Don't allow response data to be chunked

     

    if { [HTTP::version] eq "1.1" } {

     

     

     

    Force downgrade to HTTP 1.0, but still allow keep-alive connections.

     

    Since HTTP 1.1 is keep-alive by default, and 1.0 isn't,

     

    we need make sure the headers reflect the keep-alive status.

     

     

    Check if this is a keep alive connection

     

    if { [HTTP::header is_keepalive] } {

     

     

    Replace the connection header value with "Keep-Alive"

     

    HTTP::header replace "Connection" "Keep-Alive"

     

    }

     

     

    Set server side request version to 1.0

     

    This forces the server to respond without chunking

     

    HTTP::version "1.0"

     

    }

     

     

    Trigger collection for up to 1MB of data

     

    if {[HTTP::header exists "Content-Length"] && [HTTP::header "Content-Length"] <= 1048576}{

     

    set content_length [HTTP::header "Content-Length"]

     

    }

     

     

    Check if $content_length has been set and is not set to 0

     

    if { [info exists content_length] && $content_length > 0} {

     

    HTTP::collect $content_length

     

    log local0. " content_length: $content_length"

     

    }

     

    }

     

    }

     

     

     

     

    when HTTP_REQUEST_DATA {

     

    do stuff with the payload set payload [HTTP::payload]

     

    log local0. " HTTP::payload: [HTTP::payload]"

     

    }

     

     

     

     

     

    But still i don't get the complete payload logged. please advice.

     

     

    Thanks,

     

    Ravi

     

     

     

     

     

     

     

  • Hi Ravi,

     

     

    Sorry, I completely missed that you're talking about the request payload. I don't know of a way to collect a chunked request payload. If you have control over the client, you could disable chunking.

     

     

    Anyone else have ideas? If not, I suggest you open a case with F5 Support and ask them to consider supporting a method to unchunk request payloads.

     

     

    Aaron
  • Hi,

     

     

    dont forget Syslog. If you try to log more than 1022 bytes, syslog will cut your message.

     

    Try to split and log your payload or configure your syslog to log more than 1014 bytes.

     

     

    Regards,

     

     

    Sören