Forum Discussion

GuyG's avatar
GuyG
Icon for Altocumulus rankAltocumulus
Dec 06, 2023

Reading chunked data in iRule

Hey, I have an iRule in which I read all the payloads in each request and response.

When there iRule receives chunked data, because there is no content-length I just use some big number as a limit on how much to read with HTTP::collect.

There problem is that by doing that I keep the api occupied while waiting for the EOF or to reach the limit in collect, and only then does it send the request/response forward, instead of each chunk reaching the other side when sent.

If I put a small limit to HTTP::collect I will miss out on more data as the payloads will be truncated.

Any way to read the whole chunked payload at the very end, or reading in a loop each chunk while also letting it reach the destination?

devops

9 Replies

Sort By
  • Lucas_Thompson's avatar
    Lucas_Thompson
    Icon for Employee rankEmployee
    Dec 06, 2023

    Sounds like a fun and challenging project!

    To do this the right way you'll have to read the byte count in the chunk header and collect that much OR a fixed value, rather than collecting a fixed value all the time. This way your loop can be sure to exit at the right time. The last line in a chunked response is always "0\r\n", and you can also test for that to make sure the response is completed. 

    One other gotcha here is that there are architectural oddities about how BIG-IP's HTTP profile works with HTTP chunk mode together with HTTP compression to coordinate traffic between the server-side and client-side halves of the flow. If you find something weird, try turning off chunk mode, or compression, or both. 

    One trick I've used successfully in the past is to avoid chunked mode altogether by using iRules to remove or modify the "accept-encoding" header so that the HTTP participants assume the other one doesn't support it. Clients and servers will usually fall back to normal Content-Length mode if instructed to. 

    • GuyG's avatar
      GuyG
      Icon for Altocumulus rankAltocumulus
      Dec 12, 2023

      Thanks for the reply.

      This is sort of what I'm trying to do.
      I have a simple iRule to start with where I'm trying to log the whole payload (all chunks combined) at the end of the response, while allowing the VS to send each chunk back to the client instead of buffering it.

       

       

       

      when HTTP_RESPONSE {
    log local0.debug "Start of HTTP_RESPONSE"
    set readSoFar 0
    catch { HTTP::collect 1 }
}

when HTTP_RESPONSE_DATA {
    log local0.debug "Start of HTTP_RESPONSE_DATA"

    set prev_readSoFar $readSoFar
    set readSoFar [HTTP::payload length]
    log local0.debug "readSoFar: |$readSoFar|, prev_readSoFar: |$prev_readSoFar|"
    
    if { $readSoFar == $prev_readSoFar } {
        log local0.debug "Whole body: |[HTTP::payload]|"
    } else {
        HTTP::release

        catch { HTTP::collect [expr {$readSoFar + 1}] }
    }
}

when HTTP_RESPONSE_RELEASE {
    log local0.debug "Start of HTTP_RESPONSE_RELEASE"
}

       

       

       

      I tried using HTTP::release to allow the VS to send each chunk to the client, then called HTTP::collect again to collect the next chunk, the iRule will stop calling collect when it stopped receiving chunks (that's the job of the if).
      But when I do this, HTTP_RESPONSE_RELEASE is only triggered once at the very end of the iRule, and it seems like it isn't triggered when I call HTTP::release.

      Logs:

       

       

       

      Rule /Common/xxxxx <HTTP_RESPONSE>: Start of HTTP_RESPONSE
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |6|, prev_readSoFar: |0|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |17|, prev_readSoFar: |6|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |43|, prev_readSoFar: |17|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |74|, prev_readSoFar: |43|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |116|, prev_readSoFar: |74|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |167|, prev_readSoFar: |116|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |183|, prev_readSoFar: |167|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |207|, prev_readSoFar: |183|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |686|, prev_readSoFar: |207|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |701|, prev_readSoFar: |686|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |712|, prev_readSoFar: |701|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |722|, prev_readSoFar: |712|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |727|, prev_readSoFar: |722|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Start of HTTP_RESPONSE_DATA
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: readSoFar: |727|, prev_readSoFar: |727|
Rule /Common/xxxxx <HTTP_RESPONSE_DATA>: Whole body: |1     6      {   14          "Status": {   19              "Code": "0",   24              "Desc": "交易成功",   2d              "TrxTime": "2023-12-05 03:04:01"   b          },   12          "Body": {   1d8              "key": "sajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvlsdkafnslkfnsdknfsalknvaslknfsajlfjsadfljsaflkajnvakvls"   a          }   6      }   5         0    |
Rule /Common/xxxxx <HTTP_RESPONSE_RELEASE>: Start of HTTP_RESPONSE_RELEASE

       

       

       

       Any idea why it seems like HTTP::release is not working? Did I misunderstand how to use it?

    • GuyG's avatar
      GuyG
      Icon for Altocumulus rankAltocumulus
      Dec 13, 2023
      Splitting my reply to a few messages, because it didn't let me reply saying I was "Post Flooding".
      Thanks for the reply.
      This is sort of what I'm trying to do.
      I have a simple iRule to start with where I'm trying to log the whole payload (all chunks combined) at the end of the response, while allowing the VS to send each chunk back to the client instead of buffering it.
      • GuyG's avatar
        GuyG
        Icon for Altocumulus rankAltocumulus
        Dec 13, 2023
        when HTTP_RESPONSE {
    log local0.debug "Start of HTTP_RESPONSE"
    set readSoFar 0
    catch { HTTP::collect 1 }
}

when HTTP_RESPONSE_DATA {
    log local0.debug "Start of HTTP_RESPONSE_DATA"

    set prev_readSoFar $readSoFar
    set readSoFar [HTTP::payload length]
    log local0.debug "readSoFar: |$readSoFar|, prev_readSoFar: |$prev_readSoFar|"
    
    if { $readSoFar == $prev_readSoFar } {
        log local0.debug "Whole body: |[HTTP::payload]|"
    } else {
        HTTP::release

        catch { HTTP::collect [expr {$readSoFar + 1}] }
    }
}

when HTTP_RESPONSE_RELEASE {
    log local0.debug "Start of HTTP_RESPONSE_RELEASE"
}