stream + content-length + chunking

Question

Hi,&nbsp;
&nbsp;i have an xml application for which i'm using a stream profile to do hostname rewrites. as i've seen from deb's comments on a number of posts, the resulting variance in the payload size and the content-length can be overcome by using rechunking in the attached http profile. seemed great but...&nbsp;
&nbsp;  1) there appears to be all sorts of crap added after each chunk according to wireshark (stuff like "10e", "d" and other tiny bits of raw hex)
&nbsp;  2) the client app doesn't seem to like the chunking at all, possibly becasue of the above corrupting the xml, maybe just because it doesn't support it.&nbsp;
&nbsp;another option i saw was to change the content-length header manunally. this however causes the client software to hang until a timeout occurs, which is unacceptable.&nbsp;
&nbsp;essentially i really think i need the data to reach the client "correctly", without any hacks in the data (which i can't predict the contents of - so no idea how many replacements may occur). the only other option i can think of is to actually collect the full payload and do a regsub and ditch the stream profile, but this would presumably have a massive performance hit.&nbsp;
&nbsp;any takers?&nbsp;
&nbsp;Cheers&nbsp;
&nbsp;Chris

john_klemm_4418 · Answer

I may be able to help you out.  I had some issues with sharepoint and trying to stream.  show me your stream profile as it is now.  Also, make sure that you have streaming enabled under you vip.

chris_phillips · Answer

the streaming in itself is working fine, it's the fact that it changes the size of the payload i am having issues with.

deb_allen_18 · Answer

Hi Chris --&nbsp;
&nbsp;Stream profile is low-overhead, performing replacements on the fly mid-stream, but requires chunking because the resulting payload size cannot be known when the Content-Length header would be sent.&nbsp;
&nbsp;Collecting the entire payload to perform replacement so size may be calculated is high overhead but allows explicit Content-Length declaration in header.&nbsp;
&nbsp;You can certainly use regsub to manipulate the entire payload, but as you suspected, it will carry a considerable performance hit.  Search the forum &amp; codeshare for "regsub" + "::payload" and you'll find a number of relevant examples.&nbsp;
&nbsp;/deb&nbsp;&nbsp;

chris_phillips · Answer

Thanks for the replies, i've since got the HTTP_RESPONSE_DATA event firing after the HTTP::collect completes and the data is being rewritten correctly.&nbsp;
&nbsp;But... I guess there is no actual solution to what i'm asking then? just a case of having to live with whichever evil i think is the easiest to cope with, whether it's performance on the box or complexity of iRules..?&nbsp;
&nbsp;if so, that does kinda suck. I can totally appreciate developers wanting to add functionality whever they can, but it does leave a number of really great options which are incompatible with one another...

hooleylist · Answer

If I understand Deb's explanation of the process correctly, it's not a limitation in the rule functionality, but a limitation of the scenarios.&nbsp;
&nbsp;The stream profile is fast because it doesn't buffer all of the content.  As the full content isn't being collected before the substitution is done, you don't know what the content length will be after the substitution(s) are complete.  Therefore, you can't set the content-length.  To resolve this you need to use chunking.&nbsp;
&nbsp;If you want to avoid chunking and set the content-length correctly, then you have to buffer the full content.  This can be done with TCP/HTTP::collect, but uses more resources to do.&nbsp;
&nbsp;Aaron

Forum Discussion

stream + content-length + chunking

6 Replies

Recent Discussions

F5Access | MacOS Sonoma

Active- Active HA setup

syslog server connection

Cannot see floating MAC address outside of ESXi

Rewrite uri translation not working

Related Content

Setting up F5 Telemetry Streaming with Splunk Cloud

iRule [string range...] not chunking data properly

LTM HTTP Profile Option: Response Chunking

Secure your gRPC Bidirectional Streaming APIs with F5 NGINX

Automation Toolchain - Telemetry Streaming - Grafana StatsD Graphite