What is HTTP Part X - HTTP/2
In the penultimate article in this What is HTTP? series we covered iRules and local traffic policies and the power they can unleash on your HTTP traffic. To date in this series, the content primarily focuses on HTTP/1.1, as that is still the predominant industry standard. But make no mistake, HTTP/2 is here and here to stay, garnering 30% of all website traffic and climbing steadily. In this article, we’ll discuss the problems in HTTP/1.1 addressed in HTTP/2 and how BIG-IP supports the major update. What’s So Wrong withHTTP/1.1? It’s obviously a pretty good standard since it’s lasted as long as it has, right? So what’s the problem? Well, let’s set security aside for this article, since the HTTP/2 committee pretty much punted on it anyway, and let’s instead talk about performance. Keep in mind that the foundational constructs of the HTTP protocol come from the internet equivalent of the Jurassic age, where the primary function was to get and post text objects. As the functionality stretched from static sites to dynamic interactive and real-time applications, the underlying protocols didn’t change much to support this departure. That said, the two big issues with HTTP/1.1 as far as performance goes are repetitive meta data and head of line blocking.HTTP was designed to be stateless. As such, all applicable meta data is sent on every request and response, which adds from minimal to a grotesque amount of overhead. Head of Line Blocking For HTTP/1.1, this phenomenon occurs due to each request needs a completed response before a client can make another request. Browser hacks to get around this problem involved increasing the number of TCP connections allowed to each host from one to two and currently at six as you can see in the image below. More connections more objects, right? Well yeah, but you still deal with the overhead of all those connections, and as the number of objects per page continues to grow the scale doesn’t make sense. Other hacks on the server side include things like domain sharding, where you create the illusion of many hosts so the browser creates more connections. This still presents a scale problem eventually. Pipelining was a thing as well, allowing for parallel connections and the utopia of improved performance. But as it turns out, it was not a good thing at all, proving quite difficult to implement properly and brittle at that, resulting in a grand total of ZERO major browsers actually supporting it. Radical Departures - The Big Changes in HTTP/2 HTTP/2 still has the same semantics as HTTP/1. It still has request/response, headers in key/value format, a body, etc. And the great thing for clients is the browser handles the wire protocols, so there are no compatibility issues on that front. There are many improvements and feature enhancements in the HTTP/2 spec, but we’ll focus here on a few of the major changes. John recorded a Lightboard Lesson a while back on HTTP/2 with an overview of more of the features not covered here. From Text to Binary With HTTP/2 comes a new binary framing layer, doing away with the text-based roots of HTTP. As I said, the semantics of HTTP are unchanged, but the way they are encapsulated and transferred between client and server changes significantly. Instead of a text message with headers and body in tow, there are clear delineations for headers and data, transferred in isolated binary-encoded frames (photo courtesy of Google). Client and server need to understand this new wire format in order to exchange messages, but the applications need not change to utilize the core HTTP/2 changes. For backwards compatibility, all client connections begin as HTTP/1 requests with an upgrade header indicating to the server that HTTP/2 is possible. If the server can handle it, a 101 response to switch protocols is issued by the server, and if it can’t the header is simply ignored and the interaction will remain on HTTP/1. You’ll note in the picture above that TLS is optional, and while that’s true to the letter of the RFC law (see my punting on security comment earlier) the major browsers have not implemented that as optional, so if you want to use HTTP/2, you’ll most likely need to do it with encryption. Multiplexed Streams HTTP/2 solves the HTTP/1.1 head of line problem by multiplexing requests over a single TCP connection. This allows clients to make multiple requests of the server without requiring a response to earlier requests. Responses can arrive in any order as the streams all have identifiers (photo courtesy of Google). Compare the image below of an HTTP/2 request to the one from the HTTP/1.1 section above. Notice two things: 1) the reduction of TCP connections from six to one and 2) the concurrency of all the objects being requested. In the brief video below, I toggle back and forth between HTTP/1.1 and HTTP/2 requests at increasing latencies, thanks to a demo tool on golang.org, and show the associated reductions in page load experience as a result. Even at very low latency there is an incredible efficiency in making the switch to HTTP/2. This one change obviates the need for many of the hacks in place for HTTP/1.1 deployments. One thing to note on the head of line blocking: TCP actually becomes a stumbling block for HTTP/2 due to its congestion control algorithms. If there is any packet loss in the TCP connection, the retransmit has to be processed before any of the other streams are managed, effectively halting all traffic on that connection. Protocols like QUIC are being developed to ride the UDP waveand overcome some of the limitations in TCP holding back even better performance in HTTP/2. Header Compression Given that headers and data are now isolated by frame types, the headers can now be compressed independently, and there is a new compression utility specifically for this called HPACK. This occurs at the connection level. The improvements are two-fold. First, the header fields are encoded using Huffman coding thus reducing their transfer size. Second, the client and server maintain a table of previous headers that is indexed. This table has static entries that are pre-defined on common HTTP headers, and dynamic entries added as headers are seen. Once dynamic entries are present in the table, the index for that dynamic entry will be passed instead of the head values themselves (photo courtesy of amphinicy.com). BIG-IP Support F5 introduced the HTTP/2 profile in 11.6 as an early access, but it hit general availability in 12.0. The BIG-IP implementation supports HTTP/2 as a gateway, meaning that all your clients can interact with the BIG-IP over HTTP/2, but server-side traffic remains HTTP/1.1. Applying the profile also requires the HTTP and clientssl profiles. If using the GUI to configure the virtual server, the HTTP/2 Profile field will be grayed out until use select an HTTP profile. It will let you try to save it at that point even without a clientssl profile, but will complain when saving: 01070734:3: Configuration error: In Virtual Server (/Common/h2testvip) http2 specified activation mode requires a client ssl profile As far as the profile itself is concerned, the fields available for configuration are shown in the image below. Most of the fields are pretty self explanatory, but I’ll discuss a few of them briefly. Insert Header - this field allows you to configure a header to inform the HTTP/1.1 server on the back end that the front-end connection is HTTP/2. Activation Modes - The options here are to restrict modes toALPN only, which would then allow HTTP/1.1 or negatiate to HTTP/2 or Always, which tells BIG-IP that all connections will be HTTP/2. Receive Window - We didn’t cover the flow control functionality in HTTP/2, but this setting sets the level (HTTP/2 v3+) where individual streams can be stalled. Write Size - This is the size of the data frames in bytes that HTTP/2 will send in a single write operation. Larger size will improve network utilization at the expense of an increased buffer of the data. Header Table Size - This is the size of the indexed static/dynamic table that HPACK uses for header compression. Larger table size will improve compression, but at the expense of memory. In this article, we covered the basics of the major benefits of HTTP/2. There are more optimizations and features to explore, such as server push, which is not yet supported by BIG-IP. You can read about many of those features here on this very excellent article on Google’s developers portal where some of the images in this article came from.
ICAP with iRule Response Page without ASM
Hello, We are running on Big IP 13.1.1.4 TMOS code and set up Content Adaptation for HTTP request to check files uploaded through one our Website using ICAP. It's working fine but in case any virus is detected the ICAP server modify the response and show it's own response. But we would like to redirect the end-user to a dedicated and corporate web page of our website. I prepared the below Irule but it's now working. when ADAPT_REQUEST_RESULT { if { ([ADAPT::result] contains "respond") } { log local0. "ICAP Response is [ADAPT::result], let's customized reject page" set response { <html> <head> <title>Virus Detected</title> <meta http-equiv="refresh" content="0;URL='https://int-www-01.citizensfla.com/virus-test'" /> </head> </html> } HTTP::header remove Content-Length #HTTP::payload replace 0 [HTTP::payload length] "" HTTP::payload replace 0 0 $response } } How we could redirect the POST of the user to a dedicated page within our website if a virus is found using ICAP internal VS. Many thanks in advance for any help on this matter. Regards Vijay
Rewrite http:// to https:// in response content
Problem this snippet solves: (Maybe I missed it, but) I didn't see a code share for using a STREAM profile to rewrite content from http to https. This share is just to make it easier to find a simple iRule to replace http:// links in page content to https://. It's taken directly from the STREAM::expression Wiki page. How to use this snippet: You'll need to assign a STREAM profile to you virtual server in order for this to work (just create an empty stream profile and assign it). Code : # Example which replaces http:// with https:// in response content # Prevents server compression in responses when HTTP_REQUEST { # Disable the stream filter for all requests STREAM::disable # LTM does not uncompress response content, so if the server has compression enabled # and it cannot be disabled on the server, we can prevent the server from # sending a compressed response by removing the compression offerings from the client HTTP::header remove "Accept-Encoding" } when HTTP_RESPONSE { # Check if response type is text if {[HTTP::header value Content-Type] contains "text"}{ # Replace http:// with https:// STREAM::expression {@http://@https://@} # Enable the stream filter for this response only STREAM::enable } } Tested this on version: 11.5Persist On Last JSESSIONID in HTTP Parameter
Problem this snippet solves: This iRule was written with the goal to persist on only the last jsessionid that is present when multiple jsessionid HTTP Paremeters are present.. How to use this snippet: I ran into a particular challenge where a customer was receiving a request where a jsessionid was expected to be set in an HTTP Request Parameter. For those of you who are not familiar, see where the parameters lye below: <scheme>://<username>:<password>@<host>:<port>/<path>;<parameters>?<query>#<fragment> REF: http://www.skorks.com/2010/05/what-every-developer-should-know-about-urls/ Here is an example of a full request: http://www.example.com/my/resource.html;jsessionid=0123456789abcdef;jsessionid=0123456789abcdef?alice=mouse In the above example, there are two jsessionid parameters, both of which are the same value. This is what was being used to Persist "*jsessionid*" { #Parse the URI and look for jsessionid. Skip 11 characters and match up to the next "?" set session_id [findstr [HTTP::uri] jsessionid 11 "?"] } Code : when CLIENT_ACCEPTED { set debug 1 } when HTTP_REQUEST { set logTuple "[IP::client_addr]:[TCP::client_port] - [IP::local_addr]:[TCP::local_port]" set parameters [findstr [HTTP::path] ";" 1] set session_id "" foreach parameter [split $parameters ";"] { scan $parameter {%[^=]=%s} name value if {$debug}{log local0.debug "$logTuple :: Multiple JsessionID in: [HTTP::host][HTTP::uri]"} if {$name equals "jsessionid"} {set session_id $value} } if {$session_id ne ""}{ #Persist on the parsed session ID for X seconds if {$debug}{log local0.debug "$logTuple :: Single JsessionID in: [HTTP::host][HTTP::uri]"} persist uie $session_id 86400 } }
Tightening the Security of HTTP Traffic part 1
Summary HTTP is the de facto protocol used to communicate over the internet. The security challenges that exists today was not considered when designing the HTTP protocol. Web browsers are generally used to interact with web applications. However, this interaction is not secured in most deployments, despite the existence of a number of http headers that can be leveraged by application developers to tighten the security of their applications. In this article, I will give an overview of some important headers that can be added to HTTP responses in order to improve the security web applications. I will also provide sample iRules that can be implemented on F5 BigIP to insert those headers in HTTP responses. Content This article has been made in a series of 3 parts to make it easier to read and digest. The following headers will be reviewed in different parts of this article: Part 1: A brief overview of Cross-Site Scripting (XSS) vulnerability X-XSS-Protection HttpOnly flag for cookies Secure flag for cookies Part 2: The X-Frame Options Header HTTP Strict Transport Security X-Content-Type-Options Content-Security-Policy Public Key Pinning Extension for HTTP (HPKP) Part 3: Server and X-Powered-By Cookie encryption Https (SSL/TLS) A brief overview of XSS Attacks Before going further into the details of the headers we need to implement to improve the security of http traffic, it is important to make a quick review of the Cross-Site Scripting (XSS) attack,which is the most prevalent web application security flaw . This is important because any of the security measure we will review may be completely useless if an XSS flaw exist on the application.The security measures proposed in this article rely on the browsernot infected by any kind of malware (including XSS) and working properly. A Cross-Site Scripting (XSS) attack is a type of injection attack where a malicious script is injected into a trusted but vulnerable web applications. The malicious script is send to victim browser as client side script when the user uses a vulnerable web app. XSS are generally caused by poor input validation or encoding by web application developers. An XSS attack can load a malware (JavaScript) into to browser and make it behave in a completely unpredictable manner, thus making all other protection mechanism useless. Cross-Site scripting was ranked third in the last OWASP 2013 top 10 security vulnerabilities. Detailed information on XSS can be found on OWASP website. 1. TheX-XSS-Protection header The X-XSS-Protection header enables the Cross-Site Scripting filter on the browser. When enabled, the XSS Filter operates as a browser component with visibility into all requests / responses flowing through the browser.When the filter detects a likely XSS in a request or response, it prevents the malicious script from executing. The X-XSS-Protection header is supported by most major browsers. Usage: X-XSS-Protection: 0 XSS protection filter is disabled (facebook) X-XSS-Protection: 1 XSS protection filter is enable. Upon XSS attack detection, the browser will sanitize the page. X-XSS-Protection: 1; mode=block If XSS attack is detected, The browser blocks the page in order to stop the attack. (Google, Twitter) Note: If third party scripts are allowed to run on your web application, enabling XSS-Protection might prevents some scripts from working properly. As such, proper considerations should be made before enabling the header. 1.1 Example of how this header is used: the following command can be used to see which http headers are enabled on a web application. This command sends a single http request to the destination application and follows redirection. Therefore it is completely harmless and is only used for educational purposes in this article. I will use the same commandthroughout the article against somes well known sites, that are build with high security standards, in order to demonstrate how the discussed headers are used in real life. curl -L -A "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0" -sD - https://www.google.com -o /dev/null HTTP/1.1 200 OK Date: Mon, 07 Aug 2017 11:24:08 GMT Expires: -1 Cache-Control: private, max-age=0 Server: gws X-XSS-Protection: 1; mode=block [...] 1.2 iRule to enableXSS-Protection The following iRule can be used to insertXSS-Protection header to all http responses: when HTTP_RESPONSE { if { !([ HTTP::header exists "X-XSS-Protection“ ])} { HTTP::header insert "X-XSS-Protection" "1; mode=block" } } 2.HttpOnly flag for cookies When added to cookies in HTTP responses, the HttpOnly flag prevents client side scripts from accessing cookies. Existing cross-site scripting flaws on the web applicationwill not be exploited by using this cookie. The HttpOnly flag relies on the browser to prevent XSS attacks, when set by the server.This flag is supported by all major browsers in their latest versions. It was Introduced in 2002 by Microsoft in IE6 SP1. Note: Using older browser exposes you to higher security risk as older browsers may not support some of the security headers discussed in this article. 2.1 Example: curl -L -A "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0" -sD - https://www.google.com -o /dev/null HTTP/1.1 200 OK Date: Wed, 16 Aug 2017 19:34:36 GMT Expires: -1 [..] Server: gws X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN Set-Cookie: NID=110=rKlFElgqL3YuhyKtxcIr0PiSE; expires=Thu, 15-Feb-2018 19:34:36 GMT; path=/; domain=.google.com; HttpOnly Alt-Svc: quic=":443"; ma=2592000; v="39,38,37,35" [...] 2.2 iRule to add HttpOnly flag to all cookies for http responses: If added to Big-IP LTM virtual server, the following iRule will add the HttpOnly flag to all cookies in http responses. when HTTP_RESPONSE { foreach mycookie [HTTP::cookie names] { HTTP::cookie httponly $mycookie enable } } 3.Secure flag for cookies The purpose of the secure flag is to prevent cookies from being observed by unauthorized parties due to the transmission of a the cookie in clear text. When a cookie has the secure flag attribute, Browsers that support the secure flag will only send this cookie within a secure session(HTTPS). This means that, if the web application accidentally points to a hard coded http link, the cookie will not be send. Enabling secure flag for cookies is common and recommended 3.1 Example curl -L -A "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0" -s -D - https://www.f5.com -o /dev/null HTTP/1.1 200 OK Cache-Control: private Content-Type: text/html; charset=utf-8 […] Set-Cookie: BIGipServer=!M+eTuU0mHEu4R8mBiet1HZsOy/41nDC5VnWBAlE5bvU0446qCYN4w/jKbf2+U8d8EVe+BxHFZ5+UJYg=; path=/; Httponly;Secure Strict-Transport-Security: max-age=16070400 […] 3.2 Setting the secure flag with iRule If added to Big-IP LTM virtual server, the following iRule will add the secure flag to all http responses. when HTTP_RESPONSE { foreach mycookie [HTTP::cookie names] { HTTP::cookie secure $mycookie enable } }Clacks-Over-HTTP Gateway
Problem this snippet solves: This iRule implements draft-1 of the Clacks-over-HTTP protocol specification as developed by the Clacks Overhead WG. This is a partial implementation of the Internet of Roundworld portion of draft-1; it obeys the Accept-Clacks header in requests, but since all clients must accept Plain it ignores any Clacks-Encoding headers. Additionally, it is client-side only. Communication with Discworld is not yet implemented, as current releases of iRules do not include a computational demonology extension. Instead, a fixed list of overhead values is included. By default this is the names of Terry Pratchett and John Dearheart, coded to remain in the Clacks overhead forever (GNU). Posted in memory of Sir Terry Pratchett, OBE. This IS an April 1 submission and should be taken with the implied grain of salt. That said, the iRule is functional and is a compliant implementation. How to use this snippet: Add to any virtual server with an HTTP profile. Header values can be modified by changing static::clacks. The format is as follows: set static::clacks "{message 1} {message 2}" - each message must be wrapped in braces, then added to the quoted string. It is recommended that memorial messages start with GNU to ensure proper return to the Clacks networks of Discworld via gateway Towers and continued transmission. Code : when RULE_INIT { # Man's not dead while his name is still spoken. set static::clacks "{GNU Terry Pratchett} {GNU John Dearheart}" } when CLIENT_ACCEPTED { set clacks_enabled 1 } when HTTP_REQUEST { # Per spec, clients can refuse all Clacks responses with an Accept-Clacks header of "no". if {[HTTP::header Accept-Clacks] == "no"} { set clacks_enabled 0 } } when HTTP_RESPONSE { if {$clacks_enabled} { foreach clack $static::clacks { HTTP::header insert Clacks $clack } } }
LineRate HTTP to HTTPS redirect
Here's a quick LineRate proxy code snippet to convert an HTTP request to a HTTPS request using the embedded Node.js engine. The relevant parts of the LineRate proxy config are below, as well. By modifying the redirect_domain variable, you can redirect HTTP to HTTPS as well as doing a non-www to a www redirect. For example, you can redirect a request for http://example.com to https://www.example.com . The original URI is simply appended to the redirected request, so a request for http://example.com/page1.html will be redirected to https://www.example.com/page1.html . This example uses the self-signed SSL certificate that is included in the LineRate distribution. This is fine for testing, but make sure to create a new SSL profile with your site certificate and key when going to production. As always, the scripting docs can be found here. redirect.js: Put this script in the default scripts directory - /home/linerate/data/scripting/proxy/ and update the redirect_domain and redirect_type variables for your environment. "use strict"; var vsm = require('lrs/virtualServerModule'); // domain name to which to redirect var redirect_domain = 'www.example.com'; // type of redirect. 301 = temporary, 302 = permanent var redirect_type = 302; vsm.on('exist', 'vs_example.com', function(vs) { console.log('Redirect script installed on Virtual Server: ' + vs.id); vs.on('request', function(servReq, servResp, cliReq) { servResp.writeHead(redirect_type, { 'Location': 'https://' + redirect_domain + servReq.url }); servResp.end(); }); }); LineRate config: real-server rs1 ip address 10.1.2.100 80 admin-status online ! virtual-ip vip_example.com ip address 192.0.2.1 80 admin-status online ! virtual-ip vip_example.com_https ip address 192.0.2.1 443 attach ssl profile self-signed admin-status online ! virtual-server vs_example.com attach virtual-ip vip_example.com default attach real-server rs1 ! virtual-server vs_example.com_https attach virtual-ip vip_example.com_https default attach real-server rs1 ! script redirect source file "proxy/redirect.js" admin-status online Example: user@m1:~/ > curl -L -k -D - http://example.com/test HTTP/1.1 302 Found Location: https://www.example.com/test Date: Wed, 03-Sep-2014 16:39:53 GMT Transfer-Encoding: chunked HTTP/1.1 200 OK Content-Type: text/plain Date: Wed, 03-Sep-2014 16:39:53 GMT Transfer-Encoding: chunked hello worldWhat is HTTP Part VIII - Compression and Caching
In the last article of this What is HTTP? series we covered the nuances of OneConnect on HTTP traffic through the BIG-IP. In this article, we’ll cover caching and compression. We’ll deal with compression first, and then move on to caching. Compression In the very early days of the internet, much of the content was text based. This meant that the majority of resources were very small in nature. As popularity grew, the desire for more rich content filled with images grew as well, and resource sized began to explode. What had not yet exploded yet, however, was the bandwidth available to handle all that rich content (and you could argue that’s still the case in mobile and remote terrestrial networks as well.) This intersection of more resources without more bandwidth led to HTTP development in a few different areas: Methods for getting or sending partial resources Methods for identifying if resources needed to be retrieved at all Methods for reducing resources during transit that could be successfully reproduced after receipt The various rangeheaders were developed to handle the first case, caching, which we will discuss later in this article, was developed to handle the second case, and compression was developed to handle the third case. The basic definition of data compression is simply reducing the bits necessary to accurately represent the resource. This is done not only to save network bandwidth, but also on storage devices to save space. And of course money in both areas as well. In HTTP/1.0, end-to-end compression was possible, but not hop-by-hop as it does not have a distinguishing mechanism between the two. That is addressed in HTTP/1.1, so intermediaries can use complex algorithms unknown to the server or client to compress data between them and translate accordingly when speaking to the clients and servers respectively. In 11.x forward, compression is managed in its own profile. Prior to 11.x, it was included in the http profile. The httpcompression profile overview on AskF5 is very thorough, so I won’t repeat that information here, but you will want to pay attention to the compression level if you are using gzip (default.) The default of level 1 is fast from the perspective of the act of compressing on BIG-IP, but having done minimal compressing, reaps the least amount of benefit on the wire. If a particular application has great need for less bandwidth utilization toward the clientside of the network footprint, bumping up to level 6 will increase the reduction in bandwidth without overly taxing the BIG-IP to perform the operation. Also, it’s best to avoid compressing data that has already been compressed, like images and pdfs. Compressing them actually makes the resource larger, and wastes BIG-IP resources doing it! SVG format would be an exception to that rule. Also, don’t compress small files. The profile default is 1M for minimum content length. For BIG-IP hardware platforms, compression can be performed in hardware to offload that function. There is a database variable that you can configure to select the data compression strategy via sys modify db compression.strategy . The default value is latency, but there are four other strategies you can employ as covered in the manual. Caching Web caching could (and probably should) be its own multi-part series. The complexities are numerous, and the details plentiful. We did a series called Project Acceleration that covered some of the TCP optimization and compression topics, as well as the larger product we used to call Web Accelerator but is now the Application Acceleration Manager or AAM. AAM is caching and application optimization on steroids and we are not going to dive that deep here. We are going to focus specifically on HTTP caching and how the default functionality of the ramcache works on the BIG-IP. Consider the situation where there is no caching: In this scenario, every request from the browser communicates with the web server, no matter how infrequently the content changes. This is a wasteful use of resources on the server, the network, and even the client itself. The most important resource to our short attention span end users is time! The more objects and distance from the server, the longer the end user waits for that page to render. One way to help is to allow local caching in the browser: This way, the first request will hit the web server, and repeat requests for that same resource will be pulled from the cache (assuming the resource is still valid, more on that below.) Finally, there is the intermediary cache. This can live immediately in front of the end users like in an enterprise LAN, in a content distribution network, immediately in front of the servers in a datacenter, or all of the above! In this case, the browser1 client requests an object not yet in the cache serving all the browser clients shown. Once the cache has the object from the server, it will serve it to all the browser clients, which offloads the requests to server, saves the time in doing so, and brings the response closer to the browser clients as well. Given the benefits of a caching solution, let’s talk briefly of the risks. If you take the control of what’s served away from the server and put it in the hands of an intermediary, especially an intermediary the administrators of the origin server might not have authority over, how do you control then what content the browsers ultimately are loading? That’s where the HTTP standards on caching control come into play. HTTP/1.0 introduced the Pragma, If-Modified-Since, Last-Modified, and Expires headers for cache control. The Cache-Control and ETag headers along with a slew of “If-“ conditional headers were introduced in HTTP/1.1, but you will see many of the HTTP/1.0 cache headers in responses alongside the HTTP/1.1 headers for backwards compatibility. Rather than try to cover the breadth of caching here, I’ll leave it to the reader to dig into the quite good resources linked at the bottom (start with "Things Caches Do") for detailed understanding. However, there's a lot to glean from your browser developer tools and tools like Fiddler and HttpWatch. Consider this request from my browser for the globe-sm.svg file on f5.com. Near the bottom of the image, I’ve highlighted the request Cache-Control header, which has a value of no-cache. This isn’t a very intuitive name, but what the client is directing the cache is that it must submit the request to the origin server every time, even if the content is fresh. This assures authentication is respected while still allowing for the cache to be utilized for content delivery. In the response, the Cache-Control header has two values: public and max-age. The max-age here is quite large, so this is obviously an asset that is not expected to change much. The public directive means the resource can be stored in a shared cache. Now that we have a basic idea what caching is, how does the BIG-IP handle it? The basic caching available in LTM is handled in the same profile that AAM uses, but there are some features missing when AAM is not provisioned. It used to be called ramcache, but now is the webacceleration profile. Solution K14903 provides the overview of the webacceleration profile but we’ll discuss the cache size briefly. Unlike the Web Accelerator, there is no disk associated with the ramcache. As the name implies, this is “hot” cache in memory. So if you are memory limited on your BIG-IP, 100MB might be a little too large to keep locally. Managing the items in cache can be done via the tmsh command line with the ltm profile ramcache command. tmsh show/delete operations can be used against this method. An example show on my local test box: root@(ltm3)(cfg-sync Standalone)(Active)(/Common)(tmos)# show ltm profile ramcache webacceleration Ltm::Ramcaches /Common/webacceleration Host: 192.168.102.62 URI : / -------------------------------------- Source Slot/TMM 1/1 Owner Slot/TMM 1/1 Rank 1 Size (bytes) 3545 Hits 5 Received 2017-11-30 22:16:47 Last Sent 2017-11-30 22:56:33 Expires 2017-11-30 23:16:47 Vary Type encoding Vary Count 1 Vary User Agent none Vary Encoding gzip,deflate Again, if you have AAM licensed, you can provision it and then additional fields will be shown in the webacceleration profile above to allow for an acceleration policy to be applied against your virtual server. Resources RFC 2616 - The standard fine print. Things Caches Do- Excellent napkin diagrams that provide simple explanations of caching operations. Caching Tutorial - Comprehensive walk through of caching. HTTP Caching - Brief but informative look at caching from a webdev perspective. HTTP Caching - Google develops page with examples, flowcharts, and advice on caching strategies. Project Acceleration - Our 10 part series on web acceleration technology available on the BIG-IP platform in LTM and/or AAM modules. Solution K5157 - BIG-IP caching and the Vary header Make Your Cache Work For You - Article by Dawn Parzych here on DevCentral on tuning techniquesWhat is HTTP Part IX - Policies and iRules
In the previous article of this What is HTTP? series we covered optimization techniques in HTTP with compression and caching. In this article we pivot slightly from describing the protocol itself to showing ways to interact with the HTTP protocol on BIG-IP: policies and iRules. There is a plethora of content on DevCentral for how and what policies are, so rather than regurgitate that here, we’ll focus on what you can do with each approach and show a couple common examples. Before we do, however, run (don’t walk!) over to Chase Abbott’s To iRule, or not to iRule: Introduction to Local Traffic Policies article and Steve McCarthy’s LTM Policy article and read those first. Seriously, stop here and read those articles! Example 1: Simple HTTP Redirect I haven’t performed any advanced analysis on our Q&A section in quite a while, but at one time, redirection held the lion’s share of questions, and I still see them trickle in. Sometimes an application owner will move resources from one location to another within the app, either because the underlying platform has changed or navigation or site map redesign has occurred. Alternatively, the app owner may just want some vanity or random URLs to surface in advertisements so it’s easier to market and track demographically. To reorient application clients to the new resource locations, you can simply rewrite the URL before sending to the server, which will be unnoticed by the client, or you can redirect them to the new resource, which results in an HTTP response to the client and a second request for BIG-IP to handle the request for the “right” resource. The 300 level status codes are used by the “server" to instruct the client on necessary additional actions. The two primary redirection codes are 301, which instructs the client that the resource has been moved permanently, and the 302, which instructs the client that the move is temporary. For this example, we’re going to create a vanity URL to temporarily redirect clients from /http to this series landing page at/articles/sid/8311. Policy Solution Each policy has one or more rules. In this policy, we just need the one rule that has the condition of /http as the path and the action of redirecting a matching condition to the specified URL. ltm policy simple_redirect { controls { forwarding } last-modified 2017-12-14:23:22:51 requires { http } rules { vanity_http { actions { 0 { http-reply redirect location https://devcentral.f5.com/s/articles/sid/8311 } } conditions { 0 { http-uri path values { /http } } } } } status published strategy first-match } Writing policies textually can be a little overwhelming, but you can also configure them in the GUI as shown below. Note that in version 12.1 and later, you need to publish a policy before it can be used, and altering policies requires making a draft copy before activating any changes. iRules Solution The iRule solution is actually quite short. when HTTP_REQUEST { if { [HTTP::path] eq "/http" } { HTTP::redirect "https://devcentral.f5.com/s/articles/sid/8311" } } While short and simple, it is not likely to be as performant as the policy. However, since it is a very simple iRule with little overhead, the performance savings might not justify the complexity of using policies, especially if the work is being done on the command line. Also of note, if the requirement is a permanent redirect, then you must use an iRule because you cannot force a 301 from a policy alone. With a permanent redirect, you have to change the the HTTP::redirect command to an HTTP::respond and specify the 301 with a location header. when HTTP_REQUEST { if { [HTTP::path] eq "/http" } { HTTP::respond 301 Location "https://devcentral.f5.com/s/articles/sid/8311" } } There is a way to use a policy/iRule hybrid for 301 redirection, but if you need the iRule from the policy, I'd recommend just using the iRule approach. Example 2: Replace X-Forwarded-For with Client IP Address Malicious users might try to forge client IPs in the X-Forwarded-For (XFF) header, which can potentially lead to bad things. In this example, we’ll strip any XFF header in the client request and insert a new one. Policy Solution In this policy, we don't care what the conditions are, we want a sweeping decision to replace the XFF header, so no condition is necessary, just an action. You'll notice the tcl: immediately before using the iRules [IP::client_addr] . These tcl expressions are supported in policies, and are a compromise of performance for additional flexibility. ltm policy xff { last-modified 2017-12-15:00:39:26 requires { http } rules { remove_replace { actions { 0 { http-header replace name X-Forwarded-For value tcl:[IP::client_addr] } } } } status published strategy first-match } This works great if you have a single XFF header, but what if you have multiples? Let's check wireshark on my curl command to insert two headers with IP addresses 172.16.31.2 and 172.16.31.3. You can see that one of the two XFF headers was replaced, but not both, which does not accomplish our goal. So instead, we need to do a remove and insert in the policy like below. ltm policy xff { last-modified 2017-12-15:00:35:35 requires { http } rules { remove_replace { actions { 0 { http-header remove name X-Forwarded-For } 1 { http-header insert name X-Forwarded-For value tcl:[IP::client_addr] } } } } status published strategy first-match } Removing and re-inserting, we were able to achieve our goal as shown in the wireshark output from the same two IP address inserted in XFF headers from a curl request. iRules Solution The iRule is much like the policy in that we’ll remove and insert. when HTTP_REQUEST { HTTP::header remove "X-Forwarded-For" HTTP::header insert "X-Forwarded-For" [IP::client_addr] # You could also use replace since it will insert if missing # HTTP::header replace "X-Forwarded-For" [IP::client_addr] } Again, short and sweet! Like the policy example above, using HTTP::header remove in the header command removes all occurrences of the XFF header, not just the last match, which is what HTTP::header replace does. Profile Solution This is where we step back a little and evaluate whether we need an iRule or policy at all.You might recall frompart five of this seriesthe settings in the HTTP profile for the X-Forwarded-For header. But with that, I noted that enabling this just adds another X-Forwarded-For header, it doesn’t replace the existing ones. But a slick little workaround (much like our policy solution above, but completely contained in the http profile!) is to add the header to the remove and insert fields, with the values set as shown below. Keep in mind you can only erase/insert a single header, so it works for this single solution, but as soon as you have more parameters the policy or iRule is the better option. Profiles, Policies, and iRules…Where’s My Point of Control? This isn’t an HTTP problem, but since we’re talking about solutions, I’d be remiss if I didn’t address it. Performance is a big concern with deployments, but not the only concern. One other big area that needs to be carefully planned is where your operational team is comfortable with points of control. If application logic is distributed among profiles, policies, and iRules, and that changes not just within a single app but across all applications, there is significant risk for resolving issues when they do arise if documentation of the solutions is lacking. It’s perfectly ok to spread the logic around, but everyone should be aware that XYZ is in iRules, ABC is in policies, and DEF is in profiles. Otherwise, your operations teams will spend a significant amount of time just mapping out what belongs where so they can understand the application flow within the BIG-IP! I say all that to drive home the point that all iRules is an ok solution if the needs policies address is a small percentage of the overall application services configured on BIG-IP. It’s also ok to standardize on policies and push back on application teams to move the issues policies can’t address back into the application itself. The great thing is you have options. Additional Resources Getting Started with iRules iRule Recipe 1: Single URL Explicit Redirect Codeshare: iRules & HTTP Getting Started with Policies (AskF5 Manual) LTM Policy Recipes I LTM Policy Recipes II Automatically Redirect http to https on a Virtual Server(Ask F5 Solution) Forwarded HTTP Extension Insertion - RFC 7239 (Codeshare iRule)
