Sporadic 404 errors - where is this coming from?

The 404 pages are surely coming from the backend servers because F5 doesn't use jetty. Can you confirm what F5 is getting when you request the url from the CLI using curl:

curl http://member_address/uri

Try with every pool member.

I agree with Amine, there seems to be a problem with one ore more of the backend servers. You could use one of the iRules in this post to do some more troubleshooting on why en when the 404 is happening.

See: https://devcentral.f5.com/questions/irule-to-catch-4xx-and-5xx-http-error-codes

You can try a simple iRule to log the 404 responses. Usually, 404 indicates a broken link or a old link that hasn't been transitioned over. Reference link.

when HTTP_RESPONSE {
  if { [HTTP::status] == 404 } {
   log local0. "============================================="
   log local0. "$LogString (response) - status: [HTTP::status]"
   foreach aHeader [HTTP::header names] {
      log local0. "$aHeader: [HTTP::header value $aHeader]"
   }
   log local0. "=============================================" 
   }
}

If powered by Jetty it appears you are running into issues with the number of sessions. I would check the parameters for it, not familiar with it but there should be a httpd.conf or something of the sort where you can tinker with the TTL and max sessions.

So basically, the core of the issue was that there were sporadic double GET requests from the source that showed up in the same TCP stream, according to F5 support, which we also saw in our logs here. It was an issue for our Developers to look at since it appears it came into the F5 that way. The F5 VIP and traffic policy looks at the TCP stream on the back end and uses a "1st match" policy ("the Big-IP inspect every HTTP request and in doing that we will see if it correctly sends every HTTP request to the correct pool", F5 support stated), so if a GET request for one URI is in the same TCP stream as the second, different URI, it may get load balanced improperly. OneConnect was recommended to use, and one of the benefits of OneConnect, they said, is that it forces the LTM to inspect every packet since it's reusing/maintaining server-side connections, which prevents this.

Additional details:

- OneConnect increases the system’s network connection efficiency by keeping server side connections open and re-using them for new client requests
- Connection pooling allows new client side requests coming into the system to utilize existing server side requests that are already formed
- This optimization feature not only saves system resources on the BIG-IP, but also on the servers.
- OneConnect profiles allow you to manipulate various settings, such as maximum number of times server side connections can be re-used, and the maximum age of those entries.