Log Bypass for BIG-IP Monitor Traffic

Monitors are a part of just about any infrastructure that needs to be highly available. BIG-IP has a whole host of different monitoring options for keeping tabs on servers. It's important for your BIG-IP to understand the current state of the servers behind it, the services on those servers, what to do in the event of a failure, etc. The most common monitor types are simple ICMP, TCP or HTTP requests to the individual servers behind the BIG-IP device that will give some semblance of confidence that those services are still intact.

While you could simply rely on ICMP responses to show that a server is up, in the case of a web server it is usually far preferred to send an actual HTTP request to the application in question. This to avoid the quite possible scenario where the server itself is up and responding to pings, but the app tier is failing for some reason or another. These monitors are easy to create, and as a matter of fact I'll walk you through how to do just that in a moment, but the real point of this article is to address one of the few, minor concerns when creating multiple monitors such as this - log entries.

In high traffic deployments logs tend to become near sacred as they're amazingly useful in determining failure cases, current system health, traffic patterns and more. With many pieces of network infrastructure all sending out monitoring queries you can quickly muddy the waters or even, in some cases, fill up your access logs with HTTP monitors from your BIG-IP. One way to avoid that issue is to pass a custom HTTP User-Agent header along with your monitoring traffic and tell your webserver to ignore that particular User-Agent. To put it more simply:

Problem: HTTP access logs filling up with BIG-IP monitors
Solution: Pass custom HTTP User-Agent header with monitor requests and bypass logging based on User-Agent match

So let's take a look at how to achieve just that. We'll use Apache as an example, but the same concept holds true regardless of your back end webserver, so long as you can configure it to ignore log entries based on the User-Agent header.

First, we'll need to create a custom HTTP monitor on the BIG-IP. This can be done via the UI, command line, iControl or an iApp, but we'll go through the UI steps here:

Create a custom HTTP monitor

• Create a new HTTP monitor (Local Traffic > Monitors > Create)
• Fill in the following fields:
  • Name: (an intuitive name)
  • Description: (fill in with any details you might need for reference)
  • Type: HTTP
  • Send String - This will likely be a get to your desired host or IP, with the appropriate user agent tagged on, such as:
    • GET / HTTP/1.1\r\nHost: www.mytestsite.local\r\nUser-Agent: F5 BIG-IP Monitor\r\n\r\n
  • Receive String: (a string located in a page, which qualifies the pool member as healthy)
• Click Finished

Now that you have a monitor created, next you'll need to assign that monitor to the pool in question:

Assign HTTP monitor to pool

• Navigate to the Local Traffic: Pool List (Local Traffic > Pools > Pool List)
• Locate the pool you would like to monitor and click on its Name
• Select the pool name from Available Health Monitors and use the << button to move it to the Active List
• Click Update

All right, now the BIG-IP side of things is taken care of. Your BIG-IP will now send along a custom header with this particular monitor letting any webserver it hits know that it is an F5 BIG-IP Monitor. That's great and all, but without the other half of the equation, namely telling your web server to ignore those entries and not log them, it won't do you a lot of good. With that said, let's take a look at the necessary Apache config changes to make this happen.

Disable monitor logging for an Apache 2.x site

• Create a new site: vi /etc/apache/sites-available/default
• Locate the CustomLog directive in the configuration and add the following:

ErrorLog /var/log/apache2/error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

SetEnvIf User-Agent "^F5\ BIG-IP\ Monitor$" monitor

CustomLog /var/log/apache2/access.log combined env=!monitor

• Save the file: :qx!
• Reload Apache: /etc/init.d/apache2 restart
• Tail logs and verify that monitors are no longer appearing: tail –f /var/log/apache/access.log

There you have it! You now have a fully functioning monitor that will not appear in your Apache access log to clutter things up. This time saver will allow you grep free perusal of your logs, disk savings, and likely a much easier time sorting through things in whatever monitoring suite you're using. Thanks go to George Watkins for the idea on this one. Hopefully it helps you head down the path of monitors built to suit your needs. Keep in mind there is a lot more that you can do with monitors, this is just a basic example of making life a bit easier.