Monitors are a part of just about any infrastructure that needs to be highly available. BIG-IP has a whole host of different monitoring options for keeping tabs on servers. It's important for your BIG-IP to understand the current state of the servers behind it, the services on those servers, what to do in the event of a failure, etc. The most common monitor types are simple ICMP, TCP or HTTP requests to the individual servers behind the BIG-IP device that will give some semblance of confidence that those services are still intact.

While you could simply rely on ICMP responses to show that a server is up, in the case of a web server it is usually far preferred to send an actual HTTP request to the application in question. This to avoid the quite possible scenario where the server itself is up and responding to pings, but the app tier is failing for some reason or another. These monitors are easy to create, and as a matter of fact I'll walk you through how to do just that in a moment, but the real point of this article is to address one of the few, minor concerns when creating multiple monitors such as this - log entries.

In high traffic deployments logs tend to become near sacred as they're amazingly useful in determining failure cases, current system health, traffic patterns and more. With many pieces of network infrastructure all sending out monitoring queries you can quickly muddy the waters or even, in some cases, fill up your access logs with HTTP monitors from your BIG-IP. One way to avoid that issue is to pass a custom HTTP User-Agent header along with your monitoring traffic and tell your webserver to ignore that particular User-Agent. To put it more simply:

Problem: HTTP access logs filling up with BIG-IP monitors
Solution: Pass custom HTTP User-Agent header with monitor requests and bypass logging based on User-Agent match

 

So let's take a look at how to achieve just that. We'll use Apache as an example, but the same concept holds true regardless of your back end webserver, so long as you can configure it to ignore log entries based on the User-Agent header.

 

First, we'll need to create a custom HTTP monitor on the BIG-IP. This can be done via the UI, command line, iControl or an iApp, but we'll go through the UI steps here:

 

Create a custom HTTP monitor

  • Create a new HTTP monitor (Local Traffic > Monitors > Create)
  • Fill in the following fields:
    • Name: (an intuitive name)
    • Description: (fill in with any details you might need for reference)
    • Type: HTTP
    • Send String - This will likely be a get to your desired host or IP, with the appropriate user agent tagged on, such as:
      • GET / HTTP/1.1\r\nHost: www.mytestsite.local\r\nUser-Agent: F5 BIG-IP Monitor\r\n\r\n
    • Receive String: (a string located in a page, which qualifies the pool member as healthy)
  • Click Finished

image

 

 

Now that you have a monitor created, next you'll need to assign that monitor to the pool in question:


Assign HTTP monitor to pool

  • Navigate to the Local Traffic: Pool List (Local Traffic > Pools > Pool List)
  • Locate the pool you would like to monitor and click on its Name
  • Select the pool name from Available Health Monitors and use the << button to move it to the Active List
  • Click Update

All right, now the BIG-IP side of things is taken care of. Your BIG-IP will now send along a custom header with this particular monitor letting any webserver it hits know that it is an F5 BIG-IP Monitor. That's great and all, but without the other half of the equation, namely telling your web server to ignore those entries and not log them, it won't do you a lot of good. With that said, let's take a look at the necessary Apache config changes to make this happen.

 

image
Disable monitor logging for an Apache 2.x site

  • Create a new site: vi /etc/apache/sites-available/default
  • Locate the CustomLog directive in the configuration and add the following:

ErrorLog /var/log/apache2/error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

SetEnvIf User-Agent "^F5\ BIG-IP\ Monitor$" monitor

CustomLog /var/log/apache2/access.log combined env=!monitor

  • Save the file: :qx!
  • Reload Apache: /etc/init.d/apache2 restart
  • Tail logs and verify that monitors are no longer appearing: tail –f /var/log/apache/access.log

There you have it! You now have a fully functioning monitor that will not appear in your Apache access log to clutter things up. This time saver will allow you grep free perusal of your logs, disk savings, and likely a much easier time sorting through things in whatever monitoring suite you're using. Thanks go to George Watkins for the idea on this one. Hopefully it helps you head down the path of monitors built to suit your needs. Keep in mind there is a lot more that you can do with monitors, this is just a basic example of making life a bit easier.

 

Comments on this Article
Comment made 02-May-2012 by Arie 1968
Wouldn't this allow a hacker to run wild without a trace of his (or her?) evil deeds in the web logs?

It would seem that this solution requires a bit more tweaking before implementing it in a Production environment...
1
Comment made 03-May-2012 by George Watkins 473
In theory yes, but that would require someone to somehow be aware of this
monitoring string. If you are worried about this scenario, it is very easy
to block external requests with this User-Agent with an iRule:

when HTTP_REQUEST {
log local0. "[IP::client_addr]:[TCP::client_port]: User-Agent: [string
tolower [HTTP::header "User-Agent"]] requested [HTTP::host][HTTP::uri]"

if { [string tolower [HTTP::header "User-Agent"]] contains "F5 BIG-IP
Monitor"} {
log local0. "[IP::client_addr]:[TCP::client_port]: Rejected
request"
reject
}
}

You can also add this logic to the Apache server config. Since you've already set an environment variable for the monitoring User-Agent we can reuse that to enforce access. You'll want to locate the VirtualHost's Allow and Deny directives and change the order and parameters to block external monitoring requests:

Order Deny,Allow
Deny from env=monitor
# set subnet or addresses of BIG-IP self-IPs
allow from 10.40.0.0/16

Great question. Hope that helps!
0
Comment made 07-May-2012 by Nick T 187
Any idea how to do this with IIS?
0
Comment made 18-May-2012 by Colin Walker 3759
Actually Nick, I'm not sure. I've got way more experience with Apache than IIS, personally. I'll dig around a bit online and see what I can find though!

#Colin
0
Comment made 14-Mar-2013 by Kevin Leicht 28
I know it's been a long time since this was posted, but has anyone found a way to do this with IIS? It would great not to be filling up our web servers with clutter.
0
Comment made 14-Mar-2013 by Jason Rahm
IIS is all or nothing by default, but you can modify or create your own log module. Details here:

http://www.iis.net/learn/extensions/advanced-logging-module/advanced-logging-for-iis-custom-logging
0
Comment made 14-Mar-2013 by Kevin Leicht 28
Many thanks!
0
Comment made 04-Sep-2015 by dmgeurts 64
How about using "DNT: 1" in the header? Regarding the use by hackers. One could always use an iRule to disallow connections from clients using the browser string suggested above. Same goes for other header options like DNT.
0