iRules to the Rescue

Why is my Internet so slow today?

Scene: Monday morning. Early. Too early to deal with this crap, to be sure

Drinking: Coffee.

Thinking: Why can't I login to the VPN? Why is my Internet so slow today? Is there a problem with the line? The access point? Need more coffee. Nope, still slow. Hmm...there's an e-mail from Don saying "everything's dogging". Wonder what's up with that?

When I say "slow" I mean "so slow I can't even maintain a session with the corporate VPN". The bandwidth usage is high, and there are suddenly thousands of connections on an internal web server that should never have that many connections. Not even when my blog ended up slashdotted did the connections ever get this high. While the intent is likely not a DoS attack, it's sure acting like it. I can't use the Internet and my server is about to explode. Not a good way to begin a Monday at all.

I get some more coffee and start checking log files and machines. Don runs a miniature wargaming site, maybe he posted something about his German army running ramshod over my Screaming Eagles Friday night. My WWII-era Airborne companies usually win, so maybe he was talking it up? Nope. There's traffic out there, but it's not enough to cause the slow down.

Okay, check the other sites. Hey, what's this? Connections coming in nearly every second, but they aren't requesting files on any of my sites. netstat says they connections are originating from ... httpd processes. Oh crap. Didn't expect to see that. Time to search the logs.

Okay, nothing in the logs. Doesn't look like a successful break in. Still, telnet shouldn't be running (even though the firewall doesn't allow telnet) so I turn it off and decide to update Apache, just in case there's an exploit I missed a couple patches ago.

Scene: half an hour later

Well that didn't help. Another cup of coffee is necessary. I decide to cut the virtual out of httpd.conf and see what happens. Hey, look, all the connections went away. Interesting. Especially as that virtual is the wildcard default for all domains. Let's put it back in. Ooh...connections, connections, connections. It almost looks like someone is using me as a proxy.

Cue light bulb above head and sound of bell ringing

Aha! That's it. A while back I was working with my brother on a resolution issue for his site and we tried setting up mod_proxy as a solution. It didn't work, but I failed to remove the entries from the httpd.conf file. I disable the proxy and suddenly ... well, connections are still being made but throughput is down at least, which means I can log in to the VPN and get my day started.

But I can't seem to get focused while watching the log continuously spew 404 errors on my personal laptop as my server tries to keep up with the requests. Why should my web server even need to expend the resources telling these losers (I will not provide examples of the URLs being requested) to go away, the bar is now closed?

Answer: It shouldn't have to. Lucky for me it doesn't have to because I have a BIG-IP 6800 in front of the site in question. I'll just make a quick modification to the existing iRule that directs traffic based on URI and fix these guys for good.

when HTTP_REQUEST {
   set uri [HTTP::host]

   if {$uri contains XXXX} {
   // a string of checks that sets the pool based on the existence of a host name actually hosted here

   }

   else {
      HTTP::respond 200 content "

}

Wham! The connections stop dead as BIG-IP refuses to pass on illegitimate requests. The continual tail of the log file stops abruptly, and the number of connections on the server side evens out to where it normally sits on a Monday morning.

Nice. My server is much happier (its load average is much more normal now) because it doesn't need to respond to bad requests, my bandwidth is freed up and I can go back to drinking coffee and reading my e-mail now that everything's running smoothly thanks to BIG-IP.

Imbibing: (lukewarm) Coffee