Forum Discussion

MocoSpace_63233's avatar
MocoSpace_63233
Icon for Nimbostratus rankNimbostratus
Oct 10, 2007

Redirecting User-Agent Googlebot to new pool or server

Ok - simple problem but I am a newbie. We are getting killed with Googlebot-Mobile spiders. Chewing up all ouer connections. We don't want to stop it but ultimately we may want to rate limit it or something similar. For now we what to redirect the spider to a non-production server so we can better evaluate why it's behaving the way it is. Wrote a quick and dirtyt iRule but does not seem to be working. Ideas on what to look for.

 

 

when HTTP_REQUEST {

 

if {[HTTP::header "User-Agent"] contains "*Googlebot-Mobile.*"} {

 

pool HTTP-SearchEngines

 

}

 

}

 

 

Also any hints on running the traffic generator to test this iRule would also be useful.

 

 

THanks!

 

 

Cheers

 

Mark

5 Replies

  • Deb_Allen_18's avatar
    Deb_Allen_18
    Historic F5 Account
    The "contains" operator will interpret the * as a literal character.

     

     

    But the wildcarding is implied with "contains", so if you remove the *, I think your iRule will work as expected.

     

     

    You can insert headers with the iRuler traffic generation dialog by populating the Headers/Name and Headers/Value fields & popping them into the list on the right.

     

     

    HTH

     

    /deb
  • So I already have a similar iRule. We parse against a large datalist of "unapproved" users and redirect them to a different set of servers. How would I go about rewriting the User-agent as well to something like "x-bot"?

     

     

    THanks!
  • So I already have a similar iRule. We parse against a large datalist of "unapproved" users and redirect them to a different set of servers. How would I go about rewriting the User-agent as well to something like "x-bot"?

     

     

    THanks!
  • There is a codeshare entry which gives two options for selecting a pool based on the user-agent header:

     

     

    Controlling bots (Click here)

     

     

    Aaron

     

  • Actually, I think I misread your request. You want to rewrite the user-agent header to a static value? If so, you can use something like this:

    
    when HTTP_REQUEST {
       if {$something == 1}{
           Replace the User-Agent header (inserting it if it doesn't exist already)
          HTTP::header replace "User-Agent" "x-bot"
       }
    }

    Aaron