Learn F5 Technologies, Get Answers & Share Community Solutions Join DevCentral

Filter by:
  • Solution
  • Technology
Answers

iRule to block multiple user agents.

I have this clunky iRule to block unwanted user agent bots that I have identified hit our site. Unfortunately it is a pain to manage. Is there a better way to write this and not have to use a big if statement with a lot of "or" operators?

when HTTP_REQUEST {
    if {[HTTP::header "User-Agent"] starts_with "AndroidDownloadManager" or [HTTP::header "User-Agent"] contains "Baiduspider" or [HTTP::header "User-Agent"] contains "GrapeshotCrawler" or [HTTP::header "User-Agent"] contains "ScreenerBot" or [HTTP::header "User-Agent"] contains "YandexBot" or [HTTP::header "User-Agent"] contains "SISTRIX" or [HTTP::header "User-Agent"] contains "SPUTNIK"} {
        HTTP::respond 503 content "<html><head><title>Unavailable</title><style type=\"text/css\"> body { font-family: arial; color: black; background: white; } a { color: black; }</style></head><body><center><p>Request denied.<br /><br /></p></center></body></html>"
    }
}

I really wish ASM had a way to block unwanted user agents. I have bot detection enabled but a lot of the time these bots come in at a slow enough rate that they are undetected by bot detection.

Any feedback is appreciated.

Thanks

0
Rate this Question
Comments on this Question
Comment made 29-May-2015 by royceking 183
I think I figured it out. I needed to add the match after the class. when HTTP_REQUEST { if { [class match [string tolower [HTTP::header "User-Agent"]] contains user_agent_blacklist ] } { log local0. "User_agent [HTTP::header "User-Agent"] is blacklisted. from: [IP::client_addr] " drop return } } Seems to be working now. Thanks for the help.
0

Answers to this Question

placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Go to Local Traffic ›› iRules : Data Group List and then hit Create...

From there you type in the data group name (this will be the variable name you insert into the irule) and select 'String' for the type.

Enter in the String Records String field the names of the user agents. I would recommend all lower case. This way the irule logic 'string tolower' will be able to compare apples-to-apples.

1
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Is using a string type data group an option for you? Something like this:

    ltm data-group internal blocked_user_agents {
    records {
        androiddownloadmanager { }
        baiduspider { }
        grapeshotvrawler { }
    }
    type string

    when HTTP_REQUEST {
        if { [class [string tolower [HTTP::header "User-Agent"]] contains $blocked_user_agents ] } {
            drop
            return 
        }
    }
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Tim,

I was actually just researching data groups. So do I just add the data group right into the iRule?

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Do I need to do something special to reference the data group? Right now my testing is blocking all requests.

iRule:

when HTTP_REQUEST {
  if { [class [string tolower [HTTP::header "User-Agent"]] contains user_agent_blacklist ] } {
    log local0. "User_agent [HTTP::header "User-Agent"] is blacklisted. from: [IP::client_addr] "
    drop
    return 
  }
}

Data Group:

Image Text

0