Forum Discussion

Daniel_S_135288's avatar
Daniel_S_135288
Icon for Nimbostratus rankNimbostratus
Nov 26, 2013
Solved

irule to log googlebots

Hello

 

My goal is to log the googlebot's. I want to log the time and which page was crawled...

 

Can anybody help me?

 

Regards

 

Danisch

 

application delivery
devops
iRules
security
  • IheartF5_45022's avatar
    IheartF5_45022
    Nov 26, 2013

    something like (not tested);

    when HTTP_REQUEST {
    set fBot 0
    if {[HTTP::header value "User-Agent"] contains "Googlebot"} {
        set fBot 1
        set ua [HTTP::header value "User-Agent"]
        set uri [HTTP::uri]
    }
}
when HTTP_RESPONSE {
    if {$fBot} {
        log local0. "[clock clicks -milliseconds],$uri,$ua,[HTTP::status]"
    }
}

4 Replies

Sort By
  • Daniel_S_135288's avatar
    Daniel_S_135288
    Icon for Nimbostratus rankNimbostratus
    Nov 26, 2013
    What I need exactly is: Timestamp URL requested User agent (limit to googlebot) Response code
  • PeteWhite's avatar
    PeteWhite
    Icon for Employee rankEmployee
    Nov 26, 2013

    How about this:

    when HTTP_REQUEST {
if { [HTTP::header "User-Agent"] contains "Googlebot" } {
    log local0 "Googlebot: URI:[HTTP::uri] User-Agent  [HTTP::header "User-Agent"]"
}

    }

  • IheartF5_45022's avatar
    IheartF5_45022
    Icon for Nacreous rankNacreous
    Nov 26, 2013

    something like (not tested);

    when HTTP_REQUEST {
    set fBot 0
    if {[HTTP::header value "User-Agent"] contains "Googlebot"} {
        set fBot 1
        set ua [HTTP::header value "User-Agent"]
        set uri [HTTP::uri]
    }
}
when HTTP_RESPONSE {
    if {$fBot} {
        log local0. "[clock clicks -milliseconds],$uri,$ua,[HTTP::status]"
    }
}