Learn F5 Technologies, Get Answers & Share Community Solutions Join DevCentral

Filter by:
  • Solution
  • Technology
Clear all filters
Answers

How to allow Google to crawl my Site, when DOS Profile is active?

Hello all,

we activated the DDos Protection on our F5 Cluster, but after that Google is no longer able to crawl our site. Although I already set all "Google" Signatures on the "Whitelist".

But always when I'm checking the crawl status the Google Bot get's this response:

...
<script type="text/javascript" src="/TSPD/08c1b354d1ab20004ebbacd6cb039eaad8f5d779cf06a9a537d26644e0088d20e75f7b3be3570e19?type=11"></script>
<noscript>Please enable JavaScript to view the page content.</noscript>
</head><body>
</body></html>

And here is the config from our current Profile:

security dos profile Homepage {
    app-service none
    application {
        Homepage {
            bot-defense {
                browser-legit-captcha disabled
                browser-legit-enabled disabled
                mode during-attacks
            }
            bot-signatures {
                categories {
                    "DOS Tool" {
                        action block
                    }
                    "E-Mail Collector" {
                        action block
                    }
                    "Exploit Tool" {
                        action block
                    }
                    "Network Scanner" {
                        action block
                    }
                    "Search Engine" {
                        action report
                    }
                    "Spam Bot" {
                        action block
                    }
                    "Vulnerability Scanner" {
                        action block
                    }
                    "Web Spider" {
                        action block
                    }
                    "Webserver Stress Tool" {
                        action block
                    }
                    Spyware {
                        action block
                    }
                }
                check enabled
                disabled-signatures {
                    "Facebook External Hit" { }
                    "Google AdsBot" { }
                    "Google Desktop" { }
                    "Google Feedfetcher" { }
"Google Page Speed Insights" { }
                    "Google Translate" { }
                    "Google favicon" { }
                    "Nokia-WAPToolkit.\* googlebot" { }
                    AppEngine-Google { }
                    Bing { }
                    Google { }
                    Google-Adwords-Instant { }
                    Google-Calendar-Importer { }
                    Google-Sitemaps { }
                    GoogleWebLight { }
                    Google_Analytics_Snippet_Validator { }
                    Java { }
                    Mediapartners-Google { }
                    YahooSeeker { }
                }
            }
            captcha-response {
                failure {
                    body "You have entered an invalid answer for the question. Please, try again.
<br>
%DOSL7.captcha.image% %DOSL7.captcha.change%
<br>
<b>What code is in the image\?</b>
%DOSL7.captcha.solution%
<br>
%DOSL7.captcha.submit%"
                }
                first {
                    body "This question is for testing whether you are a human visitor and to prevent automated spam submission.
<br>
%DOSL7.captcha.image% %DOSL7.captcha.change%
<br>
<b>What code is in the image\?</b>
%DOSL7.captcha.solution%
<br>
%DOSL7.captcha.submit%"
                }
            }
            ip-whitelist {
             xxx.xxx.xxx.xxx/xx    { }
             xxx.xxx.xxx.xxx/xx { }
             xxx.xxx.xxx.xxx/xx { }
             xxx.xxx.xxx.xxx/xx { }
             xxx.xxx.xxx.xxx/xx { }
             xxx.xxx.xxx.xxx/xx { }
             xxx.xxx.xxx.xxx/xx { }
            }
stress-based {
                mode blocking
            }
            tcp-dump {
                record-traffic enabled
            }
            tps-based {
                device-client-side-defense enabled
                device-rate-limiting enabled
                ip-client-side-defense enabled
            }
        }
    }
    partition Common
    whitelist none
}

Maybe you have a hint for me how to solve this. Current Big-IP version: 12.1.2 - ASM Signatures: v12.1.2/ASM-SignatureFile_20170403_145743

Thanks, Christoph

0
Rate this Question

Answers to this Question

placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

So I opened a ticket for this topic and get this as solution from F5: 1) For this feature to be fully functional F5 recommend to configure a DNS resolver 2) We will recommend to configure a forward zone with "." in FQDN name 3) Also we will recommend to configure the DoS profile on "Bot Signatures", with "Report" enabled for the "Search Bot" and "Search Engine" categories. Remove the Bot Signature List since this overrides the configured actions for the bot signature categories. 4) To validate/verify if this change solves the issue, You can use the google tool to test your website https://support.google.com/webmasters/answer/6065812?hl=en

Now everything is working. Hope this answer will also help others with similar issues

Regards, Christoph

0
Comments on this Answer
Comment made 12-Jul-2017 by nathan 7337

Thanks for posting this Christoph - very informative.

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Christoph - i don't know if this will specifically sort your issue out, but i know that BOT defence needs the BIG-IP to have a DNS server configured. Do you have one setup? Not sure whether the sympton of not having one is the same as your issue, but thought i'd throw it out there. Something you can tick off pretty quickly anyway.

Let us know,

N

0
Comments on this Answer
Comment made 10-Jul-2017 by Christoph Frischhut 134

Hey Nathan,

we have configured a DNS Server under System -> Configuration -> Device -> DNS

But this is just our internal DNS Server, is there also a external DNS needed? Or is there a special config needed for the Bot defence?

Thanks, Christoph

0
Comment made 10-Jul-2017 by nathan 7337

I don't believe so, as long as the BIG-IP can resolve the host names. May be worth checking for DNS queries from the BIG-IP to see if this is working. Also, what does the Event Log for DOS show?

0
Comment made 10-Jul-2017 by Christoph Frischhut 134

The DNS queries are working, just tested from the Big IP CLI. And I also had a look into the DOS Event Logs, but there i can only see the recent attacks. Which we had.

But nothing that some Google Crawling was blocked.

Any other idea?

0
Comment made 10-Jul-2017 by nathan 7337

Odd that you don't see anything in the DoS event logs. Anything in the normal event log? Also, does this log file shed any light? /var/log/dosl7d/dosl7d.log (you may need to increase the process to debug, temporarily, for more information).

0
Comment made 10-Jul-2017 by Christoph Frischhut 134

No also in the dosl7d.log is nothing specific about any blocked google request also I can't set the log level over the web GUI, has this to be done over TMSH?

0
Comment made 11-Jul-2017 by nathan 7337

On reflection, it's probably best to raise a ticket with F5 support before popping this into debug. Wouldn't want to kill your BIGIP if a legitimate attack happens at the same time. Hopefully they will be able to diagnose the issue more fully.

0