Forum Discussion
Sep 23, 2013
Afaik, the google bot can be identified by the user agent.
But this can be faked. And sometimes the google bot does not identify itself to make sure you are not tuning your site for google.
How to proceed?
First thing might be a log analysis of your tomcat logs to figure out, what these requests have in common. Perhaps it´s always the same source IP, same user agent, same path, whatever ...
Based on this, an iRule can be applied to limit the number of requests based on specific criterias. There are some table-based sample iRules for request rate limitation on DC.
Just dropping all google requests may have some negative business impact ...