Forum Discussion

Dave_Burnett_20's avatar
Dave_Burnett_20
Icon for Nimbostratus rankNimbostratus
Nov 10, 2008

How to allow Search Engine Robots/Slurps through ASM?

We have recently installed a pair of F56400s (v9.4.3) in front of our website with ASM in blocking mode.     We are seeing and blocking loads of Non-RFC compliant request violations. Exami...
app sec
BIG-IP Access Policy Manager (APM)
security
hooleylist's avatar
hooleylist
Icon for Cirrostratus rankCirrostratus
Nov 11, 2008
As Aaron said, I haven't seen this in other customer implementations and I'm surprised to see that Yahoo's spider would be making invalid requests. I'd guess that this might be a "custom" interpretation of an HTTP RFC requirement that ASM is enforcing. Can you post an anonymized copy of a request or two that is being marked illegal?

 

 

Thanks,

 

Aaron