Throttling 404 requests to protect against DoS attacks

I'm not sure you've thought this through. Wouldn't an attacker prefer to make multiple requests for correct but 'heavy' URLs? What happens if someone makes a mistake coding a web page and genuine clients make invalid requests because of it?

Whilst it's more than possible to use an iRule to meet your stated goal, introducing logic to reduce false negatives and cover other eventualities will make it very complex and considering other possible attack vectors it's probably not worth the effort.

ASM is of course an option.