As Chris suggests, you could use an iRule implement the checks you've described. This wouldn't be a simple iRule, but it could be done. It would be a lot simpler and efficient if you're running (or able to upgrade to) 10.1 or higher as you'll be able to use the table command to track the client IP addresses. Keep in mind that if you have a lot of clients connecting from behind the same proxy address, you could potentially block legitimate users who share the same proxy.
You'd first check for requests to login pages and then the response headers or content to determine if the login attempt failed. If you can determine this from the response headers it would be a lot more efficient than checking the response payload. You could keep track keep a blacklist of client IP addresses which you use to ban users who fail the login process too many times.
To check the requested URI, you can use HTTP::path or HTTP::uri in the HTTP_REQUEST event. To check response headers you can use HTTP::header in the HTTP_RESPONSE event. To collect the response payload, you can use HTTP::collect in the HTTP_RESPONSE event. Once the data has been collected, you can retrieve it using HTTP::payload in the HTTP_RESPONSE_DATA event.
See the HTTP wiki page for details on these commands and events:
http://devcentral.f5.com/wiki/default.aspx/iRules/http
You can check the table command wiki page for details on this command:
http://devcentral.f5.com/wiki/default.aspx/iRules/table
Table article series
http://devcentral.f5.com/Default.aspx?tabid=63&articleType=ArticleView&articleId=2375
Also, the Application Security Manager (ASM) provides protection against brute force attacks via simple GUI configuration. It might be worth looking into this as an option.
http://www.f5.com/products/big-ip/product-modules/application-security-manager.html
Aaron