A requirement arose whereby we needed to return a more friendly page to users who were attempting to connect to our services, but didn't have the relevant SSL requirements fulfilled. E.g. they weren't presenting a Client SSL certificate, or their certificate was expired.
The trick was to change the 'peer_cert_mode' from 'require' to 'request'. This allows the browser to open a Secure connection with the F5 and the F5 perform the required validation on the certificate being presented. Thanks to Hoolio and Nitass for their pointers...
The rule uses iFiles to get the error page being returned to the client. You need v11 for iFiles to work, but you should be able to replace the functionality with External files or inline HTML if you want to use this rule on v10 or earlier... As part of the rule, I've added a keyword called $failure_result. This stores the X509 Verify result string value. If you want to return this value to the client as part of the HTML, then simply add the variable somewhere within your HTML. The inline subst below will then replace it with the relevant failure string. A full list can be found on the 'SSL::Verify' page.
Comments/improvements always welcome.