Hi Hille,
Are you using the advanced client auth license to check the CRL? I worked with a poster here for a similar situation with OCSP authentication where the person had two different CA's OCSP servers they needed to check client certs against. The solution we came up with was to use a VIP and iRule to select the correct OCSP URL based on the client cert that was being validated. We then configured this VIP as the URL for the OCSP responder. I imagine you could do something similar for the CRL. This CRL load balancing VIP would be used internally to perform the CRL validation--not as a second VIP that the clients connect to.
If this seems like it might work for your scenario, let me know and I'll try to clean up and anonymize the OCSP example. It might also be worth checking with either an F5 account manager/presales person or F5 Support to see if there's a simpler solution.
Aaron