I believe I have the answer...
My monitor is UDP, and it was using the same source IP and port for each connection, so this meant each monitor connection was re-using an existing connection table entry, so changes to my VIP were making no difference.
Ultimately I enabled the VIP only on the vlans that had a default route and then cleared the connection table entries for the monitor. If I enabled the VIP for no vlans, then the monitor traffic gets sent out the default route vlans. Not sure why this is the case, but hopefully this will help someone in the future.
Cheers,
JH