Hosted environment - Multiple VLANs outbound routing

I've got it to work now, but it caused another issue. I have a Virtual Server 10.21.4.11 (external) pointing directly to a server on the internal network (10.21.2.11). I set this up to access it directly with RDP. The connection works initially, but when it times out I cannot re-establish a RDP session. I've done a tcpdump and watch the session come in from the client but the F5 Virtual server will not respond...until I reset the virtual server configuration...(i.e. force a "enabled on" vlan change). This seems to re-set some and it comes up again for a while.

 

 

Any clue?