Getting a solution working with a wide open Virtual Server is a reasonable plan as a first step in working on the configuration, but after implementation is shown to work, the Virtual Server could be locked down to match traffic-selectors as tightly as possible. Outside of matching the inner (private) IPsec traffic, a wildcard Virtual Server may be handling completely unexpected types of traffic, unrelated to the IPsec traffic.
Here's a config example of configuring two BIG-IPs, each with two Virtual Servers for one specific tunnel. Assume the tunnel is between 10.20.131.0/24 <-> 10.10.131.0/24.
bigip1
tmsh create ltm virtual ipsec-131-encap { destination 10.20.131.0:any ip-forward mask 255.255.255.0 source 10.10.131.0/24 profiles add { fastL4 { } } vlans add { vlan_internal } vlans-enabled }
bigip2
tmsh create ltm virtual ipsec-131-decap { destination 10.20.131.0:any ip-forward mask 255.255.255.0 source 10.10.131.0/24 profiles add { fastL4 { } } vlans add { vlan_external } vlans-enabled }
At this point note, if a flow was initiated by the host on bigip2 to the host on bigip1, the connection would not work. On bigip2, the flow will not match any existing connection and there is no Virtual Server listening to this traffic 10.20.131.1 -> 10.10.131.1. So it is necessary to make a second Virtual Server on both BIG-IP devices to facilitate two-way traffic initiation.
bigip1
tmsh create ltm virtual ipsec-131-decap { destination 10.10.131.0:any ip-forward mask 255.255.255.0 source 10.20.131.0/24 profiles add { fastL4 { } } vlans add { vlan_external } vlans-enabled }
bigip2: Add encapsulating-side side Virtual Server
bigip2
tmsh create ltm virtual ipsec-131-encap { destination 10.10.131.0:any ip-forward mask 255.255.255.0 source 10.20.131.0/24 profiles add { fastL4 { } } vlans add { vlan1555 } vlans-enabled }
One final note, not all types of virtual servers and not all types of profiles will work for IPsec traffic. Stick with the ip-forward.