Two third party hosts connect through VIP to one pool member

Two third party hosts are NATed and come in as internal addresses through one VIP on the LTM. If only one is active the connection works fine. The tcpdump shows the connection coming in to the VIP, then from the self IP to the pool member.

For some reason when both host IPs are active it seems only one will stay connected at a time. The connection seems to flap.

Here is one of the error messages we receive on the pool member when the secondary is connecting.

Remote system <self IP> has reconnected, destroying existing socket 2656

We are using Performance Layer 4 via TCP.

Would a snatpool help for each host coming in. If so, what would the IRule syntax be? At least this way the pool member would see each coming in from the SNAT IP instead of the self IP.