I need to clarify this, i actually confused myself!
My users ssh over port 22 directly to a server. The default gateway of that server is my F5 bigip LTM, so the server replies to the SSH request throught the bigip. The return path is different for this reason. I have this same setup for other devices without SSH issues, this is the only site I am having a strange timeout of around 6 minutes. All configurations checkout, which is why I am dumbfounded.
This will effect the iRule though because the bigip LTM is only catching the SSH session on the return. I'm not even sure it will see it as a session.