How to convert iRules from bigip 10.1.0 to bigip 11.6.3?

There is no such animal (that I'm aware of). Your best option would be to import your irules through the GUI and work through any incompatibilities you encounter. The syntax checker on the GUI should inform you of deprecation or the like and the basic logic should still be sound, if possibly no longer optimal.

If that's too much hassle we also have professional services who can likely do the conversion for you. It depends on how many irules you have and how much time verses money you have.