Where can I download Hotfix-BIGIP-11.3.0-3164.0-HF10

Release 11.5.3 is End of Technical Support. Upgrade device to latest version per below link.

Please refer the BIG-IP software support policy- https://support.f5.com/csp/article/K5903

Do I have to pay any additional license fees for the upgrade?

No.. Generate the QKview, upload to ihealth.f5.com and see the F5 Health advice. It will help you to decide which version is good for you.. thanks