You must pick one as you cannot use rate shaping and bandwidth controller policies together. Nevertheless, there's no need to, as Bandwidth controller is the advanced version of rate shaping feature. Neither is available in stock BigIP LTM configuration, unfortunately.
How sophisticated bandwidth control you require? Bandwidth control can be implemented in LTM iRules but it will not be pretty. With PEM AddOn license, you will get access to Bandwidth Controller policies. That would be my 1st choice. Makes it easier to implement and manage. PEM AddOn license cost varies, you're charged more the more powerful your hardware is. Bandwidth controller policies are also included in AAM. Newer appliances with "Best" license bundle should include AAM. Not sure how well the price of AAM AddOn compares against PEM AddOn - if both are more or less equal, AAM is probably a better option as it appears to include more features. Check this with your re-seller to understand the differences in detail. https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/tmos-implementations-11-3-0/8.html
An alternative to PEM is going with rate shaping AddOn module (F5-ADD-BIG-RS). This module is likely to cover your needs as well. That AddOn license has a flat price, costs significantly less than PEM. https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/tmos-concepts-11-4-0/15.html