Forum Discussion

DevBabu_124276's avatar
DevBabu_124276
Icon for Nimbostratus rankNimbostratus
Mar 31, 2016

Regarding optimization of iControl REST- PYTHON code

This code calls the iControl REST API provided by BIG-IP LTM. Trying to get the list of the pools, it's current status and the pool members associated with the pool.

 

My code works like this, based on three calls:

 

Get the pool names (/mgmt/tm/ltm/pool) Based on the pool names (path), get its status (mgmt/tm/ltm/pool/~pool~name/stats) Based on the pool names (path), get its pool members (/mgmt/tm/ltm/pool/~pool~names/members) If I do only the first call, I get the output within milliseconds. After adding the second call, it becomes worse. With a third call, it takes good amount of time (more than 6 minutes) to get all the output for about 400 pools.

 

How can I optimize this code? Basically, I am trying to draw a Network Map. Any help is appreciated.

 

import requests
from requests.auth import HTTPBasicAuth

BASE_URL = "https://bigip/mgmt/tm"
username = "admin"
password = "admin"

def makeRequest(username, password, url):
    response_data = requests.get(url, auth = HTTPBasicAuth(username, password), verify = False)

    return response_data.json()


pool_data = makeRequest(username, password, BASE_URL + "/ltm/pool")

for pools in pool_data['items']:
    print pools['name']
    tildPath = pools['fullPath'].replace('/','~')

    GET the Pool stats
    pool_stats = makeRequest(username, password, BASE_URL + "/ltm/pool/" + tildPath + "/stats")
    print pool_stats['entries']['status.availabilityState']['description']

    GET the Pool Members
    pool_members = makeRequest(username, password, BASE_URL + "/ltm/pool/" + tildPath + "/members")

    for members in pool_members['items']:
        print members['name'] +" " + members['address'] + " " + members['state']

6 Replies

  • Your problem is serial-execution, or in other words - lack of parallelism.

     

    If you have Python2.5 or older, use threading.Thread: https://docs.python.org/2/library/threading.htmlthreading.Thread

     

    If you have Python2.6 or newer, use multiprocessing https://docs.python.org/2/library/multiprocessing.html

     

    • Threading will enable you to run multiple IO-wait streams but you're still stuck to a single-core processing on the client-side. For this task, you are limited on IO-wait, and not on client-side processing power, so it should still help you a lot.
    • Multiprocessing offers the same as Threading, but also allows you to take advantage of more than one CPU core on the client-side.

    Beware that running too many parallel API calls can easily utilize 100% of your F5 appliance CPU. (Test in QA first, and look at performance graphs to find the sweet spot)

     

  • Your problem is serial-execution, or in other words - lack of parallelism.

     

    If you have Python2.5 or older, use threading.Thread: https://docs.python.org/2/library/threading.htmlthreading.Thread

     

    If you have Python2.6 or newer, use multiprocessing https://docs.python.org/2/library/multiprocessing.html

     

    • Threading will enable you to run multiple IO-wait streams but you're still stuck to a single-core processing on the client-side. For this task, you are limited on IO-wait, and not on client-side processing power, so it should still help you a lot.
    • Multiprocessing offers the same as Threading, but also allows you to take advantage of more than one CPU core on the client-side.

    Beware that running too many parallel API calls can easily utilize 100% of your F5 appliance CPU. (Test in QA first, and look at performance graphs to find the sweet spot)

     

  • lara's avatar
    lara
    Icon for Nimbostratus rankNimbostratus

    Multiprocessing Vs Threading

     

    The threading module uses threads, the multiprocessing module uses processes. The difference is that threads run in the same memory space, while processes have separate memory. This makes it a bit harder to share objects between processes with multiprocessing. Since threads use the same memory, precautions have to be taken or two threads will write to the same memory at the same time. This is what the global interpreter lock is for. Spawning processes is a bit slower than spawning threads. Once they are running, there is not much difference.

     

  • You can get the pool member references for each pool by adding the 'expandSubcollections=true' option to the first call (getting the pools). Then, you can skip the third one (getting the pool member references). e.g.,

     

    GET /mgmt/tm/ltm/pool?\expandSubcollections=true