Forum Discussion

saidshow's avatar
saidshow
Icon for Cirrus rankCirrus
Jul 24, 2019

API Calls to F5 limited to 1024 KB download

Hi,

I am interacting with the F5 API in order to download ASM policies for the purpose of automating the backups. The process works fine however policies larger than 1024 KB are cut off at this size of 1024 KB. Initially I suspected that there was a default limit on the curl request however I have not been able to find information on how to increase this with the curl request.

Is this a limitation on the F5 API or the Curl request? wget is not an option as this is not natively supported on the F5 virtual appliance. My script lives on the appliance, downloads the relevant policies and then pushes them to a SMB share. The only issue is that the ASM policies that are larger than 1024 KB are being cut off at 1024 KB.

The API calls are as per the documentation here:

http://cdn.f5.com/websites/devcentral.f5.com/downloads/icontrol-rest-api-user-guide-13-0-0.pdf

specifically:

GET https://x.x.x.x/mgmt/tm/asm/policies

POST https://x.x.x.x/mgmt/tm/asm/tasks/export-policy

GET https://x.x.x.x/mgmt/tm/asm/file-transfer/downloads/$asmPolicy

Excluding the processing in my script the API calls I make are shown below: I expect the issue resides in the download api call. Is there a switch I can add to increase this limit?

curl -ku 'username:password' -X GET https://x.x.x.x/mgmt/tm/asm/policies | jq '.items[] | "pol_name:" + .name + ";api_id:" + .id' >> $wdir/asmDetails.txt
 
 
curl -ku 'username:password' -X POST https://x.x.x.x/mgmt/tm/asm/tasks/export-policy -H 'Content-Type: application/json' -d '{"filename":"'$asmPolicy'","policyReference":{"link":"https://localhost/mgmt/tm/asm/policies/'$asmIDs'"}}'
 
 
curl -ku 'username:password' -X GET https://x.x.x.x/mgmt/tm/asm/file-transfer/downloads/$asmPolicy > $wdir/asmBackup/"$folderName"/$number-$asmPolicy-$hostname-"$dateStamp".xml
 

Thanks

1024
api
ASM Advanced WAF
backup
curl
devops
iControlREST
limit

16 Replies

Sort By
Show More
  • Satoshi_Toyosa1's avatar
    Satoshi_Toyosa1
    Icon for Employee rankEmployee
    Jul 30, 2019

    I found my files disappeared (including bogus test files I created) from the /var/ts/var/rest directory too. I haven't been able to find the process responsible for deletion .

    • saidshow's avatar
      saidshow
      Icon for Cirrus rankCirrus
      Jul 30, 2019

      I am trying to post my solution but get an error from the devcentral portal that reads "Maximum number of topic assignments have been exceeded"

      • saidshow's avatar
        saidshow
        Icon for Cirrus rankCirrus
        Jul 30, 2019

          - my working solution with creds removed is shown below. You were CRUCIAL to getting this to work. Thank you so very much!!!

        Unfortunately I get an error when trying to post this as code... hopefully users can zoom in on the image below.

         

        0691T000005ljbfQAA.png

         

        I have an iCall period handler kicking off an iCall script that executes this bash file. Note, this does push the exported asm policies in xml format to an external SMB share and forces a cleanup.

         

        In the event that the unix box has rm pointing to rm -i, I specify /bin/rm to prevent prompting for the clean up.

         

        I will leave this run in TEST every 15 mins for an hour, when I am comfortable all is well, I will have it run every 28 days at 2am starting on Sunday. This is controlled with the iCall period handler.

  • JG's avatar
    JG
    Icon for Cumulonimbus rankCumulonimbus
    Jul 30, 2019

    The underlying purpose of the question has been served by a work-around, and thanks for the sharing of the script.

     

    However, the subject issue of curl failing to download beyond the first 1024KB remains.

     

    In my tests, specifying the "Range:" header in the client request was also not successful; the server always returned the first 1024KB no matter what byte range was requested.

     

    [Edit]

    Satoshi Toyosawa has just clarified (in his linked post above) that "Content-Length" should be used rather than "Range:", as this is how F5 implemented it.