API Calls to F5 limited to 1024 KB download

See Demystifying iControl REST Part 5: Transferring Files.

It says: "Any file larger than 1M needs to be chunked with this header as that limit is enforced by the worker. "

I have reproduced the symptom: 200 OK is received even the file data is partially transferred.

The following curl call against the file 'large.xml' (12,000,000 bytes) responded with 1 MB (1024 * 1024 bytes) only.

curl -D x -sku admin:<pass> https://<host>/mgmt/tm/asm/file-transfer/downloads/large.xml | wc
      0       1 1048576
 
HTTP/1.1 200 OK
...
Content-Length: 1048576

The behaviour is different from the other downloading endpoint. For example,

curl -D x -sku admin:<pass> https://<host>/mgmt/cm/autodeploy/software-image-downloads/BIGIP-12.1.2.0.0.249.iso -H "Content-Type: application/octet-stream" | wc
   3687   13306 1048576 
 
HTTP/1.1 206 Partial Content
Content-Length: 1048576
Content-Type: application/octet-stream
Content-Range: 0-1048575/2011930624

As you can see, the call returns the first 1 MB along with the 206 Partial Content. You can see the size and the location of the partial content from the Content-Length and Content-Range headers.

The location of the above file transfer (download) is /shared/images. If you want to utilize the 206 response and Content-xxx headers, you can move the asm xml file to that location (use /mgmt/tm/util/unix-mv). Note that the asm file name contains additional user information: e.g., instead of just asmPolicy.xml, it is admin~asmPolicy.xml, so specify the name accordingly in the calls.

Again, please refer to Demystifying iControl REST Part 5: Transferring Files.

Hi , thank you again for replying. I too have been able to validate that the /software-image-downloads/ endpoint gives the 206 web response when files are larger than 1M while the /downloads/ gives only the 200 even when the file size exceed 1M.

The missing part for me is how to move the files. The "export-policy" api call does not provide a means to specify a different location other than the /downloads/ location that is not the ideal endpoint.

As per your suggestion, I have attempted to use the /mgmt/tm/util/unix-mv api call however I can't find details on what parameters this api call requires. As an example of what I have been trying see below.

curl -ku 'admin:<passwd>' -H 'Content-Type: application/json' -X POST https://x.x.x.x/mgmt/tm/util/unix-mv -d '{"name":"/mgmt/tm/asm/file-transfer/downloads/policyName","target":"/shared/images/policyName"}}'

This results in the following error: "{"code":400,"message":"Found invalid JSON body in the request.","errorStack":[],"apiError":1}" the api document does not show any information on this api call.

Any detail on moving the files would be greatly appreciated, this appears to be the missing part at the moment. Since I can get a 206 web response and specify the byte-range I will be able to loop around the file once I get them in the /shared/images location.

Is there any drawback placing the files in the /shared/images location? eg. Can it affect the GUI? I have not seen any issues in the GUI after creating some test files in this location.

I know. The call is tricky. The API for 'tmsh run util' is shown in the iControl REST User Guide Version 14.1 (PDF). Grab the file and look for the section "Using the run command".

For example, to move /tmp/sat (on the Unix filesystem) to /tmp/sat2, run this.

curl -sku admin:<pass> https://<host>/mgmt/tm/util/unix-mv \
 -X POST -H "Content-Type: application/json" \
 -d '{"command":"run", "utilCmdArgs": "/tmp/sat /tmp/sat2"}'

>>> Is there any drawback placing the files in the /shared/images location?

To my knowledge, none (other than the disk space issue).

The /shared/images directory is typically used for storing the BIG-IP iso images. If you have stored a bogus ISO 9660 file with the extension 'iso', it will show up in System > Software Management > Image List on GUI or 'tmsh list sys software images'. Since your ASM file is an XML, they should be benign. If you are super disk-space conscious, you can of course remove the file any time by calling the /mgmt/tm/util/unix-rm endpoint (specify the name of the file in the "utilCmdArgs" property).

One last question - any idea where in the file system files are placed with the /mgmt/tm/asm/tasks/export-policy command?

Empirically, exported policy files are stored under /var/ts/var/rest. The file name contains additional information such as the user name or dates. For example, when a file is created by POSTING {"filename":"admPolicy.xml"}, the file name becomes "admin~asmPolicy.xml". The file name has different format when created via GUI. I learnt this from the 'find' Unix command on my 13.1. Please let me know if you can't find your file there.

Thanks again . I tried the find command however it was coming up blank - there seems to be a clean up taking place. I was able to verify the location of /var/ts/var/rest by specifying the file by the MD5 hash in the export-policy command - running this only once and then listing the contents of this directory.

The naming is as you have noted:

username~policyname

With the naming convention like this, cutting for the f2 after the ~ delimeter will give me a clean policy name.

  - thank you so very much for your time. My script is now working and able to easily handle files above 1MB.

Originally, I wrote this script to sit on an external linux box that would interact with the API and do everything. Due to concerns around credentials in the script, I decided to move the script to the appliance. This way, if the script is discovered no new privileges are gained since the user finding the file already needs access to the F5's advanced shell.

I was actually quite confined to my original thinking in that I needed to do everything with the API. While I do still need 2 API calls, I was able to swap out 2 api calls for bash. This eliminates the 1MB limit entirely.

By following what you provided I was able to find the answer to another issue I had. When I ran my script for the first time, the downloads would be empty, when I ran it a second time etc, all was well. When I was watching the /var/ts/var/rest location I could see that the policy export actually took a few seconds - so when the download did work, it was the previous policy export command that had completed. I placed a sleep in my loop to cater for this and this part is sorted.

I have a significantly better understanding of how all of this fits together now thanks to your assistance. It is greatly appreciated.

I will clean up my solution, perform some testing and provided it here once ready. I do expect others will benefit greatly from your input here. Thank you again!

Informational: I found a related DevCentral question:

Error code 400 while downloading a file via REST API.