Learn F5 Technologies, Get Answers & Share Community Solutions Join DevCentral

Filter by:
  • Solution
  • Technology
Clear all filters
code share

F5 iApp Automated Backup

Problem this snippet solves:

This is now available on GitHub!

Please look on GitHub for the latest version, and submit any bugs or questions as an "Issue" on GitHub:

https://github.com/tabernarious/f5-automated-backup-iapp

Intro

Building on the significant work of Thomas Schockaert (and several other DevCentralites) I enhanced many aspects I needed for my own purposes, updated many things I noticed requested on the forums, and added additional documentation and clarification. As you may see in several of my comments on the original posts, I iterated through several 2.2.x versions and am now releasing v3.0.0. Below is the breakdown!

Also, I have done quite a bit of testing (mostly on v13.1.0.1 lately) and I doubt I've caught everything, especially with all of the changes. Please post any questions or issues in the comments.

Cheers!

Daniel Tavernier (tabernarious)

Related posts:

Original v1.x.x and v2.x.x features kept (copied from an original post):

  • It allows you to choose between both UCS or SCF as backup-types. (whilst providing ample warnings about SCF not being a very good restore-option due to the incompleteness in some cases)
  • It allows you to provide a passphrase for the UCS archives (the standard GUI also does this, so the iApp should too)
  • It allows you to not include the private keys (same thing: standard GUI does it, so the iApp does it too)
  • It allows you to set a Backup Schedule for every X minutes/hours/days/weeks/months or a custom selection of days in the week
  • It allows you to set the exact time, minute of the hour, day of the week or day of the month when the backup should be performed (depending on the usefulness with regards to the schedule type)
  • It allows you to transfer the backup files to external devices using 4 different protocols, next to providing local storage on the device itself
  • SCP (username/private key without password)
  • SFTP (username/private key without password)
  • FTP (username/password)
  • SMB (now using TMOS v12.x.x compatible 'mount -t cifs', with username/password)
  • Local Storage (/var/local/ucs or /var/local/scf)
  • It stores all passwords and private keys in a secure fashion: encrypted by the master key of the unit (f5mku), rendering it safe to store the backups, including the credentials off-box
  • It has a configurable automatic pruning function for the Local Storage option, so the disk doesn't fill up (i.e. keep last X backup files)
  • It allows you to configure the filename using the date/time wildcards from the tcl [clock] command, as well as providing a variable to include the hostname
  • It requires only the WebGUI to establish the configuration you desire
  • It allows you to disable the processes for automated backup, without you having to remove the Application Service or losing any previously entered settings
  • For the external shellscripts it automatically generates, the credentials are stored in encrypted form (using the master key)
  • It allows you to no longer be required to make modifications on the linux command line to get your automated backups running after an RMA or restore operation
  • It cleans up after itself, which means there are no extraneous shellscripts or status files lingering around after the scripts execute

New v3.0.0 features:

  • Supports multiple instances! (Deploy multiple copies of the iApp to save backups to different places or perhaps to keep daily backups locally and send weekly backups to a network drive.)
  • Fully ConfigSync compatible! (Encrypted values now in $script instead of local file.)
  • Long passwords supported! (Using "-A" with openssl which reads/writes base64 encoded strings as a single line.)
  • Added $script error checking for all remote backup types! (Using 'catch' to prevent tcl errors when $script aborts.)
  • Backup files are cleaned up after any $script errors due to new error checking.
  • Added logging! (Run logs sent to '/var/log/ltm' via logger command which is compatible with BIG-IP Remote Logging configuration (syslog). Run logs AND errors sent to '/var/tmp/scriptd.out'. Errors may include plain-text passwords which should not be in /var/log/ltm or syslog.)
  • Added custom cipher option for SCP! (In case BIG-IP and the destination server are not cipher-compatible out of the box.)
  • Added StrictHostKeyChecking=no option. (This is insecure and should only be used for testing--lots of warnings.)
  • Combined SCP and SFTP because they are both using SCP to perform the remote copy. (Easier to maintain!)

New v3.1.0 features:

  • Removed "app-service none" from iCall objects. The iCall objects are now created as part of the Application Service (iApp) and are properly cleaned up if the iApp is redeployed or deleted.
  • Reasonably tested on 11.5.4 HF2 (SMB worked fine using "mount -t cifs") and altered requires-bigip-version-min to match.
  • Fixing error regarding "script did not successfully complete: (can't read "::destination_parameters__protocol_enable": no such variable" by encompassing most of the "implementation" in a block that first checks $::backup_schedule__frequency_select for "Disable".
  • Added default value to "filename format".
  • Changed UCS default value for $backup_file_name_extension to ".ucs" and added $fname_noext.
  • Removed old SFTP sections and references (now handled through SCP/SFTP).
  • Adjusted logging: added "sleep 1" to ensure proper logging; added $backup_directory to log message.
  • Adjusted some help messages.
How to use this snippet:

Copy the snippet below into a text file. Import the text file as an iApp Template. Create an Application Service using the imported Template. Answer the questions (paying close attention to the help sections). Check /var/tmp/scriptd.out for general logs and errors.

Tested on Version:
13.0
Comments on this Snippet
Comment made 09-Feb-2018 by Darren Walker 226

I've tried this with SCP and SMB and it is working great! I am wondering if it could also cleanup the files in the mount destination: Would I just write that bash logic somewhere in the iAPP?

I noticed a # Clean up local files section and wondered if it's as easy as adding bash there that would rm items older than X days. Something like:

find /var/tmp/cifs/$mountp -mtime +30 -exec rm {} \

0
Comment made 09-Feb-2018 by tabernarious 244

@Darren Walker, I have added your request to the Issues list on the GitHub repository:

https://github.com/tabernarious/f5-automated-backup-iapp/issues

Feel free to leave comments or ideas there! There are some nuances/checks that would have to be worked out. If done improperly this could destroy other critical files on the remote backup volume.

1
Comment made 13-Feb-2018 by Tristan Rhodes 21

Thanks for creating this! You have done a really excellent job on a very valuable tool.

It is embarrassing that an enterprise appliance does not include any methods for automated remote backups.

Tristan

1
Comment made 10-May-2018 by snl 511

HI tabernarious

I am facing one issue with your template were destination ip field only accepting the IP address not the FQDN , is there any way we can fix this .

cheers snl

0
Comment made 10-May-2018 by tabernarious 244

@snl I have relaxed the destination IP restriction in iApp v3.1.2 (available in the Codeshare above and on GitHub)! You can now use destination FQDN as long as you have DNS configured and functional.

0
Comment made 10-May-2018 by snl 511

HI tabernarious

gr8 , thanks for your kind support.

Cheers Snl

0
Comment made 10-May-2018 by tabernarious 244

@Darren Walker, I have implemented SMB auto-pruning in iApp v3.1.3 (available in the Codeshare above and on GitHub)! Finally :) Cheers! -Daniel

0
Comment made 16-May-2018 by Stanislas Piron 10481

Hi,

I tried to use this iApp to backup ucs to remote CIFS server.

I encountered an issue with this iApp because the AD team deny browsing intermediate folders.

if I mount with commands :

mount -t cifs //${server}/${msshare}${mssubdir} ${mountp} -o user=${username}%${password},domain=${msdomain}
ls ${mountp}

I can read files in the mount point, but if I mount with commands (like in your script) :

mount -t cifs //${server}/${msshare} ${mountp} -o user=${username}%${password},domain=${msdomain}
ls ${mountp}${mssubdir}

I have following error:

ls: cannot access /var/tmp/cifs/MY/PATH/: Permission denied

after changing the line 296 to :

exec echo -e "\#\!/bin/sh\nf5masterkey=\$(f5mku -K)\nusername=\$(echo \"ENCRYPTEDUSERNAME\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\npassword=\$(echo \"ENCRYPTEDPASSWORD\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\nmsdomain=\$(echo \"ENCRYPTEDMSDOMAIN\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\nserver=\$(echo \"ENCRYPTEDSERVER\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\nmsshare=\$(echo \"ENCRYPTEDMSSHARE\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\nmssubdir=\$(echo \"ENCRYPTEDMSSUBDIR\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\nmountp=\$(echo \"ENCRYPTEDMOUNTP\" | openssl aes-256-ecb -salt -a -A -d -k \${f5masterkey})\ncd /var/local/ucs\nif \[ \! -d \${mountp} \]\nthen\n\tmkdir -p \${mountp}\n\tif \[ \$? -ne 0 \]\n\tthen\n\t\trm -f ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT\n\t\texit 1\n\tfi\nfi\nmount -t cifs //\${server}/\${msshare}\${mssubdir} \${mountp} -o user=\${username}%\${password},domain=\${msdomain} 2>> /var/tmp/scriptd.out\nif \[ \$? -ne 0 \]\n\tthen\n\trm -f ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT\n\texit 1\nfi\nfONSMB=\$(ls -t \${mountp}/\*.ucs 2>/dev/null| head -n 1 2>/dev/null)\nif \[ \"X\"\${fONSMB} \!= \"X\" \]\n\tthen\n\tsum1=\$(md5sum ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT | awk '{print \$1}')\n\tsum2=\$(md5sum \${fONSMB} | awk \'{print \$1}\')\n\tif \[ \${sum1} == \${sum2} \]\n\tthen\n\t\techo \"ERROR: File ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT already exists in //\${server}/\${msshare}/\${mssubdir}\" >> /var/tmp/scriptd.out\n\t\tumount \${mountp}\n\t\trm -f ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT\n\t\texit 1\n\tfi\nfi\ncp ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT \${mountp}\nrm -f ${fname_noext}BACKUPFILENAMEEXTENSION_WITHDOT\n\nif \[ \"PRUNINGMODE\" \!= \"Disabled\" \]; then\n\n\tfiles_tokeep=\$(ls -t \${mountp}\${mssubdir}/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT 2>/dev/null | head -n CONSERVE\)\n\tfor current_archive_file in `ls \${mountp}\${mssubdir}/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT 2>/dev/null` ; do\n\t\tcurrent_archive_file_basename=`basename \$current_archive_file`\n\t\tcheck_file=\$(echo \$files_tokeep | grep -w \$current_archive_file_basename)\n\t\tif \[ \"\$check_file\" == \"\" \] ; then\n\t\t\trm -f \$current_archive_file\n\t\tfi\n\tdone\n\tif \[ \"BACKUPFILENAMEEXTENSION_NODOT\" == \"scf\" \] ; then\n\t\ttar_files_tokeep=\$(ls -t \${mountp}\${mssubdir}/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT.tar 2>/dev/null | head -n CONSERVE\)\n\t\tfor current_archive_tar_file in `ls \${mountp}\${mssubdir}/*PRUNINGSUFFIX.BACKUPFILENAMEEXTENSION_NODOT.tar 2>/dev/null` ; do\n\t\t\tcurrent_archive_tar_file_basename=`basename \$current_archive_tar_file`\n\t\t\tcheck_file=\$(echo \$tar_files_tokeep | grep -w \$current_archive_tar_file_basename)\n\t\t\tif \[ \"\$check_file\" == \"\" \] ; then\n\t\t\t\trm -f \$current_archive_tar_file\n\t\t\tfi\n\t\tdone\n\tfi\nfi\n\numount \${mountp}\n\nexit 0\n\n" > $scriptfile

it works!

I didn't worked with Pruning mode so I didn't try to correct the script in pruning section.

0
Comment made 16-May-2018 by tabernarious 244

@Stanislas Piron, Awesome! I have added this as an issue on GitHub:

https://github.com/tabernarious/f5-automated-backup-iapp/issues/9

I'll see if I can work this out sooner than later while it's fresh on my mind.

0
Comment made 29-May-2018 by PhillyPDXmike 20

Given that clustered BIG-IP appliances synchronize iApp templates and applications created from iApps, has anyone come across a method with this iApp to backup each appliance in the cluster independently?

0
Comment made 29-May-2018 by tabernarious 244

@PhillyPDXmike, As long as you use the ${host} variable in the file name iApp setting (this is included in the default file name setting) it will generate archives with unique names per device (regardless of clustering). Are there other settings you are hoping to set independently?

0
Comment made 29-May-2018 by PhillyPDXmike 20

@tabernarious, Thanks for the quick response!! I am using version 3.1.2 of the iApp and can confirm that the backup worked on both the active and standby units after I copied the public/private key from the active to the standby, edited the hostname in the public key (/root/.ssh/id_rsa.pub) on the standby, then added that public key to my sftp server's authorized_keys file.

My initial question was prompted from a cosmetic situation where the sample output for the "Select the filename format > Destination Parameters" setting carried over from the active appliance (dev02a) to the standby appliance (dev02b). Image Text

Your question: "Are there other settings you are hoping to set independently?" Since the UCS backup is specific to each appliance, it'd be nice to somehow have device independence within the iApp which is probably more of an F5 iApp software architecture thing than anything to do with this specific iApp. What are your thoughts?

0
Comment made 29-May-2018 by tabernarious 244

I’m glad it’s working, though I don’t think you should have had to mess with the keys (copying between boxes)—I’ll test that too. I see what you mean about the cosmetic file name example issue. I’ll have to think about if that can be fixed.

As for your question about the overall architecture, I definitely agree that there are situations when it would be nice to keep apps independent, but I think the vast majority of cases support the existing keep-everything-in-sync model.

0
Comment made 30-May-2018 by aj1 59

Will this work in version 13.1.x also?

0
Comment made 30-May-2018 by tabernarious 244

@aj1, This should definitely work on 13.1.x. Most of my recent testing has been on 13.1.0.x.

0
Comment made 31-May-2018 by keithhubb

Hi,

I'm using v3.1.3, and getting this message:

script did not successfully complete: (can't read "::destination_parameters__pruning_mode": no such variable while executing "if { $::destination_parameters__pruning_mode eq "Only Prune iApp-Generated Archives" } { set pruning_suffix $::destination_par..." invoked from within "if { $freq != "Disable" } { Ensure a default $filename_format is set if { $::destination_parameters__filenam..." line:44)

I Fixed it by changing line 54 from this:

if { $::destination_parameters__pruning_mode eq "Only Prune iApp-Generated Archives" } {

To this:

if { [info exists ::destination_parameters__pruning_mode] && ($::destination_parameters__pruning_mode eq "Only Prune iApp-Generated Archives") } {

Basically validating the variable exists as a condition to the 'if' statement.

0
Comment made 01-Jun-2018 by aj1 59

I'm having some issues with authentication: In the logs, it shows the warning banner on the host I'm trying to send the backup to, but then shows this:

Permission denied, please try again.
Permission denied, please try again.
Received disconnect from xx.xx.xx.xx: 2: Too many authentication failures for backuphost
lost connection
f5.automated_backup iApp automated_backups_iapp:loadbalancer.com_20180601_085600.ucs REMOTE COPY (SCP) FAILED (check for errors above)
f5.automated_backup iApp automated_backups_iapp: FINISHED

We tried to enter the username/password, and I also copied the private key from the backuphost but still the same. Also, I checked /var/local/ucs for the local copy it would (should?) be copying over but there is only a "config.ucs" file (~70M) and the file's last dated a few weeks ago when we first installed it, I believe. It's been running every few minutes but failing and I thought it was just the copy part that fails. Any guidance would help. Thanks.

0
Comment made 01-Jun-2018 by PhillyPDXmike 20

In my testing, using the backup files via SCP/SFTP option doesn't save a copy locally.

For the "Copy/Paste the SSH private key to be used for passwordless authentication:" option, I copied/pasted the entire RSA private key including -----BEGIN RSA PRIVATE KEY-----, -----END RSA PRIVATE KEY-----, and everything in between. Also, I found these links F5 K13454 and tecmint article very helpful in getting the ssh passwordless authentication working properly, although I didn't have to change any settings on my linux server.

  1. On F5 BIG-IP appliance, create 2048 bit RSA key using the following command at a bash prompt: ssh-keygen -t rsa -b 2048
  2. Accept the default location to save the key (/root/.ssh/id_rsa) by hitting enter
  3. Leave the passphrase blank by hitting enter then hit enter again.
  4. The private key will be save as /root/.ssh/id_rsa and the public key as /root/.ssh/id_rsa.pub
  5. If necessary, create the .ssh directory on the SSH server using the command: ssh user@X.X.X.X mkdir -p .ssh
  6. Copy the public key from the F5 to your SSH server using the following using the command: cat /root/.ssh/id_rsa.pub | ssh user@X.X.X.X 'cat >> .ssh/authorized_keys'
  7. If necessary, change the permission on the .ssh directory and authorized_keys file using the command: ssh username@ip.address "chmod 700 .ssh; chmod 640 .ssh/authorized_keys"
0
Comment made 01-Jun-2018 by tabernarious 244

@aj1, Regarding your search for the local copies of the UCS Archives, the iApp automatically cleans up the local copy whether the remote copy succeeds or fails. If you want local copies I would recommend deploying a second copy of the iApp that just backs up locally and auto-prunes to prevent filling up the /var partition.

0
Comment made 04-Jun-2018 by cbrandys 0

Hello, I a using v3.1.3. When I attempt to implement the script using FTP for remote copy, I receive the following error.

script did not successfully complete: (can't read "::destination_parameters__pruning_mode": no such variable
while executing
"if { $::destination_parameters__pruning_mode eq "Only Prune iApp-Generated Archives" } {
set pruning_suffix $::destination_par..."
invoked from within
"if { $freq != "Disable" } {
Ensure a default $filename_format is set
if { $::destination_parameters__filenam..." line:44)

I am trying to understand the reason for this error. When I choose the "Select the Filename Format" drop-down, I am selecting an option that does include ${host}. Am using code 11.6.1 HF2 on my box. I saw @keithhubb post and resolution but would like to understand the reason for the texterror before I implement on prod boxes. Thank you.

0
Comment made 04-Jun-2018 by tabernarious 244

@cbrandys, see keithhubb’s earlier comment for a quick fix (you have to edit the template). I am planning to fix this officially with a release this week (possibly today). Stay tuned.

0
Comment made 04-Jun-2018 by tabernarious 244

Released v3.1.4 to GitHub and updated CodeShare above. Release Notes:

  • Fixed can't read "::destination_parameters__pruning_mode" errors using [info exists ...].
  • LTM from "Required Modules" list; now no modules are required.
  • SMB/CIFS now directly mounts the target directory instead of the mount point. This allows administrators to deny access to intermediate directories.
  • Set the pruning_keep_amount default to 3 (previously no default existed).
0
Comment made 15-Jun-2018 by Sylvain Q 0

The script runs perfectelly.

I'd like to know if there is a way to have the ssh done on port 224 instead of 22?

Thank you

Sly

0
Comment made 15-Jun-2018 by tabernarious 244

@Silvain Q, If you're referring to the SCP/SFTP option, this could be manually configured relatively easily (it will take me more work to make this customizable through the iApp configuration). In a text editor open the iApp template (f5.automated_backup.v3.1.4.tmpl.tcl), find the line that starts with:

exec echo -e "scp_function()

...then find this part of the line:

\n\tscp -i /var/tmp/TMSHAPPNAME_scp.key

...and replace it with:

\n\tscp -P 224 -i /var/tmp/TMSHAPPNAME_scp.key

(NOTE: -P 224 is the scp option to modify the destination port.)

I would then recommend changing this line:

sys application template /Common/f5.automated_backup.v3.1.4 {

to

sys application template /Common/f5.automated_backup.v3.1.4_scp224 {

(NOTE: Changing the template name will allow you to have both the original v3.1.4 and the new v3.1.4_scp224 templates on the BIG-IP at the same time.)

Finally, save the file you've been editing (I recommend selecting a new name to avoid confusion with the original) and upload it to the BIG-IP (you will have to select "overwrite" unless you edited the sys application template name above).


Let me know if you're looking for something else.

0
Comment made 15-Jun-2018 by Sylvain Q 0

Thank you...

That worked perfectelly...

It could be a nice "new feature" for v3.1.5 ;)

0
Comment made 07-Aug-2018 by Suresh Jo 64

All,

I tried this but I got below error in /var/log/script.log

Saving active configuration... /var/local/ucs/abcd.ucs is saved. abcd.ucs GENERATING f5.automated_backup iApp F5autobackp: abcd.ucs SAVED LOCALLY (/var/local/ucs) f5.automated_backup iApp F5autobackp: abcd.ucs REMOTE COPY (SCP) STARTING

xxxxxxxxxx

Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-with-mic,password). lost connection f5.automated_backup iApp F5autobackp: abcd.ucs REMOTE COPY (SCP) FAILED (check for errors above) f5.automated_backup iApp F5autobackp: FINISHED

Any suggestion?

0
Comment made 08-Aug-2018 by tabernarious 244

@Suresh Jo, Do you have the SCP/SFTP server set up for key-based authentication? And do you have the non-encrypted private key pasted into the iApp?

0
Comment made 28-Aug-2018 by Support 0

Hi,

Could you add a feature to specify the number of occurence you would like with remote backup ? I mean like a logrotate. It's just to avoid a space disk full.

As a workaround, I created a cron on the remote machine to delete file more than XX days.

Thanks,

0
Comment made 28-Aug-2018 by tabernarious 244

@Support, This feature is currently available for "On this F5" and "Remotely via SMB/CIFS". I have not yet taken the time to devise a similar method for FTP or SFTP/SCP. If you have ideas feel free to share them here or on the GitHub repository: https://github.com/tabernarious/f5-automated-backup-iapp

Cheers!

0
Comment made 18-Sep-2018 by efouli 20

Thanks a lot for that iApp, i realy appreciate your effort , i have deployed the iApp and worked from the first time but the second time i got the following error :

script did not successfully complete: (invalid command name "exec" while executing "exec f5mku -K" invoked from within "if { $freq != "Disable" } {

Ensure a default $filename_format is set

if { $::destination_parameters__filenam..." line:44)

0
Comment made 5 months ago by Anthony 0

Hi Suresh,

The issue is that the file you want to transfer doesn't exists, had the same issue.

If you debug, you will see that it tries to transfers a file xxx.ucs.ucs.

Replace {fname}BACKUPFILESCRIPTEXTENSION with {fname}

This will fix the issue.

Kind regards,

Anthony

0
Comment made 5 months ago by tabernarious 244

@Suresh and @Anthony, This filename extension issue for FTP should have been fixed in v3.1.1 (see changelog) which is carried forward to the latest version (currently v3.1.4). What version(s) of the template are you running?

0
Comment made 5 months ago by tabernarious 244

@efouli, I don't believe I've seen that "exec" error before. What version of this iApp are you running, and what version of TMOS are you running? And what method are you using for saving backups (local, SFTP, SMB, or FTP)?

0
Comment made 4 months ago by Markie Parkie 4

Hi all,

Quick question on the iApp and the passwordless scp copy to remote storage.

I have the solution working fine on the first pair of BIGIPs where the keys are already stored on the device, bit doing a fresh pair now that do not have the private key in the default file but pasted into the iApp the device fails to upload to the scp destination, so it looks like I will have to copy the private key to each device to get this working.. Is this how it's meant to work, as I thought pasting it into the iApp config would do away with this part?

Currently using latest version of the template off GitHub.

Thanks..

0
Comment made 4 months ago by tabernarious 244

@Markie Parkie, You should be able to just have the SSH key in the iApp, BUT by default this will fail unless the destination server's SSH fingerprint has been added to '/root/.ssh/known_hosts' (or you set "StrictHostKeyChecking" to "No" in the iApp (not recommended). Have you looked at the iApp logs (/var/tmp/scriptd.out)? If you log into the BIG-IP as root and SSH to the destination server are you prompted about 'Host verification' or similar?

0
Comment made 4 months ago by Markie Parkie 4

Hi Tabernarious,

I have ssh to the destination server to add the fingerprint but still have the same issue. The log just says about too many Auth attempts and falls pass to password Auth.

Thanks..

0
Comment made 4 months ago by tabernarious 244

@Markie Parkie, Could you paste the error from scriptd.out?

0
Comment made 4 months ago by Markie Parkie 4

f5.automated_backup iApp DEVICE_BACKUP: xxxxxxx.xxxxxxx.xxxxxxx_20181112_020000.ucs REMOTE COPY (SCP) STARTING

This system is for the use of authorized users only.

Permission denied, please try again.

Received disconnect from xx.xx.xx.xx: 2: Too many authentication failures

lost connection

f5.automated_backup iApp DEVICE_BACKUP: xxxxxxx.xxxxxxx.xxxxxxx_20181112_020000.ucs REMOTE COPY (SCP) FAILED (check for errors above)

f5.automated_backup iApp DEVICE_BACKUP: FINISHED

0
Comment made 4 months ago by tabernarious 244

@Markie Parkie, I cannot replicate your issue, though I ran into something similar when the destination directory had the wrong permissions (username configured in iApp didn't have write permissions to the directory). Also ensure that you're pasting the OpenSSH b64 private key in this format (with newlines), though I know you've done this successfully elsewhere.

-----BEGIN RSA PRIVATE KEY----- MIIEoAIBAAKCAQEAvnvuC/FWty8k6vrccyGvNP5uabqtT6CJNpKsfgnN0aHzPQ8T xiLI007Bad6+2yW38zvUmXe2u49mFA3KGsOn02NIgehCdCQCJQEEuTW+T9W022Z1 dmAqfeFtz9H7tjq9JSoRfJXxl4lMOjB6QD0DhjE9YVpm8wSB8U4Fr560iKNLRBf6 v1UzqcxT2rqnsIlxEXVBCfC5waiNIMVO+Ipfj9ycNDLgrBpCymR6clc9IZmuJPpt ... -----END RSA PRIVATE KEY-----

Please let me know if you figure anything out!

Does the username or path have any special characters?

0
Comment made 4 months ago by tabernarious 244

v3.1.8 RELEASED

These are the fixes/enhancements (I released v3.1.5 and v3.1.6 recently, so they are included):

v3.1.5 - 20181112

  • Updated KNOWN ISSUES section (below)
  • Fixed SCF passphase issue (v11 "tmsh save sys config file NAME" works and applies no passphase ("no-passphase" flag does not exist); v12+ requires use of "no-passphrase" or "passphrase PHRASE"). (github Issue #18)
  • Reordered filename_format list; default remains ${host}_%Y%m%d_%H%M%S
  • Tested on 11.6.3.3, 12.1.3.7, and 13.1.1.2

v3.1.6 - 20181114

  • Fixed lots of issues with SCF files for SFTP/SCP, FTP, and SMB/CIFS (mainly, tar files were not being copied and were not being cleaned up locally). (github Issue #18)
  • Added logging clarification that when using SCF archives a .tar file is also generated and saved/uploaded.
  • Added debug logging for the SMB/CIFS script.
  • Now including on github an expanded form of the upload scripts for better understanding (see "f5.automated_backup.v3.1.6.scripts.sh"; etc.).

v3.1.7 - 20181115

  • Now supporting many special characters for passwords (without manually escaping with backslashes). (github Issue #3 and #16)
  • SMB/CIFS does NOT support comma, single-quote, and double-quote. I successfully tested this exact password to Windows Server 2012: `~!@#$%^&*()aB1-_=+[{]}|;:<.>/?
  • FTP should support all characters (based on limited testing). I successfully tested this exact password to a Linux FTP server: `~!@#$%^&*()aB1-_=+[{]}|;:,"<.>'/?

v3.1.8 - 20181115

  • Scripts for SMB/CIFS and FTP will again be deleted after each backup. (This was in place as of v3.1.6 but was turned off for debugging in v3.1.7 and was not put back.)

v3.1.9 - 20181120

  • Fixed comment in SMB/CIFS script which was breaking everything due hash escape and a variable reference--I must not have actually tested after I added the comment :(
0
Comment made 4 months ago by Travis.Kamish 0

Has anyone run into cipher issues. My SCP server was recently hardened and now only except the following cipher "aes128-ctr,aes192-ctr,aes256-ctr" I am getting the following error. I also tried to change the cipher in the template but that gets same results. I am running auto_backup v3.1.8 and I am

generated this Tcl error: script did not successfully complete: (no matching cipher found: client aes128-cbc,aes256-cbc server aes128-ctr,aes192-ctr,aes256-ctr lost connection while executing

0
Comment made 4 months ago by tabernarious 244

@Travis.Kamish, What version of TMOS are you running? Did you read through the help section under "Cipher" when configuring SCP/SFTP in the iApp? What is the output if you manually attempt to ssh or scp from the F5 CLI to the destination server?

0
Comment made 4 months ago by Travis.Kamish 0

@tabernarious I am running 13.1.0.8 and when I try to ssh from CLI to server I get same error. "no matching cipher found: client aes128-cbc,aes256-cbc server aes128-ctr,aes192-ctr,aes256-ctr"

I even tried to set the ssh ciphers.

(tmos)# list sys ssh include sys sshd { include "Ciphers aes128-ctr,aes192-ctr,aes256-ctr" }

and with ssh set to none I get same issues with cipher mismatch.

0
Comment made 4 months ago by Travis.Kamish 0

Also if I do this test manually from F5 is works. ssh -vvv -c aes256-ctr username@X.X.X.X

0
Comment made 4 months ago by tabernarious 244

@Travis.Kamish, If you paste "aes256-ctr" into the Cipher field of the iApp does it still fail?

0
Comment made 4 months ago by Travis.Kamish 0

Ok my fault I kept missing the cipher option on scp setup. It is working.

Thanks and sorry for wasting your time.

0
Comment made 4 months ago by tabernarious 244

@Travis.Kamis, No problem at all. I'm just glad it's working for you!!

By the way, I just released v3.1.9 to fix an issue with SMB.

0
Comment made 4 months ago by Markie Parkie 4

In regards to my issue.... This was fixed if I copied from the shell itself and into the iApp. Copying from notepad++ or any other way apart from within shell it failed.

Worked a treat when I did this.

Thanks..

0
Comment made 4 months ago by tabernarious 244

@Markie Parkie, it may have to do with newline/carriage return formatting etc. I may do some testing to see if I can catch that.

0
Comment made 3 months ago by DocHoliday 6

Nice iAPP A cool extra feature would be e-mail notification after the backup was successfully copied to the remote location.

0
Comment made 3 months ago by tabernarious 244

@DocHoliday, Thank you! Regarding email alerts, over the years as an F5 consultant I have received many requests to set up email alerts from the F5 for various things. In the end I find in most cases the very best way to handle this is to send the logs to a centralized log and alert system (syslog/SEIM) and alert from there. Often this is due to internal SMTP restrictions, but it is also a win for tidiness as you don't have so many places to update email settings if ever needed. This iApp does produce log messages that should be easy to filter on. Cheers!

0
Comment made 3 months ago by DocHoliday 6

Hi tabernarious, thanks for your reply. I'll check if this is a possibility.

0
Comment made 3 months ago by Philip Jonsson 1094

Perhaps I'm missing something from the instruction. But I downloaded the above code into notepad++, saved and uploaded to the BIG-IP. Local backups work fine but as soon as I'm trying to send it to a remote storage location I'm hit with this error:

script did not successfully complete: (couldn't create pipe: too many open files
while executing
"exec echo "$::destination_parameters__ftp_remote_username" | openssl aes-256-ecb -salt -a -A -k ${f5masterkey}"
invoked from within
"if { $freq != "Disable" } {
# Ensure a default $filename_format is set
if { $::destination_parameters__filenam..." line:48)

I can see there are some script files in the github repository but the instruction (if I haven't missed anything) does not imply where these should be installed if they need to be. Tried altering the schedule, the encryption setting etc.

Anyone got an idea what I need to do?

0
Comment made 3 months ago by tabernarious 244

@Philip Jonsson, You are correct that you do not need to do anything with the scripts. I added those to GitHub to make it easier to read and maintain the scripts that are embedded in the template.

From the error it looks like it may be having issues with the FTP username. Are you using FTP? If so, does your username have any special or foreign characters?

0
Comment made 3 months ago by Philip Jonsson 1094

@tabernarious - Really weird, it did not work on my secondary device. But if I added the iApp to my primary and synced it over, it worked successfully. I tried both SMB and FTP and both gave the same results on the secondary. Must be some time of bug. Tried the iApp on a customer BIG-IP and it worked straight out of the box as well.

Thanks for the quick response! :)

0
Comment made 1 month ago by Josh W 0

I've implemented the iApp & have it running (for smb/cifs), but encounter the following error when a backup runs:

CLI output from F5:

Broadcast message from root@ (Wed 2019-01-23 12:55:07 PST): Password entry required for 'Password for f5_backup%'Password'@//10.0.0.9/Backups:' (PID 11811).
Please enter password with the systemd-tty-ask-password-agent tool!

Any assistance is greatly appreciated.

0
Comment made 1 month ago by tabernarious 244

@Josh W, What version of TMOS are you running? Any special characters in the password? I have not yet encountered the error you are seeing, but things are always changing :)

0
Comment made 1 month ago by Josh W 0

@tabernarious, thanks for the quick response. There are special characters in the password. I attempted without using escape characters & using escape characters, but both still gave the error above when executing the iApp.

Here is the output for the TMOS version: Sys::Version Main Package Product BIG-IP Version 14.1.0 Build 0.0.116 Edition Final Date Wed Nov 14 18:41:56 PST 2018

0
Comment made 1 month ago by tabernarious 244

I suspect this is a change with 14.1 and how it handles password input. I’ll dig a little to see if this can be accommodated.

0
Comment made 1 month ago by Markie Parkie 4

@Josh W Its not to do with the F5 connecting to the remote system is it when dumping out the file. Are you using the cert key to authenticate to the file storage device?

0
Comment made 1 month ago by Josh W 0

@Markie, No, I'm not using the cert key to authenticate. In the iApp it prompts for username & password for cif share. Since it's a cif share, a cert shouldn't be needed.

0
Comment made 1 month ago by Delbrugge 1

Getting this error on 11.6.0

Jan 29 23:14:47 bigip1 err scriptd[20085]: 014f0013:3: Script (/Common/f5.automated_backup__auto-backup-local) generated this Tcl error: (script did not successfully complete: (UCS saving process failed. while executing "tmsh::save /sys ucs $fname passphrase testpass#$%0 " line:18))

EDIT: This was an issue on the VE that I was running this on with encryption. Nothing wrong with this script.

0
Comment made 3 weeks ago by Sebastian Aguirre G. 249

Sick job dude, you saved me a lot of time, but i have a problem importing the template on some devices, i'm getting the following message: "Loading configuration... Loading schema version: 13.1.1.2 Unexpected Error: Can't find specified cli schema data for 13.1.1.2" on BIG-IP 13.1.1.4 Build 0.0.4 Point Release 4.

0
Comment made 1 week ago by aries 106

Hi! First off, thank you very much for sharing this. So cool and super helpful!

Can anyone tell me what the field "Amount of files to keep at any given time:" is for? I thought this was in relation to automatic pruning - will only prune files exceeding the value for this field, however it seems it is not for this purpose.

0
Comment made 11 hours ago by Josh 0

It appears this iApp template is no longer working on Big-IP 14.1.0 or higher.

I was testing v3.1.9 on a Big-IP 14.1.0.2 VE appliance with following results:

The .tcl code imported normally when creating a new iApp template. But when attempting to create a new iApp app service and selecting the template, most of the presentation elements are now missing on the configuration page.

@tabernarious, can you work your glorious magic please? Cheers!

0