Learn F5 Technologies, Get Answers & Share Community Solutions Join DevCentral

Filter by:
  • Solution
  • Technology
code share

Complete F5 Automated Backup Solution

Problem this snippet solves:

Hi all,

Often I've been scouring the devcentral fora and codeshares to find that one piece of handywork that will drastically simplify my automated backup needs on F5 devices. Based on the works of Jason Rahm in his post "Third Time's the Charm: BIG-IP Backups Simplified with iCall" on the 26th of June 2013, I went ahead and created my own iApp that pretty much provides the answers for all my backup-needs.

Here's a feature list of this iApp:

  • It allows you to choose between both UCS or SCF as backup-types. (whilst providing ample warnings about SCF not being a very good restore-option due to the incompleteness in some cases)
  • It allows you to provide a passphrase for the UCS archives (the standard GUI also does this, so the iApp should too)
  • It allows you to not include the private keys (same thing: standard GUI does it, so the iApp does it too)
  • It allows you to set a Backup Schedule for every X minutes/hours/days/weeks/months or a custom selection of days in the week
  • It allows you to set the exact time, minute of the hour, day of the week or day of the month when the backup should be performed (depending on the usefulness with regards to the schedule type)
  • It allows you to transfer the backup files to external devices using 4 different protocols, next to providing local storage on the device itself

    *   SCP (username/private key without password)
    
    • SFTP (username/private key without password)
    • FTP (username/password)
    • SMB (using smbclient, with username/password)
    • Local Storage (/var/local/ucs or /var/local/scf)
  • It stores all passwords and private keys in a secure fashion: encrypted by the master key of the unit (f5mku), rendering it safe to store the backups, including the credentials off-box

  • It has a configurable automatic pruning function for the Local Storage option, so the disk doesn't fill up (i.e. keep last X backup files)

  • It allows you to configure the filename using the date/time wildcards from the tcl clock command, as well as providing a variable to include the hostname
  • It requires only the WebGUI to establish the configuration you desire
  • It allows you to disable the processes for automated backup, without you having to remove the Application Service or losing any previously entered settings
  • For the external shellscripts it automatically generates, the credentials are stored in encrypted form (using the master key)
  • It allows you to no longer be required to make modifications on the linux command line to get your automated backups running after an RMA or restore operation
  • It cleans up after itself, which means there are no extraneous shellscripts or status files lingering around after the scripts execute

Enjoy!

Thomas Schockaert

Contributed by: Thomas Schockaert

How to use this snippet:

minimum version 11.4

Tested on Version:
11.4
Comments on this Snippet
Comment made 14-May-2015 by f5_rock 2841
can you please share the script
0
Comment made 05-Jun-2015 by NetworkNerd 0
F5 Automated Backups - The Right Way https://devcentral.f5.com/articles/f5-automated-backups-the-right-way On that page, there's a pastebin link to: http://pastebin.com/YbDj3eMN
0
Comment made 01-Dec-2015 by quanquan 0
Please share your script,Thanks.
0
Comment made 07-Apr-2016 by LDS 1
willl this work on 11.6
0
Comment made 30-May-2016 by Squeak 75
If I want to specify which Route domain do I just add the "%x" after the destination address I want to use? Is is possible at all to use "%"?
0
Comment made 31-May-2016 by Xian Zhong 0
Hi Thomas I have encounter this issue totally similar to another user posted back in 2014. The problem is as such: "Has anyone got this to work 100% properly? I am not able to restore from FTP. I have no problem with making a job FTP the UCS file (without passphrase) to a FTP server. But when I download it back to the F5, and try to restore it from the file, it fails with: Saving active configuration... Current configuration backed up to /var/local/ucs/cs_backup.ucs. tar: Skipping to next header tar: Archive contains obsolescent base-64 headers gzip: stdin: invalid compressed data--format violated tar: Child returned status 1 tar: SPEC-Files: Not found in archive tar: Error exit delayed from previous errors Fatal: executing: tar -zxf /var/local/ucs/20140919_backup_test.com.ucs SPEC-Manifest SPEC-Files Operation aborted. /var/tmp/configsync.spec: Error installing package Config install aborted. Unexpected Error: UCS loading process failed. I am not sure what this mean, but I have notices that the version is “unknown” when opening the backup file on the F5. Furthermore, if I set the job to save on F5 instead of FTP, I can restore it without problem, and the version is now shown right version. Please advice. P.S. This is testet in both version 2.0, 2.0.1-tdd and 2.0.2-tdd " Please advice what could be a solution to this.
0
Comment made 27-Jun-2016 by Sylvain Q 0
After the SFTP scheduled backup runs, it doesn't seems to delete the file in /var/local/ucs folder. Is there a way to be sure that the deletion is in place?
0
Comment made 05-Jul-2016 by Delalegro 0
Has anyone tested this iApp on v12.0?
0
Comment made 18-Jul-2016 by Richard Reszler 0

Broken in 12. :) Waiting for the updated build, thank you!

0
Comment made 18-Jul-2016 by Richard Reszler 0

Error is "Error parsing template:can't eval proc: "script::run" field not present: "hostname" while executing "tmsh::get_field_value [lindex [tmsh::get_config sys global-settings] 0] hostname" (procedure "script::run" line 2) invoked from within "script::run" line:1"

0
Comment made 01-Aug-2016 by Christoph Frischhut 134

Is there any update for 12.1.0? Because when I deploy it on this firmware the config files are empty...

0
Comment made 19-Aug-2016 by Julio Flores 53

hi can you help me please, i use this procedure in one big ip standalone and Works fine!!! But recently i do an DCS configuration with 4 big ip, and this procedure doesn't work, have you another information to do in this Type of cluster or configuration.

Thanks

Julio F

0
Comment made 24-Aug-2016 by svs 313

@Xian Zhong: I've probably found the reason for this issue. My customer encountered the same issue, when he was using the FTP transfer method from the iApp. It seems, that the iApp is using ASCII instead Binary transport to the FTP server (ASCII is the default mode of the builtin ftp client in Linux). Therefore the compressed file is corrupted. The issue was solved by using SCP/SFTP for transfer.

If necessary you can repair the broken files on your FTP server by using "fixgz" (http://www.gzip.org/#faq1)..) It worked for me to rescue the corrupted files.

@Thomas: If you would add the command "binary" to the scriptfile before the transport starts (put command), it should work as expected.

Regards, Sven

0
Comment made 24-Aug-2016 by svs 313

Hi Thomas,

this is really a great scripts. Thank your very much!

It seems that there are some open issues, but it works like charm most of the time (when using SCP/SFTP). Regarding SCP/SFTP...where is the difference in your script, except for the filenames created during runtime? SCP is used for the transport in both cases. Wouldn't it make sense to really use the sftp command?

Regards, Sven

0
Comment made 07-Nov-2016 by EmsNetSec 0

Thanks Thomas.

You made the life easier with F5 Backup

0
Comment made 10-Nov-2016 by Tony N 0

I'm getting the following error when I run this on BIG-IP 11.5.4 Build 1.0.286 Hotfix HF1. Does anyone have a fix for this?

Nov 10 10:10:08 slot1/f5ltm01 err scriptd[3555]: 014f0013:3: Script (/Common/f5.automated_backup) generated this Tcl error: (script did not successfully complete: (bad decrypt 47145560865920:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:601: while executing "exec $scriptfile" line:17)) Nov 10 10:10:08 slot1/f5ltm01 err mcpd[7254]: 0107167d:3: Data publisher not found or not implemented when processing request (unknown request), tag (2901).

Thank you, Tony

0
Comment made 17-Nov-2016 by Roflcopter 194

I am getting the exact same error as you Tony, but only on the standby unit.

Nov 18 09:58:01 PROD2-F5-4000S err scriptd[8190]: 014f0004:3: script has exceeded its time to live, terminating the script Nov 18 09:58:01 PROD2-F5-4000S err mcpd[7006]: 0107167d:3: Data publisher not found or not implemented when processing request (unknown request), tag (2901).

0
Comment made 02-Dec-2016 by Jon Swick 69

Script (/Common/f5.automated_backup) generated this Tcl error: (script did not successfully complete: (ftp: connect: Connection timed out while executing "exec $scriptfile" line:17))

Data publisher not found or not implemented when processing request (unknown request), tag (2901).

I am getting those two errors when trying to ftp

** Server Mis Config. Im all good

0
Comment made 13-Dec-2016 by Nathaneil0227 401

Hi,

I got an error in version 12.0

Script (/Common/f5.automated_backup) generated this Tcl error: (script did not successfully complete: (Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,password). lost connection while executing "exec $scriptfile" line:18)) Dec 13 18:09:11 F5-Lab err mcpd[5154]: 0107167d:3: Data publisher not found or not implemented when processing request (unknown request), tag (2901).

Could you please help me?

Regards,

Nat

0
Comment made 13-Dec-2016 by Nathaneil0227 401

Help please :)

0
Comment made 26-Dec-2016 by mplaksin 167

We have to test it. Thanks you.

0
Comment made 06-Jan-2017 by Pinko_Commie 66

I ran into the same issue as Nathaneil0227.

In my case it was because we were using DSA keys instead of RSA keys. So the cleaning of the keyfile portion didn't work correctly.

So i changed the line:

set cleaned_privatekey [exec echo "$::destination_parameters__scp_sshprivatekey" | sed -e "s/BEGIN RSA PRIVATE KEY/BEGIN;RSA;PRIVATE;KEY/g" -e "s/END RSA PRIVATE KEY/END;RSA;PRIVATE;KEY/g" -e "s/ /\\\n/g" -e "s/;/ /g"]

to:

set cleaned_privatekey [exec echo "$::destination_parameters__scp_sshprivatekey" | sed -e "s/BEGIN RSA PRIVATE KEY/BEGIN;RSA;PRIVATE;KEY/g" -e "s/END RSA PRIVATE KEY/END;RSA;PRIVATE;KEY/g" -e "s/BEGIN DSA PRIVATE KEY/BEGIN;DSA;PRIVATE;KEY/g" -e "s/END DSA PRIVATE KEY/END;DSA;PRIVATE;KEY/g" -e "s/ /\\\n/g" -e "s/;/ /g"]

I also made another tweak to allow you to set the remote folder to be the hostname of the device by entering $hostname for the path.

0
Comment made 12-Jan-2017 by dsjustin 2

This is an awesome Iapp and I approve wholeheartedly. I would like to create a portion for aws client to be able to use that for uploading backups in AWS environment. I am looking at modifying the SMB client portion, and then adding it to the template for AWS. Has anyone else done something like this?

0
Comment made 01-Mar-2017 by mike.drennen 268

Just as a side note, We fought with the SMB using a domain account to authenticate as. We had to use domain\\username (two slashes between) so that the \ passed through to smbclient. Just an FYI.

0
Comment made 21-Apr-2017 by arzhukov 0

I have another problem, It seem like work perfectly. But when script try to connect with credentials that I typed in, server doesnt give acces But I`m absolutely sure that I typed valid credentials. Please help me to resolve that issue..

Saving active configuration... /var/local/ucs/ltm51.cetelem-bank.ru_20170421.ucs is saved. Script (/Common/f5.automated_backup) generated this Tcl error: script did not successfully complete: (Permission denied, please try again. Permission denied, please try again. Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). lost connection while executing "exec $scriptfile" line:18)

0
Comment made 04-May-2017 by RobertS 166

Question about using SCP server. Which private key is used? How do I find it? Change it?

In ltm log I get: (/Common/f5.automated_backup) generated this Tcl error: (script did not successfully complete: (Host key verification failed. lost connection while executing "exec $scriptfile" line:18))

0
Comment made 21-May-2017 by Ramprasath Sadasivam 0

Hello Guys,

While executing the f5_automated_backup using FTP. I am getting the following error. Can any one help on this.

err mcpd[6853] 0107167d Data publisher not found or not implemented when processing request (unknown request), tag (2901).

err scriptd[1797] 014f0013 Script (/Common/f5.automated_backup) generated this Tcl error: (script did not successfully complete: ("ucs" unexpected argument while executing "tmsh::save /sys ucs $fname passphrase XXXXXX " line:8))

regards Ram

0
Comment made 18-Aug-2017 by Nick.Linney 0

Is it possible to alter the SFTP option on this iApp to use password based authentication rather than private key?

0
Comment made 22-Aug-2017 by Troy Murray 125

@Ramparasath Sadasivam I have the same problem and I'm not sure how to correct this yet.

0
Comment made 22-Aug-2017 by Troy Murray 125

@Thomas Schockaert any possibility that this iApp could be hosted on GitHub (https://github.com/f5networks), like some other iApps are, for code contributions or issue tracking?

1
Comment made 29-Aug-2017 by dlee@biztecharchitects.com 2

How do you change this script to just backup the config locally?

0
Comment made 08-Sep-2017 by Troy Murray 125

dlee@biztecharchitects.com it has a built in way to save the config locally.

Under Destination Parameters, for the question "Where do the backup files need to be saved?", select "on this F5" drop the drop down list.

1
Comment made 02-Nov-2017 by Andreas Schiermeier 15

To work around "script has exceeded its time to live, terminating the script" errors in /var/log/ltm adjust the allowed script run time by scriptd:

TMSH:

modify /sys scriptd max-script-run-time 600
save /sys config
run /cm config-sync to-group …
0
Comment made 15-Nov-2017 by Rosieodonell 365

This is the error that I am getting when trying to use this iapp: "f5.automated_backup.v2.1.1";

010715bc:3: The application service (/Common/f5_pratcice_backup.app/f5_pratcice_backup) has strict updates enabled, the object (icall periodic handler /Common/f5.automated_backup_pruning-handler) must be updated using an application management interface.

It's nice to back up your F5 and I can't seem to get backups workign again. I am running 12.1.2 Build 1.0.271 Hotfix HF1.

0
Comment made 06-Dec-2017 by Seven11 2

I want to use the Backuptool with SCP and Private Key but the login is not working. Which Format, Encryption or else is needed to get is running with a key?

Can you Post an example to see what i need to paste in the GUI.

TMOS 13.0.0 HF3

Thanks!

0
Comment made 13-Dec-2017 by tabernarious 244

f5.automated_backup.v3.0.0 now available on DevCentral CodeShare:

https://devcentral.f5.com/codeshare/f5-iapp-automated-backup-1114

(see later comment on this forum)

1
Comment made 10-Jan-2018 by AhmedGalal219 181

Greate App thanks man it works perfectly with me in version 12.1.2

0
Comment made 10-Jan-2018 by kkohegyi 55

Hi,

It is a great App.

But when the remote destination is not reachable the script does not remove the backup file(s) from local directory.

So the /var directory may fill up and it may cause unpredictable behaviour. A little modification is necessary if anybody want to use it production environment.

1
Comment made 15-Jan-2018 by Robert Goetze 0

Hi, Is there a repository of all the versions(V2.0 - V2.2.5) ? I am trying to run multiple instances of the iApp on 11.6.x and think version v2.2.1 would do it or atleast maybe point me in the right direction.

Thanks in advance.

0
Comment made 17-Jan-2018 by tabernarious 244

@Robert, The only change that I am aware of that may not work in v11 is related to SMB (now using "mount -t cifs" instead of smbclient). Does v2.2.5 not load or work properly on v11.6.x?

You can find v2.0.0 at https://pastebin.com/YbDj3eMN but that does not include the changes to allow multiple instance.

0
Comment made 24-Jan-2018 by tabernarious 244

This is now available on GitHub! Please look here for the latest version, and submit any bugs or questions as an "Issue" on GitHub:

https://github.com/tabernarious/f5-automated-backup-iapp

As a BACKUP, look in the DevCentral CodeShare. This MAY NOT HAVE THE LATEST VERSION:

https://devcentral.f5.com/codeshare/f5-iapp-automated-backup-1114

I needed to troubleshoot an issue and ended up rolling in many of the proposed and posted fixes into a new iApp.

# iApp VERSIONS (From what I gathered perusing DevCentral)
# ~v2.0  - 20140312 - Initially posted releases (v11.4.0-11.6.x? compatibility). (Developed/posted by Thomas Schockaert)
# v2.1.1 - 20160916 - Retooled SMB upload from smbclient to "mount -t cifs" (v12.1+ compatibility). (Developed/posted by MAG)
# v2.2.1 - 20171214 - Allowed multiple instances of iApp by leveraging $tmsh::app_name to create unique object names. (Developed by Daniel Tavernier/tabernarious)
# v2.2.2 - 20171214 - Added "/" to "mount -t cifs" command and clarified/expanded help for SMB (CIFS) Destination Parameters. (Developed by Daniel Tavernier/tabernarious)
# v2.2.3 - 20171214 - Set many fields to "required" and set reasonable default values to prevent loading/configuration errors. Expanded help regarding private keys. (Developed by Daniel Tavernier/tabernarious)
# v2.2.4 - 20171214 - Added fix to force FTP to use binary upload. (Copied code posted by Roy van Dongen, posted by Daniel Tavernier/tabernarious)
# v2.2.4a - 20171215 - Added items to FUTURE list.
# v2.2.5 - 20171228 - Added notes about special characters in passwords. Added Deployment Information and ConfigSync sections. (Developed by Daniel Tavernier/tabernarious)
# v2.2.5a - 20180117 - Added items to FUTURE list.
# v2.2.5b4 - 20180118 - Moved encrypted values for SMB/CIFS to shell script which eliminates ConfigSync issues. Fixed long-password issue by using "-A" with openssl so that base64 encoded strings are written and read as a single line. (Developed by Daniel Tavernier/tabernarious)
# v2.2.5b4+ - 20180118 - Refining changes to SMB/CIFS and replicating to other remote copy types. (Developed by Daniel Tavernier/tabernarious)
# v3.0.0 - 20180124 - (Developed by Daniel Tavernier/tabernarious)
#                   - Eliminated ConfigSync issues and removed ConfigSync notes section. (Encrypted values now in $script instead of local file.)
#                   - Passwords now have no length limits. (Using "-A" with openssl which reads/writes base64 encoded strings as a single line.)
#                   - Added $script error checking for all remote backup types. (Using 'catch' to prevent tcl errors when $script aborts.)
#                   - Backup files are cleaned up after $script error due to new error checking.
#                   - Added logging. (Run logs sent to '/var/log/ltm' via logger command which is compatible with BIG-IP Remote Logging configuration (syslog). Run logs AND errors sent to '/var/tmp/scriptd.out'. Errors may include plain-text passwords which should not be in /var/log/ltm or syslog.)
#                   - Added custom cipher option for SCP.
#                   - Added StrictHostKeyChecking=no option.
#                   - Combined SCP and SFTP because they are both using SCP to perform the remote copy.
# v3.1.0 - 20180201 - (Developed by Daniel Tavernier/tabernarious)
#                   - Removed "app-service none" from iCall objects. The iCall objects are now created as part of the Application Service (iApp) and are properly cleaned up if the iApp is redeployed or deleted.
#                   - Reasonably tested on 11.5.4 HF2 (SMB worked fine using "mount -t cifs") and altered requires-bigip-version-min to match.
#                   - Fixing error regarding "script did not successfully complete: (can't read "::destination_parameters__protocol_enable": no such variable" by encompassing most of the "implementation" in a block that first checks $::backup_schedule__frequency_select for "Disable".
#                   - Added default value to "filename format".
#                   - Changed UCS default value for $backup_file_name_extension to ".ucs" and added $fname_noext.
#                   - Removed old SFTP sections and references (now handled through SCP/SFTP).
#                   - Adjusted logging: added "sleep 1" to ensure proper logging; added $backup_directory to log message.
#                   - Adjusted some help messages.
0
Comment made 25-Jan-2018 by Troy Murray 125

@tabernarious

Excellent job with the changelog included in the file. I love it! Thank you for compiling this and your work to improve this iApp.

Question, would you be willing to move this over to a GitHub repository for storing the code and tracking changes? This would also provide an issue tracker to make it easier for individuals to find a bug or resolution to a problem. I would be willing to help you maintain this.

If you'd like to chat about this you can DM on Twitter @tmurray_pro or find me in the F5 User Group Slack team at https://bit.ly/F5Slack

Thanks again!

0
Comment made 25-Jan-2018 by Robert Goetze 0

@tabernarious - Sorry for the delay, I was "watching" this thread and must have missed the change notification. Thank you for the update and posting of V3.0.0.

Thanks again.

0
Comment made 01-Feb-2018 by tabernarious 244

@Troy Murray, I created a Git repository for this iApp which contains my latest release (v3.1.0). I would welcome your collaboration. You can also find me on Twitter @tabernarious.

https://github.com/tabernarious/f5-automated-backup-iapp

0
Comment made 02-Feb-2018 by Troy Murray 125

@tabernarious This is excellent news! I've starred the repo and forked it on GitHub.

0
Comment made 16-Mar-2018 by Fiseha 0

Thanks Thomas for this excellent script. Is there a way you can add on the script to send email when the backup job is completed or failed.

1
Comment made 17-Mar-2018 by tabernarious 244

@Fiseha, The latest version (v3.x) (look in the comments for a link) includes logging to /var/log/ltm which will send to syslog. The easiest way to configure email alerts would be to have your syslog/reporting server send the email when it sees the relevant log message(s). There are other ways to get the F5 to send email directly but it’s very custom and I would not recommended it.

0