Automated off-site Linux Backups using Duply and Duplicity

Off-site backups are important, and even though I know this, I rarely implement them in my own servers. Lately, I’ve been setting up rsnapshot to do hourly and daily backups locally(to the same server), and I only do manual backups to remote servers occasionally. I decided to install duply on all of my servers/virtual machines(that I care about) and have them back up to a single backup server. This backup server will also do daily encrypted backups to Amazon S3, effectively giving me 3 redundant layers of backups.

If you haven’t heard of Duplicity or Duply before, Duply is basically a wrapper for Duplicity which makes it easier to manage. Duplicity itself is similar to rsnapshot except, it uses tar to efficiently store differences between backups (instead of hardlinks). Here’s the description from the man page:

Duplicity incrementally backs up files and directory by encrypting tar-format volumes with GnuPG and uploading them to a remote (or local) file server. Currently local, ftp, ssh/scp, rsync, WebDAV, WebDAVs, HSi and Amazon S3 backends are available. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Currently duplicity supports deleted files, full Unix permissions, directories, symbolic links, fifos, etc., but not hard links.

I wrote this mainly as a reference for myself when I need to set duply up on another server, but it might be useful for others as well.

CentOS 5 / 6 Instructions

Install the EPEL repo:

#Cent 6: rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm #Cent 5: rpm -Uvh http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

Install duplicity:

yum --enablerepo=epel install duplicity

Install duply:
Get the URL for the latest version here: http://duply.net/?title=Duply-downloads
download it to your server, extract it, and copy duply to /usr/local/bin/duply then chmod +x /usr/local/bin/duply

wget http://dev.justynshull.com/duply_1.5.5.1.tgz tar xvzf duply_1.5.5.1.tgz cp duply_1.5.5.1/duply /usr/local/bin/duply chmod +x /usr/local/bin/duply

Note: Bug 675234 is a request to have duply put into the Fedora repo, but there is also a .spec and source rpm if you wish to build an rpm yourself.

Setup a basic Duply profile

mkdir /etc/duply && chmod 700 /etc/duply duply testvm1 create

Note: If you don’t create /etc/duply, then it will use $HOME/.duply by default.

Take a look at the options in /etc/duply/testvm1/conf and configure it to your liking. There are many different TARGET formats you can use, including ssh, rsync over ssh, ftp, and even amazon’s S3. You can view them all here: URL Formats
This is what mine usually look like(without encryption) using rsync over ssh:

# egrep -v '^#|^$' /etc/duply/testvm1/conf GPG_KEY='disabled' TARGET='rsync://backups.justynshull.com//home/testvm1/backups' TARGET_USER='testvm1' SOURCE='/' MAX_AGE=3M MAX_FULL_BACKUPS=3 MAX_FULLBKP_AGE=30D VOLSIZE=3500 DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE " DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE " DUPL_PARAMS="$DUPL_PARAMS --include=/etc \         --include=/home \         --include=/root \         --include=/var/www \         --include=/var/lib/mysql \         --include=/var/log \         --exclude=/** "

I prefer to use multiple –include= options rather than fill /etc/duply/testvm1/exclude with every directory I *don’t* want backed up. Either way will work though. Also, if you’re going to use rsync or ssh/sftp, I’d recommend setting up the backup server so that you can log in with ssh keys and generate a separate key for each server you’re backing up from. You’ll also have to have ssh’d into the backup server as that user at least once to avoid errors about the target host key.

Encryption
Duply/Duplicity supports using GPG to encrypt volumes before uploading them to the remote server, and the easiest way to enable encryption is by putting this in your conf:

#comment out #GPG_KEY from earlier #GPG_KEY='disabled' GPG_PW='secret_password'

This will encrypt the volumes using the passphrase you put in GPG_PW, but you can refer to the documentation for how to set it up to use actual gpg keys.

Including MySQL Backups
If you’re running mysql on the server, you should consider adding something similar to this to /etc/testvm1/pre, which gets run automatically by duply, to dump all databases before backing up the server.

#!/bin/sh mkdir -pv /root/db_backups for db in $(mysql -uroot -e 'show databases' -s --skip-column-names | grep -v 'information_schema'); do         mysqldump -uroot $db > /root/db_backups/$db.sql;         sleep 10; done

Run your first backup:

duply testvm1 backup

Automate it
If all goes well(no errors), then you should be okay to set up a cronjob to run duply backup.

# crontab -l 30      3       *       *       *       /usr/local/bin/duply testvm1 backup 30      5       *       *       sun       /usr/local/bin/duply testvm1 backup_verify_purge --force

Purge Old Backups
If you use the above crontab, the 2nd line will run once a week, purging old backups from the remote server. If you’re worried about keeping too many backups, you might want to increase how often this runs and also decrease the options in the duply profile configuration.

Verify Backups
Duply makes it easy to see what you’re currently backing up.
To see a list of backups stored on the remote server:

duply testvm1 status

To see a list of files that have changed since the last backup:

duply testvm1 verify

List all files in a backup yesterday(leave out to show latest):

duply testvm1 list 1D

Restore Backups
It’s just as easy to restore complete or partial backups.
Restore the entire latest backup to /tmp/restore:

duply testvm1 restore /tmp/restore

Restore backup from 7 days ago to /tmp/restore:

duply testvm1 restore /tmp/restore 1W

Restore single file or directory to /tmp/restore:

duply testvm1 fetch home/justyns /tmp/restore #When using 'fetch', make sure you leave off the leading slash.

Restore a file from a month ago:

duply testvm1 fetch home/justyns/plans_for_world_dom.txt /home/justyns/plans_for_world_dom.txt 1M

Monitoring VMware ESXi and vSphere with Nagios

Requirements

• Perl 5.8
• Several supporting Perl modules:
• Crypt-SSLeay (0.51) [Crypt::SSLeay]
• Data-Dumper (2.102) [Data::Dumper]
• MethodMaker (2.0.8) [Class::MethodMaker]
• XML-LibXML (1.60) [XML::LibXML]
• libwww-perl (5.805) [LWP]

This article describes how to monitor a VMWare ESXi or vSphere host with Nagios, using the OP5 Check ESX Plugin written in PERL. The plugin can monitor either a single ESXi/vSphere server or a VirtualCenter/vCenter Server and individual virtual machines. We’ll see here how to monitor an ESXi 4 host.
The following tutorial has been made on a CentOS server, you may have to adapt some paths with other distributions.

Install all the dependencies of Perl SDK.

perl -MCPAN -e shell

The above command shall bring you to the perl cpan install CLI. This will allow you to install requirements for Perl.

Example:

cpan> install Crypt::SSLeay

This article describes how to monitor a VMWare ESXi or vSphere host with Nagios, using the OP5 Check ESX Plugin written in PERL. The plugin can monitor either a single ESXi/vSphere server or a VirtualCenter/vCenter Server and individual virtual machines. We’ll see here how to monitor an ESXi 4 host.

The following tutorial has been made on a CentOS server, you may have to adapt some paths with other distributions.

Installation

The prerequisite for the plugin to work is to install the VMWare Perl SDK available on the manufacturer website.
Download the file on your server, for example in the root directory, untar it and run the installer that way :

# cd /root # tar xvzf VMware-vSphere-Perl-SDK-4.1.0-254719.i386.tar.gz  # cd vmware-vsphere-cli-distrib/ # ./vmware-install.pl

"Creating a new vSphere CLI installer database using the tar4 format.

Installing vSphere CLI.

You must read and accept the vSphere CLI End User License Agreement to continue.
Press enter to display it."

"Read through the License Agreement"

"Do you accept? (yes/no)"

yes

"In which directory do you want to install the executable files?
[/usr/bin]"

"The following Perl modules were found on the system but may be too old to work
with vSphere CLI:

Crypt::SSLeay
Compress::Zlib

The installation of vSphere CLI 4.0.0 build-161974 for Linux
completed successfully. You can decide to remove this software from your system
at any time by invoking the following command:
"/usr/bin/vmware-uninstall-vSphere-CLI.pl".

Enjoy,

--the VMware team"

If ever you're having an issue installing the SDK and complaining about http_proxy, issue the following command:

Run commands before vmware-install.pl:

export http_proxy=

export ftp_proxy=

Download nagios check plugin check_esx3.pl from op5.com

http://www.op5.org/community/plugin-inventory/op5-projects/op5-plugins

Follow the instructions given by the script. Depending on your setup, some PERL dependencies must be installed prior for the SDK to work correctly. When it’s done, we can get the plugin here, and copy it to /usr/lib/nagios/plugins/. Make it executable :

# cd /usr/lib/nagios/plugins/ # chmod a+x check_esx

Take NOTE: change check_esx to check_esx3.pl

Configuration

Now, we can start the real configuration for Nagios. We’ll need a username and password to access the ESXi host, let’s define those Nagios variables in a safe place in /etc/nagios/resource.cfg, so that this information will be hidden from the CGIs :

$USER11$=username $USER12$=password

In this tutorial, we’ll be monitoring these resources : CPU, memory usage, net usage, runtime status and IO/read/write. But some more are available, see the references here. Below are the new commands related to ESXi to add in the /etc/nagios/objects/command.cfg file (these are the ESXi related commands only, NOT the full command.cfg, you may append this at the end of the file) :

# check vmware esxi machine # check cpu define command{         command_name check_esx_cpu         command_line $USER1$/check_esx -H $HOSTADDRESS$ -u $USER11$ -p $USER12$ -l cpu -s usage -w $ARG1$ -c $ARG2$         }   # check memory usage define command{         command_name check_esx_mem         command_line $USER1$/check_esx -H $HOSTADDRESS$ -u $USER11$ -p $USER12$ -l mem -s usage -w $ARG1$ -c $ARG2$         }   # check net usage define command{         command_name check_esx_net         command_line $USER1$/check_esx -H $HOSTADDRESS$ -u $USER11$ -p $USER12$ -l net -s usage -w $ARG1$ -c $ARG2$         }   # check runtime status define command{         command_name check_esx_runtime         command_line $USER1$/check_esx -H $HOSTADDRESS$ -u $USER11$ -p $USER12$ -l runtime -s status         }   # check io read define command{         command_name check_esx_ioread         command_line $USER1$/check_esx -H $HOSTADDRESS$ -u $USER11$ -p $USER12$ -l io -s read -w $ARG1$ -c $ARG2$         }   # check io write define command{         command_name check_esx_iowrite         command_line $USER1$/check_esx -H $HOSTADDRESS$ -u $USER11$ -p $USER12$ -l io -s write -w $ARG1$ -c $ARG2$         }

And an example of the configuration for a Nagios host called esxi01 in /etc/nagios/hosts/esxi01.cfg :

# Host esx01 define host{         use                     linux-server         host_name               esxi01         alias                   VMWare ESXi 01         address                 192.168.1.100         }   # Define a service to "ping" the local machine define service{         use                             generic-service         host_name                       esxi01         service_description             PING         check_command                   check_ping!100.0,20%!500.0,60%         }   # VMWare # check cpu define service{         use                             generic-service         host_name                       esxi01         service_description             ESXi CPU Load         check_command                   check_esx_cpu!80!90         }   # check memory usage define service{         use                             generic-service         host_name                       esxi01         service_description             ESXi Memory usage         check_command                   check_esx_mem!80!90         }   # check net define service{         use                             generic-service         host_name                       esxi01         service_description             ESXi Network usage         check_command                   check_esx_net!102400!204800         }   # check runtime status define service{         use                             generic-service         host_name                       esxi01         service_description             ESXi Runtime status         check_command                   check_esx_runtime         }   # check io read define service{         use                             generic-service         host_name                       esxi01         service_description             ESXi IO read         check_command                   check_esx_ioread!40!90         }   # check io write define service{         use                             generic-service         host_name                       esxi01         service_description             ESXi IO write         check_command                   check_esx_iowrite!40!90         }

It’s done. Restart Nagios and wait a while (or re-schedule) for the new resources to be monitored.

NuSOAP WSDL Complex Data

Source: http://www.koopman.me/2008/01/nusoap-wsdl-service-return-array-of-complex-data/

I’m just getting the code in the post for now… hopefully I’ll have time to come back and document it for you. I am excluding my config.php on purpose, you’ll need to define you’re own constants in there.

require_once(“config.php”);

require_once(“nusoap.php”);

/*

* This function takes an $email address and returns an array of email addresses

* that are the given email address’s recent contacts from their address book.

*

* @param string $email (the email address of the user)

* @param string $token (a password that is used for authentication for use of this function, it is NOT the email users password.)

*/

function getAutocompleteContacts( $email, $num, $token ) {

if ($token != SOAP_TOKEN)

return new soap_fault(‘Server’, ”, “Supplied token does not match our records”,”);

$email = strtolower(trim($email));

if ( ! ereg( ‘^’.email_reg.’$', $email) )

return new soap_fault(‘Client’, ”, “Must supply a valid email address: $email not valid”,”);

$aDB = DB::connect( ADDR_DSN );

if ( DB::isError($aDB) ) {

return new soap_fault(‘Server’, ”, ‘Service temporarily unavailable: could not connect to ADDR_DSN DB’,”);

}

$aDB->setFetchMode( DB_FETCHMODE_ASSOC );

// REALLY YOU SHOULD DO YOU’RE OWN QUERY, BUT FOR THE SAKE

// OF THIS EXAMPLE, I’M JUST GOING TO INJECT A COUPLE RESULTS:

$result = array();

$result[] = array( ‘contact’ => ‘Chaos Captain’, ‘email’ => ‘choas@sdfusidfousdf.com’);

$result[] = array( ‘contact’ => ‘Joe Joe’, ‘email’ => ‘choas@sdf768sdf798s7df987.com’);

return $result;

#return new soap_fault(‘Server’, ”, ‘Fallthrough error, should have faulted on invalid type above’,”);

}

$HTTP_RAW_POST_DATA = isset($HTTP_RAW_POST_DATA) ? $HTTP_RAW_POST_DATA : ”;

$server = new soap_server;

$server->configureWSDL(‘addressbook1′, ‘urn:’.$_SERVER['SCRIPT_URI']);

$server->wsdl->addComplexType(

‘Contact’,

‘complexType’,

‘struct’,

‘all’,

”,

array(

‘contact’ => array(‘name’ => ‘contact’, ‘type’ => ‘xsd:string’),

‘email’ => array(‘name’ => ‘email’, ‘type’ => ‘xsd:string’),

)

);

$server->wsdl->addComplexType(

‘ContactArray’,

‘complexType’,

‘array’,

”,

‘SOAP-ENC:Array’,

array(),

array(

array(‘ref’=>’SOAP-ENC:arrayType’,'wsdl:arrayType’=>’tns:Contact[]‘)

),

‘tns:Contact’

);

$server->register(‘getAutocompleteContacts’,

array(‘email’ => ‘xsd:string’, ‘num’ => ‘xsd:int’, ‘token’ => ‘xsd:string’), // input parameters

array(‘return’ => ‘tns:ContactArray’),

‘urn:’.$_SERVER['SCRIPT_URI'], // namespace

‘urn:’.$_SERVER['SCRIPT_URI'].”#getAutocompleteContacts”, // soapaction

‘rpc’, // style

‘encoded’, // use

‘Fetch array of address book contacts for use in autocomplete’); // documentation

#$server->wsdl->schemaTargetNamespace = $_SERVER['SCRIPT_URI'];

$server->service($HTTP_RAW_POST_DATA);

exit();

?>


Client Code:



require_once(‘config.php’);

require_once(‘nusoap.php’);

$client = new soapclient(SOAP_SERVER_WSDL_URL, true);

// Check for an error

$err = $client->getError();

if ($err) {

// Display the error

echo ‘

Constructor error

’ . $err . ‘

’;

// At this point, you know the call that follows will fail

}

$result = $client->call(‘getAutocompleteContacts’, array(‘email’=>’something@asdf97s9d8f7sdf.com’, ‘num’=>5, ‘token’=>SOAP_TOKEN) );

print_r($result);

?>


Result is as follows:



Array

(

[0] => Array

(

[contact] => Chaos Captain

[email] => choas@sdfusidfousdf.com

)

[1] => Array

(

[contact] => Joe Joe

[email] => choas@sdf768sdf798s7df987.com

)

)