How to use rsync on mac os x, windows and linux

Introduction

Using online storage to backup personal files can be good, fast and allows you to have access to your important data wherever you're (as long as you have decent internet connectivity). Sometimes unfortunately online backups does not fit for the job, for example you may want to keep safe your:

huge MP3 collections or very heavy files
files that you trust to be only on your own computers or servers
datas on PCs without internet connectivity (for example on areas with only slow connectivity or such)
...

In this cases (and more you can think of) the rsync multi-platform open-source utility comes really handy. It handles, incremental backups via network (or even in the same machine) copying just the changed files and so limiting the bandwidth consumption. It's very fast, worldwide proved and has really few drawbacks (which you can easily address). You can use it with ssh to transfer the data in a secure way and you'll not need to open up more services through your firewall.

Installation

Mac OS X

You're quite lucky, rsync it's already included in your mac os x installation. It works out of the box and, for most of the "simple stuff" it will be ok.

If you want to have more features (such as preservation of metadata and wider character support -cross-platform latins, etc.-) you have to manually download ad install it. You'll need:

Basic use of Terminal
Apple Developer Tools (you can get them here)

First of all go to http://rsync.samba.org and find out the last available version, then fire up the terminal and type (example with the 3.0.2 version):

curl -O http://rsync.samba.org/ftp/rsync/rsync-3.0.2.tar.gz
tar -xzvf rsync-3.0.2.tar.gz
rm rsync-3.0.2.tar.gz
curl -O http://rsync.samba.org/ftp/rsync/rsync-patches-3.0.2.tar.gz
tar -xzvf rsync-patches-3.0.2.tar.gz
rm rsync-patches-3.0.2.tar.gz
cd rsync-3.0.2
patch -p1 < patches/fileflags.diff
patch -p1 < patches/crtimes.diff
./prepare-source
./configure
make
sudo make install

This will leave you with the tool installed, you can even delete the source folders.

Windows

In this case you can of course compile from scratch rsync but it may be faster to just download the executable (along with the ones for ssh).
Here you will find all the files needed, download them and put everything on a "rsync" directory on a path of your choice (for example int his document we will refer to C:\rsync).
You're done!

Linux

It's very easy, some distros have it already installed, you may check typing "rsync" on a terminal window. If it's not installed ("command not found") just use your packaging system to download it, for example on Ubuntu/Debian you should type on a terminal window:

sudo apt-get update
sudo apt-get install rsync

That was it.

Toggle this advertising

Setting it up

You may want to copy in various ways between one system and another. You should note that it's easier to transfer files between equal platforms: this will keep you safe from encoding problems and missing metadata.
Anyway you may still need to backup document cross-platform, don't worry you can do that. The easiest way to use rsync is to set up a script that does all the work and then call it whenever you need it (or even put it on a scheduled job -at night for example-).
If you're not used to scripts don't get scared, it may looks intimidating at the beginning but the effort will be compensated from the results.

Mac OS X

Open a text editor (like Smultron) and write on..

LOCK_FILE="/tmp/rsync_backup.lock"
LOG_FILE="/tmp/rsync_backup.log"
RSYNC="/usr/local/bin/rsync"

if [ -e $LOCK_FILE ]; then
	TIMESTAMP=$(date)
	echo "$TIMESTAMP: lock file exists, exit now" >> $LOG_FILE
	exit
fi

cat /dev/null > $LOG_FILE

touch $LOCK_FILE

This part just set up three variables:

LOCK_FILE: it will prevent the script to start twice
LOG_FILE: all output errors and messages will be stored on a proper log file
RSYNC: the path to the installed rsync we want to use

check if the LOCK_FILE exists and then erase the log file and get ready for the transfer.

Now you have to proceed typing all the rsync command you need. For example you may want to rsync some directories from your host and some other to the remote one; you may even need to exclude some specific file, a complete list of rsync capabilities can be read on the MAN page.

This example will copy all the data from the remote host /some/path/ directory to your ~/Documents/path/ directory, using UTF8-MAC character encoding on local machine and UTF-8 on remote:
```
$RSYNC --iconv=UTF8-MAC,UTF-8 -e ssh -az --delete-after user@remotehost:/some/path/ ~/Documents/path/
```
You can, of course, do the other way round, for example from ~ to the /path/mybackup/ directory, excluding the Movies folder:
```
$RSYNC -e ssh -az --delete-after -X --exclude 'Movies' ~ user@remotehost:/path/mybackup/
```

This last instruction is needed to remove the lock_file and enable the script to run freely again:

rm $LOCK_FILE

You can download the example rsync mac os x script in a convenient file.

To launch it, from terminal just type:

sh rsync-example-mac.sh

(or whatever name you changed it into).

To make it executable just type:

chmod +x rsync-example-mac.sh

and then launch typing:

./rsync-example-mac.sh

Linux

The procedure on Linux is fairly similar to the one on Mac OS X (or other *nix OS). You'll just have to write a script and launch it, let's look at it again, just for your convenience. Open a text editor (like pico, vim, gedit, emacs, kate, etc.) and write on..

LOCK_FILE="/tmp/rsync_backup.lock"
LOG_FILE="/tmp/rsync_backup.log"

if [ -e $LOCK_FILE ]; then
	TIMESTAMP=$(date)
	echo "$TIMESTAMP: lock file exists, exit now" >> $LOG_FILE
	exit
fi

cat /dev/null > $LOG_FILE

touch $LOCK_FILE

This part just set up two variables:

LOCK_FILE: it will prevent the script to start twice
LOG_FILE: all output errors and messages will be stored on a proper log file

This example will copy all the data from the remote host /some/path/ directory to your ~/Documents/path/ directory:
```
rsync -e ssh -az --delete-after user@remotehost:/some/path/ ~/Documents/path/
```
You can, of course, do the other way round, for example from ~ to the /path/mybackup/ directory, excluding the Movies folder:
```
rsync -e ssh -az --delete-after --exclude 'Movies' ~ user@remotehost:/path/mybackup/
```

This last instruction is needed to remove the lock_file and enable the script to run freely again:

rm $LOCK_FILE

You can download the example rsync linux script in a convenient file.

To launch it, from terminal just type:

sh rsync-example-linux.sh

(or whatever name you changed it into).

To make it executable just type:

chmod +x rsync-example-linux.sh

and then launch typing:

./rsync-example-linux.sh

Windows

Being rsync an UNIX designed tool on windows things can get easily quite messed up. Anyway, fear not, if you've already downloaded the executables files and putted on a nice path, let's move there and create a windows batch script.
As in the other OS, just use your preferred text editor (such as Notepad) and type on:

@echo off
echo Inizio il backup dal server.
set PATH=c:\rsync;%PATH%
set CYGWIN=binmode tty
set TERM=ansi
set HOME=c:\rsync
set RSYNC_RSH=ssh.exe
rsync -avz --delete-after user@remotehost:/path/ C:\backup\

You can download the example rsync windows bat script in a convenient file.

This script has to be saved as a .bat file and use the ssh client included in the downloaded files.
If you need to automatically halt the system after the procedure you can add another line: WMIC OS Where Primary=TRUE Call Shutdown.
As in the other examples you can refer to the rsync MAN page to change options and capabilities.

Toggle this advertising

Automate it

To automate the sync procedure the safest way is to use the OS already present scheduling system. On all 3 OS it's very simple, the only problem is that if you're transferring the data using ssh (such as in the example on this page) the procedure will get stucked asking for the password.

Getting around the password problem

To resolve this there are several ways:

On *nix systems (like on Mac or Linux): You should exchange the ssh private key to allow authentication without password, this is more simple showing than telling, fire up a terminal shell:
```
local$ cd ~
local$ ssh-keygen -t dsa -f .ssh/id_dsa
local$ scp .ssh/id_dsa.pub user@remote:~/.ssh/temp.pub
local$ ssh user@remote
remote$ cat .ssh/temp.pub >> .ssh/authorized_keys2
remote$ chmod 640 .ssh/authorized_keys2
remote$ rm .ssh/temp.pub
remote$ exit
```
the procedure above just create a new set of dsa keys and the public one gets copied on the remote host and added on the authorized ones.
On windows you can do the same exactly the same, you'll need to generate your private/public keys pair and add the public one on the auhtorized keys on the remote host. On a terminal window, move to the path where you putted the downloaded files (example: C:\rsync\) and type:
```
ssh-keygen -t rsa -N ''
```
This will generate the key, you'll need to copy that to the remote host and add to the authorized_keys2 file (see the *nix way above).

Scheduling the execution

*nix systems (both Linux and Mac OS X)

This procedure asume that you have already installed in your system the "cron" schedule manager, which is probably the case.
After logging in as the user that should call the script just type:

crontab -e

This will open a file in your text editor allowing you to add a new line related to the script in the cron syntax, for example:

5 1 */2 * * /script/backup

the above line would call the /script/backup every other day at 05:01 AM. To understand all the cron syntax (it can be a bit difficult at first) you should read some tutorials.

A simpler alternative is to directly place your script in a directory like /etc/cron.daily/, which will execute the script daily at a predefined time (this is not present in all *nix systems)

Windows

Windows already provide you a scheduling system. You just have to click on start, open the "run" dialog and exec control schedtasks (you can access this also via the control panel), you will be prompted by this window:

The rest of the procedure is simple, just follow the wizard, browse for your script file and choose the time for the automated procedure to run.

This article is made from personal notes and is not intended to be a comprehensive tutorial, you can indeed find more information about each subjects on the web.