Data Replication Using rsync

Replication Review

Recently we discussed the importance of data replication for situations ranging from mission critical environments to home users. Replication ensures that you have a copy of your current data on a separate storage environment (secondary system) so that if you lose the first system (primary system), you still have access to the data.

In general there are two types of replication: synchronous and asynchronous. Synchronous replication, as the name implies, means that the primary storage and secondary storage are kept exactly the same. Any data writes (or deletes) have to complete on both the primary and secondary storage pools before returning to the application allowing it to continue. This means that the data in the data pools is an exact match.

Asynchronous replication allows any writes (or deletes) from an application to finish on the primary storage pool. Then the data is copied from the primary storage pool to the secondary storage pool, typically outside of the application I/O path. This means that there can be some data difference between the two pools at any instance in time. The amount of data difference you can tolerate is up to you (your requirements) but you can shrink that data difference to something fairly small and tolerable.

But one of the most important aspects of data replication that you shouldn’t forget is that data replication and data backup are not the same thing. A backup can keep prior versions of data so that you can effectively go back in time over the life of the data to get prior versions. On the other hand, replication keeps a replica (copy) of the current data. You can get the most current state of the data from a backup but it will only be as accurate as when the backup was made. Replicas are much more recent so they will, in general, capture the data changes since the last backup. The question you need to answer is when do I need replication?

There is no universal answer to that question. You need to examine your data requirements and determine how important having the latest copy of the data is to your mission and the importance of the accessibility of that data. During your examinations you should also weave in discussions about off-site disaster recovery for your data center (or your home system).

In the previous article about replication, two replication techniques for Linux were discussed – DRBD and rsync. DRBD is a kernel based replication method that automatically replicates all data from the primary storage pool to a secondary storage pool without the user or administrator having to intervene. On the the hand, rsync is a file based approach allowing you to selectively replicate directory trees so that you don’t have to replicate an entire file system. However, rsync mush be invoked manually so it’s not as automatic as DRBD increasing the possible data differences between the primary and secondary storage pools. But many times the flexibility of replicating only portions of the data to one or more secondary storage pools make it a popular choice despite the possibility of increased data differences.

Since rsync is so flexible and is file based, in this article I want to show a simple rsync example to illustrate what it can do for data replication.

Simple example using rsync

I want to present a simple rsync example to illustrate the basic steps involved in getting to replicate a directory tree. One of the advantages of rsync is that you don’t have to make a copy of the entire file system – you can just make a copy of a specific directory tree and even specific file types or file names. This makes it incredibly flexible since you can now replicate directory trees to various secondary storage servers as needed and/or focus on certain types of files or directory trees.

For this article I will be using rsync to replicate data from my laptop to my main storage box at home. The overall concept is that I come home from being on the road, I fire up the laptop on the home network and my data gets replicated to my home server. In addition, I will only be replicating a specific directory from the laptop to the home server since that is where I keep all the data I modify while I travel.

Let’s start by defining terms within the rsync framework. The first two systems or terms that we need to define are the rsync server and the rsync client. One would think that the traditional client/server terms would apply, but a common source of confusion in rsync is that the rsync server does not necessarily have to be the system that has the original copy of the data and the rsync client does not have to be the recipient of the data. To better understand rsync, remember that there is a distinction between roles and processes in rsync. So to make sure we understand all the terms used in this article, there are four terms we’ll be using (taken from this link).

• Client: This is a role within rsync where the client initiates the synchronization.
  
• Server: This is a role within rsync and refers to the remote process (system) that clients connect to either within a local transfer, a remote shell, or with a network socket.
  
• Sender: This is a role and a process within rsync and applies to the particular rsync process that has access to the original files being synchronized. So the “sender” process reads the data and sends it to the “receiver” process.
  
• Receiver: This is a role and a process within rsync and describes the receiving process that receives the updates to the data and writes it to the storage device (i.e. the secondary storage pool).
  

To be honest, things can get a little confusing between roles and processes but I like to keep things simple. So for this example where I’m replicating data from my laptop to my home server I will define the home server as the rsync server, and the laptop as the rsync client and the laptop is the rsync sender since it has the original data and the home server is the rsync receiver. The idea is that when I plug in the laptop to my home network, my data is automatically replicated to my home server so I have a copy in case the laptop dies.

A reasonably good tutorial to use to start learning rsync is here. It is a bit old (1999) but it has a very good overview of rsync and explains things fairly well. There are other tutorials that cover useful topics such as how to use ssh with rsync or using stunnel with rsync.

For the rsync command used for my simple scenario let’s start with the simple example in the “everythinglinux” tutorial. Here is the script I used to perform the rsync that is taken from the article and adapted to my situation (notice that there are few changes).

rsync --verbose --progress --stats --compress --rsh=/usr/bin/ssh \
  --recursive --times --perms --links --delete \
  --exclude "*bak" \
  /home/laytonjb/Documents/* 192.168.1.8:/data/laytonj/rsync_test