Skip to content. | Skip to navigation

Personal tools
You are here: Home TBSI Technology Blog Using rsync with an Amazon EC2 Fedora 8 image

Using rsync with an Amazon EC2 Fedora 8 image

by Eric Smith — last modified Jan 30, 2010 07:10 AM

Amazon provides a number of Fedora 8 images. Unfortunately the provided kernels cause a problem with rsync. Find out how to resolve the problem.

We've recently started investigating Amazon EC2 for some of our computing needs. So far our progress has been excellent. I've been focusing on using the Amazon-supplied Fedora 8 (F8) images, in particular ami-48aa4921, although the problem I describe here applies to all of the F8 AMIs that Amazon provides.

One significant roadblock has been a problem with rsync. In particular, we use the excellent dirvish for our online backups. Unfortunately, Amazon uses the 2.6.21 kernel in its F8 images and this version does not support the lutimes system call. The version of rsync that comes with F8 uses lutimes to set the modification time on directories. lutimes isn't available until the 2.6.22 kernels. For more information on the issue with rsync and lutimes, see the rsync bug entry.

The symptom is errors in the dirvish rsync_error log files of the form:

rsync: failed to set times on "<directory-name>": Function not implemented (38)

Dirvish sees these as fatal errors and marks the images as failed. This prevents dirvish from performing its incremental backups.

Because rsync does not have a runtime switch to ignore lutimes, the easiest way to solve this is to produce a version of rsync that doesn't use the call at all. Unfortunately rsync does not have an autoconf switch to turn off lutimes, so I had to patch configure.in and rebuild. The change is simple, here's the diff I use:

--- rsync-2.6.9/configure.in.orig        2010-01-29 15:37:35.000000000 -0500
+++ rsync-2.6.9/configure.in    2010-01-29 15:38:07.000000000 -0500
@@ -528,7 +528,7 @@
 AC_FUNC_UTIME_NULL
 AC_FUNC_ALLOCA
 AC_CHECK_FUNCS(waitpid wait4 getcwd strdup chown chmod lchmod mknod mkfifo \
-    fchmod fstat ftruncate strchr readlink link utime utimes lutimes strftime \
+    fchmod fstat ftruncate strchr readlink link utime utimes strftime \
     memmove lchown vsnprintf snprintf vasprintf asprintf setsid glob strpbrk \
     strlcat strlcpy strtol mallinfo getgroups setgroups geteuid getegid \
     setlocale setmode open64 lseek64 mkstemp64 mtrace va_copy __va_copy \

I created a new RPM for rsync. Here's the diff for the .spec file:

--- rsync.spec.orig     2008-04-09 10:36:56.000000000 -0400
+++ rsync.spec  2010-01-29 22:21:13.000000000 -0500
@@ -1,7 +1,7 @@
 Summary: A program for synchronizing files over a network.
 Name: rsync
 Version: 2.6.9
-Release: 5%{?dist}
+Release: 5%{?dist}.trueblade.0
 Group: Applications/Internet
 # TAG: for pre versions use
 #Source:       ftp://rsync.samba.org/pub/rsync/rsync-%{version}pre1.tar.gz
@@ -10,6 +10,7 @@
 Patch1: rsync-2.6.9-acl-xattr-delete-bug.patch
 Patch2: rsync-2.6.9-hlink-segv.patch
 Patch3: rsync-3.0.1-xattr-alloc.diff
+Patch4: rsync-2.6.9-disable-lutimes.patch
 BuildRequires: libacl-devel, libattr-devel, autoconf, make, gcc, popt-devel
 Prefix: %{_prefix}
 BuildRoot: /var/tmp/%{name}-root
@@ -33,6 +34,7 @@
 %patch1 -p1 -b .acl_xattrs_bug
 %patch2 -p1 -b .hlink_segv
 %patch3 -p1 -b .xattr-alloc
+%patch4 -p1 -b .lutimes
 
 %build
 rm -fr autom4te.cache
@@ -62,6 +64,10 @@
 %{_mandir}/man5/rsyncd.conf.5*
 
 %changelog
+* Fri Jan 29 2010 Eric V. Smith <eric@trueblade.com> 2.6.9-5.fc8.trueblade.0
+- Added patch4 to remove lutimes, since the EC2 kernel in ami-48aa4921 does
+  not support it.
+
 * Tue Apr  8 2008 Simo Sorce <ssorce@redhat.com> 2.6.9-5.fc8
 - Security release: http://rsync.samba.org/security.html#s3_0_2 

Once I had the new RPM, I signed it and added it to our local RPM repository. Because it has a newer version than the one supplied with F8, it will automatically be picked up by "yum update".

The only remaining complication is that the AMI I'm using supplies its own copy of rsync in /usr/local/bin, in addition to the one supplied by default in /usr/bin. I'm not sure why Amazon did this, because the /usr/local/bin version has the same problem as the /usr/bin one. The /usr/local/bin version does not come from an RPM, so I just delete it using our automated server configuration tool. The files to delete are:

/usr/local/bin/rsync
/usr/local/man/man1/rsync.1
/usr/local/man/man5/rsyncd.conf.5
/usr/local/share/man/man1/rsync.1
/usr/local/share/man/man5/rsyncd.conf.5

Once the new RPM updates rsync and the unneeded /usr/local files are deleted, rsync and dirvish are again working correctly.

I'd rather solve this problem by upgrading the kernel to 2.6.22 or newer, but upgrading the kernel is a non-trivial task with AWS. I'd rather let Amazon handle that issue and instead focus on using the provided images. This way we can more easily upgrade when Amazon produces newer AMIs.

For a number of other approaches (but with fewer specifics), see this thread.

Document Actions