Synchronize Windows and Linux with Unison

I recently added Windows to my Linux development environment.  I was already using cross-platform Unison to synchronize files between my Linux machines, so I installed Unison on the Windows machines and tried to synchronize.  Since Linux has a more powerful file system with more permissive file naming rules, I had to do some work to get my Linux files to fit into a Windows file system.

In this article, I tell you how to use Unison to synchronize files in a cross-platform environment, and how to find and reconcile the incompatibilities between Linux and Windows file trees so they can be synchronized.

Install Unison and Try Synchronizing

First, back up your files using your usual method, not Unison. You will inevitably need to rearrange your files, and you could make a mistake. While Unison might seen similar to a backup solution, it is not, especially at the beginning when you are likely to forget to synchronize directories that you really should.

Did I mention the part about how you should back up your files?

Next, install Unison on Linux and Windows. I used Unison 2.27.

• With most modern Linux distributions, you can install Unison from your usual repository with sudo yum install unison or the equivalent apt-get command.
• Windows is more complicated.  You will need Unison, GTK+, and an ssh client.  The details of this installation are beyond the scope of this article.  Consult other resources on the web.

Choose one Linux machine as the server to which you will synchronize all other machines. Enable the ssh daemon on the server and make sure you can ssh to it normally.

All your other machines are clients.  These are the machines on which you will run Unison.  They will contact the server as ssh clients.

Launch Unison on one of the client machines and try it out.  Choose a very small hierarchy of files, and make sure you can synchronize them.

Build Unison Preference Files

As you read this you will need to refer to the Unison File Synchronizer User Manual and Reference Guide.

When you tried synchronizing for the first time, Unison made a preference file named:

• ~/.unison/default.prf on UNIX
• C:\Users\userName\.unison\default.prf on Windows 7
• C:\Documents and Settings\userName\.unison\default.prf on Windows XP

Use this default.prf file as a model to create Unison preference files that do everything you want.  For each combination of,

• File hierarchy to synchronize
• Operating system you use as a client

you will need one Unison preference file. Here are the preference files I have to synchronize my home directory (“Data”) and configuration files (“AppConfig”) files on Linux, Windows 7 and Windows XP clients:

.unison/unixData.prf
.unison/winXPAppConfig.prf
.unison/winXPData.prf
.unison/win7AppConfig.prf
.unison/win7Data.prf

There are only a few configuration files common to Windows and Linux, so presently my winXPAppConfig.prf preference file just synchronizes Windows C:\Documents and Settings\voom\.unison with Linux /home/voom/.unison/.  There is no unixAppConfig.prf because unixData.prf takes care of synchronizing the .unison/ directory.

Since the winXPAppConfig.prf preference file is simple and useful (you will use it many times as you develop and propagate your Unison preference files to your other machines), let’s have a look at it first:

root = C:\Documents and Settings\voom
# rogue is the server
root = ssh://voom@rogue//home/voom

# You may need some ssh options. Sorry, this is beyond the scope of this article.
# sshcmd=ssh.exe
# sshargs= someArgumentsToTheSshCommand

# Paths to synchronize
path = .unison

# Do not synchronize the ar* files
ignore = Name .unison/ar*

This specifies the roots of the synchronization on the local machine and the server. It specifies that path .unison is to be synchronized. Unison keeps its synchronization database in files named ar*.  I do not want to synchronize them, so those names are ignored. Again, refer to the Unison manual for details.

Now, on to the big granddaddy of them all, .unison/winXPData.prf. This Unison preference file synchronizes the Windows C:\Documents and Settings\voom\My Documents folder with Linux /home/voom:

# Roots of the synchronization
root = C:\Documents and Settings\voom\My Documents
root = ssh://voom@rogue//home/voom

# You may need some ssh options. Sorry, this is beyond the scope of this article.
# sshcmd=ssh.exe
# sshargs= someArgumentsToTheSshCommand

# Paths to synchronize
path = Desktop
path = Documents
path = Music
path = Pictures
…

How to Handle Problems

When you actually try running Unison, it will inevitably encounter things about your Linux files that are not supported on Windows file systems.  These errors are reported in C:\Users\userName\unison.log on Windows 7 or C:\Documents and Settings\userName\unison.log on Windows XP.

I will also show you Unix commands that quickly inspect your UNIX file tree for problems. This allows you to address the issues before a time-consuming Unison run.

There are three major problems that you will find: symbolic links, ‘:‘ in names, and case sensitivity.

Windows Does Not Handle Symbolic Links

Windows does not support symbolic links.  When Unison encounters one during synchronization, it gives the error:

src/gmock-1.5.0/lib/.libs/libgmock.so.0
Synlink not supported under Win32

Note that this error occurs during synchronization–you do not ordinarily see it before synchronizing.

Rather than waiting around for Unison to find them, the following UNIX shell command will instantly print all the symbolic links in a Linux directory tree rooted at dirName:

find dirName -type l

There are two solutions to the link problem:

• Symbolic links are a sneaky way of making files appear to be where they are not.  If you’re being sneaky, you someday might outsmart yourself.  Consider just eliminating the symbolic link.
• A legitimate use of symbolic links is to give a single library multiple names, like linking libgmock.so.0 to libgmock.so.0.0.0.  These are Linux libraries that Windows can’t load anyway, so I ignored the library links in my home directory using:
  

  # Ignore links to Linux libraries
ignore = Name {*.so,*.so.[0-9],*.a,*.la}

Windows Does Not Allow ‘:‘ in File Names

Windows does not allow the characters \ / : * ? " < and > in file names.  Of these, the only one that might reasonably appear in a UNIX file name is ‘:‘ (colon).  When it encounters one, Unison gives the error:

The name of this Unix file is not allowed in Windows (mower/testdata/10M/CEL/blockHead:2)

You must either rename this file in Unix, or ignore it when synchronizing against Windows.

Readers in the IC design community will recognize the above example as a Synopsys Milkyway database file.  Milkyway-based tools do not run on Windows, so I just ignored all Milkyway files with this line in my Unison preference file:

# Ignore Milkyway data files
ignore = Regex .*:[0-9]+.*

But those of you who are not chip designers don’t care about Milkyway. You can probably just rename the UNIX file to eliminate the ‘:‘.

Rather than using Unison to find them, you can find all the files containing ‘:‘ in a Linux directory tree using the Unix command:

find dirName -name '*:*'

Windows File Names Are Case Insensitive

The UNIX file system is case-sensitive, so that ABC and abc are unique file names.  Windows is case insensitive, so it cannot distinguish such file or folder names.  When it encounters such a situation, Unison gives the error:

Two or more files on a case-sensitive system have names identical except for case.  They cannot be synchronized to a case-insensitive file system.  (Desktop/Library/python-2.6.5)

You need to stretch the Unison window quite a bit in order to see the troublesome file name at the end of this message.

Rather than relying on Unison to find them, you can find all file names that are identical except for case using the Unix commands:

find dirName | sort -uf > caseSensitive.txt
find dirName | sort -u | sort -f > caseInsensitive.txt
tkdiff caseSensitive.txt caseInsensitive.txt &

Of course you can use ordinary diff in place of tkdiff if you like.  tkdiff plainly shows that directories Python-2.6.5 and python-2.6.5 are identical except for the case of the “P”:

Use Include Files

Once you get serious about cross-platform synchronization, you will find it useful to put the common portion of your Unison preference files into separate files and reuse them with the include statement.  For example, here is the .unison/winXPData.prf file that I use on Windows XP to synchronize with my Linux server:

# Roots of the synchronization
root = C:\Documents and Settings\voom\My Documents
root = ssh://voom@rogue//home/voom
# You may need some ssh options. Sorry, this is beyond the scope of this article.
# sshcmd=ssh.exe
# sshargs= someArgumentsToTheSshCommand

# All the stuff that would ever be synchronized
include _dataCommon.prf

# For each platform involved, ignore the stuff that should not be synchronized
include _dataWindows.prf
include _dataUnix.prf

# Files to be ignored in every synchronization
include _alwaysIgnore.prf

# Explanation that will appear in the Profiles dialog
label = Synchronize Windows XP My Documents

My .unison/unixData.prf file synchronizes my Linux laptop with my Linux server. It is very similar to the above file, but it omits _dataWindows.prf:

# Roots of the synchronization
root = /home/voom
root = ssh://voom@rogue//home/voom

# All the stuff that would ever be synchronized
include _dataCommon.prf

# For each platform involved, ignore the stuff that should not be synchronized
include _dataUnix.prf

# Files to be ignored in every synchronization
include _alwaysIgnore.prf

# Explanation that will appear in the Profiles dialog
label = Synchronize Linux home directory

If I want to add a new path to the synchronization, I just add it to .unison/_dataCommon.prf, and all platforms will synchronize it.  If I discover some little turd that I never want to synchronize, I add it to .unison/_alwaysIgnore.prf.  For example, here is my current .unison/_alwaysIgnore.prf file:

# Text editor backup files
ignore = Name *~
ignore = Name .*~
ignore = Name *.swp
ignore = Name .#*
# Trash
ignore = Path .Trash
ignore = Path Trash
# My own naming convention for files that can be deleted any time
ignore = Path temp
ignore = Name temp.*
ignore = Name *.tmp
# Log and Journal files
ignore = Name *.log.??_??_??_??
ignore = Name *.cmd.??_??_??_??

Caution Regarding Include Files

The Unison include mechanism is far from multi-platform.  I found these problems:

• Unison for UNIX cannot parse include statements in files that use Windows CRLF line terminators, resulting in errors like,
  

  File “unixData”, line 3: Garbled ‘include’ directive: include _dataUnix.prf

  or

  Preference file _dataUnix.prf not found

  Fortunately, the Unison include command for Windows does work with UNIX newline line terminators.  Therefore, to use the include command, all your .prf files must be UNIX style.  Convert them with the UNIX command,

  dos2unix ~/.unison/*.prf

• You might like to create a .unison/include/ directory to hold all the included .prf files.  This is not possible because on Windows, the file separator must be ‘\‘, and on UNIX it must be ‘/‘,
  

  include include/dataCommon.prf # Works only on UNIX
include include\dataCommon.prf # Works only on Windows

  yet both platforms must read this file.  Therefore, all the included .prf files must be in the .unison/ directory.

Tagged as: cross-platform, system administration, Unison