Music of Distant Lands 2010 Concert, featuring my daughter on oboe.  Photo by Kathryn Carpenter.

Synchronize Windows and Linux with Unison

by John McGehee on January 9, 2011

I recently added Windows to my Linux development environment.  I was already using cross-platform Unison to synchronize files between my Linux machines, so I installed Unison on the Windows machines and tried to synchronize.  Since Linux has a more powerful file system with more permissive file naming rules, I had to do some work to get my Linux files to fit into a Windows file system.

In this article, I tell you how to use Unison to synchronize files in a cross-platform environment, and how to find and reconcile the incompatibilities between Linux and Windows file trees so they can be synchronized.

Install Unison and Try Synchronizing

First, back up your files using your usual method, not Unison. You will inevitably need to rearrange your files, and you could make a mistake. While Unison might seen similar to a backup solution, it is not, especially at the beginning when you are likely to forget to synchronize directories that you really should.

Did I mention the part about how you should back up your files?

Next, install Unison on Linux and Windows. I used Unison 2.27.

  • With most modern Linux distributions, you can install Unison from your usual repository with sudo yum install unison or the equivalent apt-get command.
  • Windows is more complicated.  You will need Unison, GTK+, and an ssh client.  The details of this installation are beyond the scope of this article.  Consult other resources on the web.

Choose one Linux machine as the server to which you will synchronize all other machines. Enable the ssh daemon on the server and make sure you can ssh to it normally.

All your other machines are clients.  These are the machines on which you will run Unison.  They will contact the server as ssh clients.

Launch Unison on one of the client machines and try it out.  Choose a very small hierarchy of files, and make sure you can synchronize them.

Build Unison Preference Files

As you read this you will need to refer to the Unison File Synchronizer User Manual and Reference Guide.

When you tried synchronizing for the first time, Unison made a preference file named:

  • ~/.unison/default.prf on UNIX
  • C:\Users\userName\.unison\default.prf on Windows 7
  • C:\Documents and Settings\userName\.unison\default.prf on Windows XP

Use this default.prf file as a model to create Unison preference files that do everything you want.  For each combination of,

  • File hierarchy to synchronize
  • Operating system you use as a client

you will need one Unison preference file. Here are the preference files I have to synchronize my home directory (“Data”) and configuration files (“AppConfig”) files on Linux, Windows 7 and Windows XP clients:

.unison/unixData.prf
.unison/winXPAppConfig.prf
.unison/winXPData.prf
.unison/win7AppConfig.prf
.unison/win7Data.prf

There are only a few configuration files common to Windows and Linux, so presently my winXPAppConfig.prf preference file just synchronizes Windows C:\Documents and Settings\voom\.unison with Linux /home/voom/.unison/.  There is no unixAppConfig.prf because unixData.prf takes care of synchronizing the .unison/ directory.

Since the winXPAppConfig.prf preference file is simple and useful (you will use it many times as you develop and propagate your Unison preference files to your other machines), let’s have a look at it first:

root = C:\Documents and Settings\voom
# rogue is the server
root = ssh://voom@rogue//home/voom

# You may need some ssh options. Sorry, this is beyond the scope of this article.
# sshcmd=ssh.exe
# sshargs= someArgumentsToTheSshCommand

# Paths to synchronize
path = .unison

# Do not synchronize the ar* files
ignore = Name .unison/ar*

This specifies the roots of the synchronization on the local machine and the server. It specifies that path .unison is to be synchronized. Unison keeps its synchronization database in files named ar*.  I do not want to synchronize them, so those names are ignored. Again, refer to the Unison manual for details.

Now, on to the big granddaddy of them all, .unison/winXPData.prf. This Unison preference file synchronizes the Windows C:\Documents and Settings\voom\My Documents folder with Linux /home/voom:

# Roots of the synchronization
root = C:\Documents and Settings\voom\My Documents
root = ssh://voom@rogue//home/voom

# You may need some ssh options. Sorry, this is beyond the scope of this article.
# sshcmd=ssh.exe
# sshargs= someArgumentsToTheSshCommand

# Paths to synchronize
path = Desktop
path = Documents
path = Music
path = Pictures

How to Handle Problems

When you actually try running Unison, it will inevitably encounter things about your Linux files that are not supported on Windows file systems.  These errors are reported in C:\Users\userName\unison.log on Windows 7 or C:\Documents and Settings\userName\unison.log on Windows XP.

I will also show you Unix commands that quickly inspect your UNIX file tree for problems. This allows you to address the issues before a time-consuming Unison run.

There are three major problems that you will find: symbolic links, ‘:‘ in names, and case sensitivity.

Windows Does Not Handle Symbolic Links

Windows does not support symbolic links.  When Unison encounters one during synchronization, it gives the error:

src/gmock-1.5.0/lib/.libs/libgmock.so.0
Synlink not supported under Win32

Note that this error occurs during synchronization–you do not ordinarily see it before synchronizing.

Rather than waiting around for Unison to find them, the following UNIX shell command will instantly print all the symbolic links in a Linux directory tree rooted at dirName:

find dirName -type l

There are two solutions to the link problem:

  • Symbolic links are a sneaky way of making files appear to be where they are not.  If you’re being sneaky, you someday might outsmart yourself.  Consider just eliminating the symbolic link.
  • A legitimate use of symbolic links is to give a single library multiple names, like linking libgmock.so.0 to libgmock.so.0.0.0.  These are Linux libraries that Windows can’t load anyway, so I ignored the library links in my home directory using:

    # Ignore links to Linux libraries
    ignore = Name {*.so,*.so.[0-9],*.a,*.la}

Windows Does Not Allow ‘:‘ in File Names

Windows does not allow the characters \ / : * ? " < and > in file names.  Of these, the only one that might reasonably appear in a UNIX file name is ‘:‘ (colon).  When it encounters one, Unison gives the error:

The name of this Unix file is not allowed in Windows (mower/testdata/10M/CEL/blockHead:2)

You must either rename this file in Unix, or ignore it when synchronizing against Windows.

Readers in the IC design community will recognize the above example as a Synopsys Milkyway database file.  Milkyway-based tools do not run on Windows, so I just ignored all Milkyway files with this line in my Unison preference file:

# Ignore Milkyway data files
ignore = Regex .*:[0-9]+.*

But those of you who are not chip designers don’t care about Milkyway. You can probably just rename the UNIX file to eliminate the ‘:‘.

Rather than using Unison to find them, you can find all the files containing ‘:‘ in a Linux directory tree using the Unix command:

find dirName -name '*:*'

Windows File Names Are Case Insensitive

The UNIX file system is case-sensitive, so that ABC and abc are unique file names.  Windows is case insensitive, so it cannot distinguish such file or folder names.  When it encounters such a situation, Unison gives the error:

Two or more files on a case-sensitive system have names identical except for case.  They cannot be synchronized to a case-insensitive file system.  (Desktop/Library/python-2.6.5)

You need to stretch the Unison window quite a bit in order to see the troublesome file name at the end of this message.

Rather than relying on Unison to find them, you can find all file names that are identical except for case using the Unix commands:

find dirName | sort -uf > caseSensitive.txt
find dirName | sort -u | sort -f > caseInsensitive.txt
tkdiff caseSensitive.txt caseInsensitive.txt &

Of course you can use ordinary diff in place of tkdiff if you like.  tkdiff plainly shows that directories Python-2.6.5 and python-2.6.5 are identical except for the case of the “P”:

Comparison in tkdiff showing the directory that is not unique in a case-insensitive file system

Click to Enlarge

Use Include Files

Once you get serious about cross-platform synchronization, you will find it useful to put the common portion of your Unison preference files into separate files and reuse them with the include statement.  For example, here is the .unison/winXPData.prf file that I use on Windows XP to synchronize with my Linux server:

# Roots of the synchronization
root = C:\Documents and Settings\voom\My Documents
root = ssh://voom@rogue//home/voom
# You may need some ssh options. Sorry, this is beyond the scope of this article.
# sshcmd=ssh.exe
# sshargs= someArgumentsToTheSshCommand

# All the stuff that would ever be synchronized
include _dataCommon.prf

# For each platform involved, ignore the stuff that should not be synchronized
include _dataWindows.prf
include _dataUnix.prf

# Files to be ignored in every synchronization
include _alwaysIgnore.prf

# Explanation that will appear in the Profiles dialog
label = Synchronize Windows XP My Documents

My .unison/unixData.prf file synchronizes my Linux laptop with my Linux server. It is very similar to the above file, but it omits _dataWindows.prf:

# Roots of the synchronization
root = /home/voom
root = ssh://voom@rogue//home/voom

# All the stuff that would ever be synchronized
include _dataCommon.prf

# For each platform involved, ignore the stuff that should not be synchronized
include _dataUnix.prf

# Files to be ignored in every synchronization
include _alwaysIgnore.prf

# Explanation that will appear in the Profiles dialog
label = Synchronize Linux home directory

If I want to add a new path to the synchronization, I just add it to .unison/_dataCommon.prf, and all platforms will synchronize it.  If I discover some little turd that I never want to synchronize, I add it to .unison/_alwaysIgnore.prf.  For example, here is my current .unison/_alwaysIgnore.prf file:

# Text editor backup files
ignore = Name *~
ignore = Name .*~
ignore = Name *.swp
ignore = Name .#*

# Trash
ignore = Path .Trash
ignore = Path Trash

# My own naming convention for files that can be deleted any time
ignore = Path temp
ignore = Name temp.*
ignore = Name *.tmp

# Log and Journal files
ignore = Name *.log.??_??_??_??
ignore = Name *.cmd.??_??_??_??

Caution Regarding Include Files

The Unison include mechanism is far from multi-platform.  I found these problems:

  • Unison for UNIX cannot parse include statements in files that use Windows CRLF line terminators, resulting in errors like,

    File “unixData”, line 3: Garbled ‘include’ directive: include _dataUnix.prf

    or

    Preference file _dataUnix.prf not found

    Fortunately, the Unison include command for Windows does work with UNIX newline line terminators.  Therefore, to use the include command, all your .prf files must be UNIX style.  Convert them with the UNIX command,

    dos2unix ~/.unison/*.prf

  • You might like to create a .unison/include/ directory to hold all the included .prf files.  This is not possible because on Windows, the file separator must be ‘\‘, and on UNIX it must be ‘/‘,

    include include/dataCommon.prf # Works only on UNIX
    include include\dataCommon.prf # Works only on Windows

    yet both platforms must read this file.  Therefore, all the included .prf files must be in the .unison/ directory.

{ 11 comments… read them below or add one }

Mirko March 10, 2011 at 07:47

Thank you very much for your warning regarding file-ending conventions in .prf files

I spent several hours trying to understand why UNISON on my linux machine could not find include files that were in the ~/.unison directory. They were migrated from my windows machine :-) . dos2unix fixed the problem.

Mirko

John McGehee March 11, 2011 at 08:15

Yes, I took special care to mention those error messages to the article because I knew some would come searching for them.

Kevin C January 10, 2013 at 12:26

In this tutorial ( http://www.openlogic.com/wazi/bid/188061/Unison-Makes-Two-Way-File-Sync-Simple ) it says,

“You also need to ensure that if you are running Unison across platforms, you should invoke Unison and control it from Linux only, because Unison is known for not handling Unicode characters properly on non-Linux platforms.”

Have you had any such problems? Do you think it’s safe to ignore this warning?

suceuse de queues choune touffue May 13, 2014 at 12:51

Excellent post, comme d’habitude

Cochonne coquine May 17, 2014 at 11:51

Une fοіs de plus un post vraiment attractif

chiennasse pubis épilé May 25, 2014 at 05:23

Je terminerai de voir tout cela ce soir

google plus android June 5, 2014 at 05:38

My brother suggested I might like this website. He was entirely right.
This post actually made my day. You cann’t imagine simply how
much time I had spent for this info! Thanks!

Lunettes Versace June 11, 2014 at 05:56

My partner and I stumbled over here coming from a different website and thought I may as well check things out. I like what I see so now i’m following you. Look forward to finding out about your web page yet again.
Lunettes Versace http://www.chaussurefrancaise.com/lunettes-versace-fr-356.html

sexy June 12, 2014 at 15:02

Puis-je ρiqauer 2 3 phrases sur mon bloց ?

shallowalibi5195.blog.fc2.com June 30, 2014 at 12:15

C’est constamment de la joie de venir sur ce blog

Dresses July 16, 2014 at 21:18

I am curious to find out what blog platform you are using? I’m experiencing some small security issues with my latest site and I would
like to find something more secure. Do you have any recommendations?

Leave a Comment

{ 1 trackback }

Previous post:

Next post: