Monthly Archives: August 2008

Backing Up Open Files on Windows with Rsync (and BackupPC)

By | | Technology
100 Comments

Update:

Versions of the files below may be downloaded here.  This post is probably still useful as documentation.

 

 

This isn’t specific to BackupPC by any means, but I’ll preface this with a brief explanation:  BackupPC is a “set it and forget it” backup system driven from the server, that allows you to back up the entire network of *nix and Windows PCs.  It doesn’t require any software on the systems it backs up at all, since it relies upon rsync and smbclient, and optionally ssh.

For *nix, this works beautifully.  For Windows, this also works beautifully, except that “open files” can’t be backed up at all.  This problem isn’t unique to BackupPC, any attempt to back up or copy these files will fail, so most commercial backup systems have special “open file” clients to cope with it.

The official Windows solution for XP and later is something called a “volume shadow copy.”  It’s probably far more complex than it possibly needs to be, but essentially, it creates a pseudo-volume for any actual volume, with the difference being that you can actually back up files on it.  So, this can be handily used for rsync in order to make full backups, including every single file…  in theory, anyway.

My goals in getting this working:

  1. The solution should work with off-the-shelf components (i.e., no binaries or code)
  2. Installation and footprint should be minimal
  3. It should “just work” — if it’s too delicate, it’s not all that useful as a backup solution

It took quite a bit of trial-and-error, so I’ll skip what didn’t work, and get straight to what actually does work.  There are a few required components:

  1. winexe, a *nix program for remotely executing commands on Windows systems
  2. vshadow, a Windows program that creates and manages shadow copies
  3. dosdev, a Windows program that maps drive letters to volumes
  4. cwrsync, a Windows version of rsync (the “server” isn’t necessary)

Once all the pieces are assembled, I created a C:\BackupPC directory on the Windows box with all the necessary files.  Note that rsync does not need to be installed as a service, it actually gets loaded on-the-fly.  (Note that this directory is hard-coded in a lot of the files.) Here’s a listing of that directory:

Directory of C:\BackupPC
08/08/2008  07:11 PM                65 backuppc.cmd
08/10/2008  12:56 PM             1,928 cwrsync.cmd
07/22/2008  04:30 PM         1,082,368 cygcrypto-0.9.8.dll
04/11/2008  07:03 AM           999,424 cygiconv-2.dll
04/11/2008  07:03 AM            31,744 cygintl-3.dll
04/11/2008  07:03 AM            20,480 cygminires.dll
07/22/2008  04:30 PM         1,872,884 cygwin1.dll
04/11/2008  07:03 AM            66,048 cygz.dll
09/28/2004  02:07 PM             6,656 dosdev.exe
08/11/2008  11:08 PM             1,000 pre-cmd.vbs
08/11/2008  11:05 PM                44 pre-exec.cmd
07/22/2008  02:26 PM           348,160 rsync.exe
08/11/2008  10:12 PM               161 rsyncd.conf
08/11/2008  10:12 PM                22 rsyncd.secrets
08/11/2008  11:26 PM             1,177 sleep.vbs
06/08/2005  03:17 PM           294,912 vshadow.exe
08/11/2008  10:09 PM               581 vsrsync.cmd
08/11/2008  11:33 PM               308 vss-setvar.cmd

So, here’s how it works.  Before each backup, BackupPC has an option to call a local script first, waiting for that script to finish.  Here’s the execution chain:

  1. preusercmd.sh launches “pre-exec.cmd” on the Windows box
  2. preexec.cmd launches “pre-cmd.vbs”
  3. pre-cmd.vbs cleans up some files, launches “sleep.vbs” in the background (more on this later) and then launches “backuppc.cmd” in the background, and waits for the pid file to appear that signals that rsyncd has been launched
  4. backuppc.cmd launches vshadow, and tells it to execute vsrsync.cmd
  5. vsrsync.cmd maps the shadow volume to B:, and launches rsyncd — it sits and waits here, leaving vshadow and rsync open while the backup or rsync process runs — on the shadow copy on B:

Once the backup is completed, another local script is run — here’s its execution chain:

  1. postusercmd.sh puts a file called “wake.up” in the C:\BackupPC directory
  2. sleep.vbs wakes up, sees this file, reads rsyncd.pid, and kills the rsyncd process
  3. vsrsync.cmd now continues, since the rsync process is dead.  It unmaps the B: drive.  Once this script completes, vshadow automatically deletes the shadow volume.

Sure, it seems simple, but a lot of work went into that, since there are a lot of nuances to sort out.  Here are the file listings:

preusercmd.sh

#!/bin/bash
WINEXE=/usr/bin/winexe
UNAME="Administrator"
PWD="admin.password"
WRKGRP="WORKGROUP"
BOX=$1
$WINEXE --interactive=0 -U $UNAME -W $WRKGRP --password=$PWD //$BOX 'cmd /c c:\backuppc\pre-exec.cmd'
sleep 5
echo "Rsync and shadow copy loaded"
kill $$
# The script needs to be killed, otherwise, winexe waits for input

pre-exec.cmd

cd \backuppc
@echo off
cscript pre-cmd.vbs

pre-cmd.vbs

Const Flag = "C:\BackupPC\rsyncd.pid"
'
' Pid file shouldn't be there already
'
If DoesFileExist(Flag)=0 Then
   Set fso = CreateObject("Scripting.FileSystemObject")
   Set aFile = fso.GetFile(Flag)
   aFile.Delete
End If
'
' Nor should "wake.up"
'
If DoesFileExist("C:\BackupPC\wake.up")=0 Then
   Set fso = CreateObject("Scripting.FileSystemObject")
   Set aFile = fso.GetFile("C:\BackupPC\wake.up")
   aFile.Delete
End If
'
Set objShell = CreateObject("WScript.Shell")
objShell.Exec "cscript C:\BackupPC\sleep.vbs"
'
Set objShell = CreateObject("WScript.Shell")
objShell.Exec "C:\BackupPC\backuppc.cmd > C:\BackupPC\file.out"
'
' Just sleep until the file "rsyncd.pid" appears
'
While DoesFileExist(Flag)
   wscript.sleep 10000
Wend
'
' functions
'
function DoesFileExist(FilePath)
Dim fso
	Set fso = CreateObject("Scripting.FileSystemObject")
	if not fso.FileExists(FilePath) then
		DoesFileExist = -1
	else
		DoesFileExist = 0
	end if
	Set fso = Nothing
end function

sleep.vbs

Const Rsync = "C:\BackupPC\rsyncd.pid"
Const Flag = "C:\BackupPC\wake.up"
'
' Just sleep until the file "rsyncd.pid" appears
'
While DoesFileExist(Rsync)
   wscript.sleep 10000
Wend
'
' Now sleep until the file "wake.up" appears
'
While DoesFileExist(Flag)
   wscript.sleep 10000
Wend
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set aFile = fso.GetFile(Flag)
aFile.Delete
'
' It's time to kill Rsync
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set aReadFile = fso.OpenTextFile(Rsync, 1)
strContents = aReadFile.ReadLine
aReadFile.Close
'
Set objShell = CreateObject("WScript.Shell")
objShell.Run "taskkill /f /pid " & strContents, 0, true
'
' Wait for Rsync to let go
'
wscript.sleep 5000
'
' Delete PID file
'
If DoesFileExist(Rsync)=0 Then
   Set objShell = CreateObject("WScript.Shell")
   objShell.Run "cmd /c del C:\BackupPC\rsyncd.pid", 0, true
End If
'
' functions
'
function DoesFileExist(FilePath)
Dim fso
	Set fso = CreateObject("Scripting.FileSystemObject")
	if not fso.FileExists(FilePath) then
		DoesFileExist = -1
	else
		DoesFileExist = 0
	end if
	Set fso = Nothing
end function

backuppc.cmd

cd \backuppc
vshadow -script=vss-setvar.cmd -exec=vsrsync.cmd c:

vsrsync.cmd

REM @ECHO OFF
call vss-setvar.cmd
cd \BackupPC
SET CWRSYNCHOME=\BACKUPPC
SET CYGWIN=nontsec
SET CWOLDPATH=%PATH%
SET PATH=\BACKUPPC;%PATH%
dosdev B: %SHADOW_DEVICE_1%
REM Go into daemon mode, we'll kill it once we're done
rsync -v -v --daemon --config=rsyncd.conf --no-detach --log-file=diagnostic.txt
dosdev -r -d B:

rsyncd.conf

use chroot = false
strict modes = false
pid file = rsyncd.pid
[C]
path = /cygdrive/B/
auth users = Administrator
secrets file = rsyncd.secrets

postusercmd.sh

#!/bin/bash
WINEXE=/usr/bin/winexe
UNAME="Administrator"
PWD="admin.password"
WRKGRP="WORKGROUP"
BOX=$1
PID=$($WINEXE -U $UNAME -W $WRKGRP --password=$PWD //$BOX 'cmd /c echo '1' > c:\backuppc\wake.up')
echo "Rsync and shadow copy unloaded"
Share