This isn’t specific to BackupPC by any means, but I’ll preface this with a brief explanation: BackupPC is a “set it and forget it” backup system driven from the server, that allows you to back up the entire network of *nix and Windows PCs. It doesn’t require any software on the systems it backs up at all, since it relies upon rsync and smbclient, and optionally ssh.
For *nix, this works beautifully. For Windows, this also works beautifully, except that “open files” can’t be backed up at all. This problem isn’t unique to BackupPC, any attempt to back up or copy these files will fail, so most commercial backup systems have special “open file” clients to cope with it.
The official Windows solution for XP and later is something called a “volume shadow copy.” It’s probably far more complex than it possibly needs to be, but essentially, it creates a pseudo-volume for any actual volume, with the difference being that you can actually back up files on it. So, this can be handily used for rsync in order to make full backups, including every single file… in theory, anyway.
My goals in getting this working:
- The solution should work with off-the-shelf components (i.e., no binaries or code)
- Installation and footprint should be minimal
- It should “just work” — if it’s too delicate, it’s not all that useful as a backup solution
It took quite a bit of trial-and-error, so I’ll skip what didn’t work, and get straight to what actually does work. There are a few required components:
- winexe, a *nix program for remotely executing commands on Windows systems
- vshadow, a Windows program that creates and manages shadow copies
- dosdev, a Windows program that maps drive letters to volumes
- cwrsync, a Windows version of rsync (the “server” isn’t necessary)
Once all the pieces are assembled, I created a C:\BackupPC directory on the Windows box with all the necessary files. Note that rsync does not need to be installed as a service, it actually gets loaded on-the-fly. (Note that this directory is hard-coded in a lot of the files.) Here’s a listing of that directory:
Directory of C:\BackupPC 08/08/2008 07:11 PM 65 backuppc.cmd 08/10/2008 12:56 PM 1,928 cwrsync.cmd 07/22/2008 04:30 PM 1,082,368 cygcrypto-0.9.8.dll 04/11/2008 07:03 AM 999,424 cygiconv-2.dll 04/11/2008 07:03 AM 31,744 cygintl-3.dll 04/11/2008 07:03 AM 20,480 cygminires.dll 07/22/2008 04:30 PM 1,872,884 cygwin1.dll 04/11/2008 07:03 AM 66,048 cygz.dll 09/28/2004 02:07 PM 6,656 dosdev.exe 08/11/2008 11:08 PM 1,000 pre-cmd.vbs 08/11/2008 11:05 PM 44 pre-exec.cmd 07/22/2008 02:26 PM 348,160 rsync.exe 08/11/2008 10:12 PM 161 rsyncd.conf 08/11/2008 10:12 PM 22 rsyncd.secrets 08/11/2008 11:26 PM 1,177 sleep.vbs 06/08/2005 03:17 PM 294,912 vshadow.exe 08/11/2008 10:09 PM 581 vsrsync.cmd 08/11/2008 11:33 PM 308 vss-setvar.cmd
So, here’s how it works. Before each backup, BackupPC has an option to call a local script first, waiting for that script to finish. Here’s the execution chain:
- preusercmd.sh launches “pre-exec.cmd” on the Windows box
- preexec.cmd launches “pre-cmd.vbs”
- pre-cmd.vbs cleans up some files, launches “sleep.vbs” in the background (more on this later) and then launches “backuppc.cmd” in the background, and waits for the pid file to appear that signals that rsyncd has been launched
- backuppc.cmd launches vshadow, and tells it to execute vsrsync.cmd
- vsrsync.cmd maps the shadow volume to B:, and launches rsyncd — it sits and waits here, leaving vshadow and rsync open while the backup or rsync process runs — on the shadow copy on B:
Once the backup is completed, another local script is run — here’s its execution chain:
- postusercmd.sh puts a file called “wake.up” in the C:\BackupPC directory
- sleep.vbs wakes up, sees this file, reads rsyncd.pid, and kills the rsyncd process
- vsrsync.cmd now continues, since the rsync process is dead. It unmaps the B: drive. Once this script completes, vshadow automatically deletes the shadow volume.
Sure, it seems simple, but a lot of work went into that, since there are a lot of nuances to sort out. Here are the file listings:
preusercmd.sh
#!/bin/bash WINEXE=/usr/bin/winexe UNAME="Administrator" PWD="admin.password" WRKGRP="WORKGROUP" BOX=$1 $WINEXE --interactive=0 -U $UNAME -W $WRKGRP --password=$PWD //$BOX 'cmd /c c:\backuppc\pre-exec.cmd' sleep 5 echo "Rsync and shadow copy loaded" kill $$ # The script needs to be killed, otherwise, winexe waits for input
pre-exec.cmd
cd \backuppc @echo off cscript pre-cmd.vbs
pre-cmd.vbs
Const Flag = "C:\BackupPC\rsyncd.pid"
'
' Pid file shouldn't be there already
'
If DoesFileExist(Flag)=0 Then
Set fso = CreateObject("Scripting.FileSystemObject")
Set aFile = fso.GetFile(Flag)
aFile.Delete
End If
'
' Nor should "wake.up"
'
If DoesFileExist("C:\BackupPC\wake.up")=0 Then
Set fso = CreateObject("Scripting.FileSystemObject")
Set aFile = fso.GetFile("C:\BackupPC\wake.up")
aFile.Delete
End If
'
Set objShell = CreateObject("WScript.Shell")
objShell.Exec "cscript C:\BackupPC\sleep.vbs"
'
Set objShell = CreateObject("WScript.Shell")
objShell.Exec "C:\BackupPC\backuppc.cmd > C:\BackupPC\file.out"
'
' Just sleep until the file "rsyncd.pid" appears
'
While DoesFileExist(Flag)
wscript.sleep 10000
Wend
'
' functions
'
function DoesFileExist(FilePath)
Dim fso
Set fso = CreateObject("Scripting.FileSystemObject")
if not fso.FileExists(FilePath) then
DoesFileExist = -1
else
DoesFileExist = 0
end if
Set fso = Nothing
end function
sleep.vbs
Const Rsync = "C:\BackupPC\rsyncd.pid"
Const Flag = "C:\BackupPC\wake.up"
'
' Just sleep until the file "rsyncd.pid" appears
'
While DoesFileExist(Rsync)
wscript.sleep 10000
Wend
'
' Now sleep until the file "wake.up" appears
'
While DoesFileExist(Flag)
wscript.sleep 10000
Wend
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set aFile = fso.GetFile(Flag)
aFile.Delete
'
' It's time to kill Rsync
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set aReadFile = fso.OpenTextFile(Rsync, 1)
strContents = aReadFile.ReadLine
aReadFile.Close
'
Set objShell = CreateObject("WScript.Shell")
objShell.Run "taskkill /f /pid " & strContents, 0, true
'
' Wait for Rsync to let go
'
wscript.sleep 5000
'
' Delete PID file
'
If DoesFileExist(Rsync)=0 Then
Set objShell = CreateObject("WScript.Shell")
objShell.Run "cmd /c del C:\BackupPC\rsyncd.pid", 0, true
End If
'
' functions
'
function DoesFileExist(FilePath)
Dim fso
Set fso = CreateObject("Scripting.FileSystemObject")
if not fso.FileExists(FilePath) then
DoesFileExist = -1
else
DoesFileExist = 0
end if
Set fso = Nothing
end function
backuppc.cmd
cd \backuppc vshadow -script=vss-setvar.cmd -exec=vsrsync.cmd c:
vsrsync.cmd
REM @ECHO OFF call vss-setvar.cmd cd \BackupPC SET CWRSYNCHOME=\BACKUPPC SET CYGWIN=nontsec SET CWOLDPATH=%PATH% SET PATH=\BACKUPPC;%PATH% dosdev B: %SHADOW_DEVICE_1% REM Go into daemon mode, we'll kill it once we're done rsync -v -v --daemon --config=rsyncd.conf --no-detach --log-file=diagnostic.txt dosdev -r -d B:
rsyncd.conf
use chroot = false strict modes = false pid file = rsyncd.pid [C] path = /cygdrive/B/ auth users = Administrator secrets file = rsyncd.secrets
postusercmd.sh
#!/bin/bash WINEXE=/usr/bin/winexe UNAME="Administrator" PWD="admin.password" WRKGRP="WORKGROUP" BOX=$1 PID=$($WINEXE -U $UNAME -W $WRKGRP --password=$PWD //$BOX 'cmd /c echo '1' > c:\backuppc\wake.up') echo "Rsync and shadow copy unloaded"
Related posts:
Is anyone else noticing very high CPU utilization for winexe during the backups ? I don’t see why it would be high at all, but when backing up 2 machines at the same time, Nagios load monitors are tripped on my Linux box.
Anything I can do to curb that?
I suppose you could always use nice or renice for the winexe process, but in my experience, it does consume a large amount of CPU while making the initial connection. I’m not sure why.
I now have one Windows 2008 server that needs to be backed up. I found out it uses something called DiskShadow not vshadow. I had to modify my files like the following.
-backuppc.cmd-
c:
cd \backuppc
diskshadow /s DiskShadowScript.txt
del c:\backuppc\*.cab /q
-DiskShadowScript.txt –
#DiskShadow script file
set context persistent
set metadata c:\backuppc\example.cab
set verbose on
begin backup
add volume c: alias SystemVolumeShadow
create
expose %SystemVolumeShadow% B:
exec c:\backuppc\Serverbackup.cmd
unexpose B:
end backup
delete shadows set %vss_shadow_set%
#End of script
-Serverbackup.cmd –
c:
cd \backuppc
REM Go into daemon mode, we’ll kill it once we’re done
rsync -v -v –daemon –config=rsyncd.conf –no-detach –log-file=diagnostic.txt
I was noticing alot of failed backups at different times. Not all of the systems were failing but several each night. (Child exited prematurely) I put “port = ” in the rsync.conf file and made sure the config file in Backuppc matched. I used a different port number for each server and I don’t seem to have as many failures now.
I found a way to deal with the excessive CPU utilization of winexe. I still don’t know why it goes bonkers when BackupPC launches it, because I couldn’t reproduce it on the command line, even sudo’ing as backuppc .
AOS2 is an XP x64 machine.
The key was to send winexe to the background so that it could be killed later. Otherwise the bash script never gets past the winexe line.
It’s fugly, but it works.
preusercmd.sh
#!/bin/bash
WINEXE=/usr/bin/winexe
UNAME=”backuppc”
PWD=”"
WRKGRP=”WORKINIT”
BOX=$1
if [ $BOX == "aos2" ] ; then
$WINEXE –interactive=0 –ostype=2 -U $UNAME -W $WRKGRP –password=$PWD //$BOX ‘cmd /c c:\backuppc\pre-exec.cmd’ &
else
$WINEXE –interactive=0 -U $UNAME -W $WRKGRP –password=$PWD //$BOX ‘cmd /c c:\backuppc\pre-exec.cmd’ &
fi
WINEXEPID=”$pids $!”
sleep 5
echo “Rsync and shadow copy loaded\n”
kill $WINEXEPID
kill $$
# The script needs to be killed, otherwise, winexe waits for input
I don’t understand the point of a backup that does not restore easily. The way this is configured the connection is to the shadowed drive or nothing. Why?
Shouldn’t the actual backup use the shadowed drive and the restore connect to the regular (not shadowed) drive/path so that files can be restored – easily?
Sure, locked files won’t restore, but those are the exception, not the norm. For me, the usual restore is because somebody deleted a file accidentally , or overwrote a file accidentally, or otherwise corrupted a needed file.
I strongly suggest creating restore scripts that link to an unshadowed, non read-only drive(path) so that the built-in restore feature works for the majority of scenarios.
P.S. You cant restore a zipped file this way either because the whole path/drive (when shadowed) is read-only!
There’s nothing in the backup that implies how it must be restored; the techniques above are generally used for backing up, not recovery — and recovery is usually via smb, rsync, or zip files to an unshadowed drive. Aside from the registry, most files can be unlocked by shutting down the application that controls them (e.g., Outlook) and recovering normally. It’s possible to recover the entire OS, registry and all, with the help of the recovery console — but as you note, most recoveries are for a file here and there, not the entire drive.
[...] Backing Up Open Files on Windows with Rsync (and BackupPC) [...]
[...] Backing Up Open Files on Windows with Rsync (and BackupPC) [...]
[...] been backed up via BackupPC (assuming, of course, that some method, such as that documented here, has been used to back up open files, such as the [...]
Hi, i just implemented this and its working fine… but i had some pitfalls and blank spaces that made this a littler harder, so this update might be useful for others:
1-the backuppc host config file… this are mostly the needed things. Of course, tune the log level after testing
# cat /etc/backuppc/test.pl
$Conf{XferMethod} = ‘rsyncd’;
$Conf{ClientCharset} = ‘cp1252′;
$Conf{RsyncShareName} = [
'users'
];
$Conf{DumpPostUserCmd} = ‘/etc/backuppc/bin/postusercmd.sh $host’;
$Conf{DumpPreUserCmd} = ‘/etc/backuppc/bin/preusercmd.sh $host’;
$Conf{XferLogLevel} = ’3′;
$Conf{RsyncdPasswd} = ‘rsync-pass’;
$Conf{RsyncdUserName} = ‘backuppc’;
2- after some search, i found a small zip with all this files, that helps alot… it is in here:
http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/backuppc-21/backuppc-file-rsyncp-issues-100060/index-30.html
sadly the included vshadow is old, XP only and doesnt work in vista and windows7, so it needs the latest version (might not work in XP, but use the older one for that).
But to this this latest version i had to download 1.3GB of a MS dev ISO to get that damn file (about 221KB)… luckily cabextract manage to get the file without having to install that huge SDK just to get the vshadow.exe
i updated the zip file, add a some new files (see below) and uploaded to here:
http://caravela.homelinux.net/~higuita/backuppc/backuppc-rsyncd+vss-windows.tar.gz
ps: dont forget to change the usernames, password and domains
4-a example rsyncd.secret, useful for people that dont know rsync, its just a “username:password” plain file, one user in each line
5-a rsyncd.conf example… please note that the backuppc do the backup to the rsyncd repository name, it doesnt accept any other path (ie: rsyncd defines the [home], backuppc makes the backup to “home”, “/home/user1/Documents” isnt valid, create a repository directly to that path if you need it)
use chroot = false
strict modes = false
pid file = rsyncd.pid
UID = 0
GID = 0
[home]
path = /cygdrive/B/Users/
auth users = backuppc
secrets file = rsyncd.secrets
# if you want to exclude files on the windows site, use this… useful for global temporary dirs, windows junction points, etc
# no example here, because i dont use it yet, testing the exclude on the backuppc side
#exclude from = /cygdrive/c/admin/backuppc/exclude_user.txt
6-A small install script for the backuppc machine, to install the client in windows:
#
#set -x
host=$1
if [ ! -z $host ] ; then
cd /root/windows
nano BackupPC/rsyncd.conf
smbclient //$host/c$ -U backup-admin%passdword -W domain <<EOF
mkdir admin
cd admin
mkdir backuppc
cd backuppc
prompt
lcd BackupPC
mput *
dir
exit
EOF
else
echo install-pc.sh [hostname]
echo install on c:\\admin\\backuppc
fi
7-for the winexe to work, you need a admin user, the firewall disables (or at least open for the backuppc server) and domain policy/local policy allowing the remote access for that user… and for my install script, the c$ share enabled
8-if you use my tar.gz package, please check the scripts and scan the binaries for virus… you shouldn't trust me when those scripts will have access to windows admin accounts
good luck
higuita
Hi higuita, i folowed you install note from the archive file http://caravela.homelinux.net/~higuita/backuppc/backuppc-rsyncd+vss-windows.tar.gz, i have changed the usernames password and domain names, i made sure that no firewall is blocking the backuppc server, i used domain admin user in preusercmd.sh and postusercmd.sh to make sure it can acces all pc-s in the domain, used cp1252 charset, the commands in backuppc webgui for post and pre look like /etc/backuppc/bin/post(an pre)usercmd.sh $host but i got the folowing message form backuppc 2010-05-27 09:46:33 Backup aborted (inet connect: Connection refused). I believe this message is returned when the connection between backuppc and the clinet pc is not made via rsync. What have i done wrong?
i’ll paste the setting files:
PREUSERCMD.SH
#!/bin/bash
#set -x
WINEXE=/etc/backuppc/bin/winexe
UNAME=”baltagul”
PWD=”miorita”
WRKGRP=”DDOO”
BOX=$1
$WINEXE –interactive=0 -U $UNAME -W $WRKGRP –password=$PWD //$BOX ‘cmd /c c:\admin\backuppc\pre-exec.cmd’
#if winexe uses too much CPU, try to background it and kill it later
#$WINEXE –interactive=0 -U $UNAME -W $WRKGRP –password=$PWD //$BOX ‘cmd /c c:\admin\backuppc\pre-exec.cmd’ &
#WINEXEPID=”$pids $!”
#sleep
#kill $WINEXEPID
echo “Rsync and shadow copy loaded”
kill $$
# The script needs to be killed, otherwise, winexe waits for input
POSTUSERCMD.SH
#!/bin/bash
#set -x
WINEXE=/etc/backuppc/bin/winexe
UNAME=”baltagul”
PWD=”miorita”
WRKGRP=”DDOO”
BOX=$1
$WINEXE –interactive=0 -U $UNAME -W $WRKGRP –password=$PWD //$BOX ‘cmd /c c:\admin\backuppc\pre-exec.cmd’
#if winexe uses too much CPU, try to background it and kill it later
#$WINEXE –interactive=0 -U $UNAME -W $WRKGRP –password=$PWD //$BOX ‘cmd /c c:\admin\backuppc\pre-exec.cmd’ &
#WINEXEPID=”$pids $!”
#sleep
#kill $WINEXEPID
echo “Rsync and shadow copy loaded”
kill $$
# The script needs to be killed, otherwise, winexe waits for input
rsyncd.secrets
backupp:miorita
You need to open your firewall — in short, “rsync” needs to be added to programs allowed to make connections through the firewall.
egrimisu:
that mean that rsync didnt start or that you cant connect to the machine… for the later, its the firewall, for the first its probably a problem with the VSS service on the windows machine.
try to run the scripts in the command line, specially the c:\admin\backuppc\pre-exec.cmd and check the windows event viewer.
sadly, VSS is hard to debug, for about 100 windows machine, i have right now 3 with VSS problems that i’m still trying to fix.
this url is useful for most problems:
http://www.gfi.com/blog/vss-troubleshooting/
if the VSS still doesn’t work, you can try to enable the debug
http://support.microsoft.com/default.aspx?scid=kb;EN-US;887013
and/or contact MS or reinstall.
good luck