Update:
Versions of the files below may be downloaded here. This post is probably still useful as documentation.
This isn’t specific to BackupPC by any means, but I’ll preface this with a brief explanation: BackupPC is a “set it and forget it” backup system driven from the server, that allows you to back up the entire network of *nix and Windows PCs. It doesn’t require any software on the systems it backs up at all, since it relies upon rsync and smbclient, and optionally ssh.
For *nix, this works beautifully. For Windows, this also works beautifully, except that “open files” can’t be backed up at all. This problem isn’t unique to BackupPC, any attempt to back up or copy these files will fail, so most commercial backup systems have special “open file” clients to cope with it.
The official Windows solution for XP and later is something called a “volume shadow copy.” It’s probably far more complex than it possibly needs to be, but essentially, it creates a pseudo-volume for any actual volume, with the difference being that you can actually back up files on it. So, this can be handily used for rsync in order to make full backups, including every single file… in theory, anyway.
My goals in getting this working:
- The solution should work with off-the-shelf components (i.e., no binaries or code)
- Installation and footprint should be minimal
- It should “just work” — if it’s too delicate, it’s not all that useful as a backup solution
It took quite a bit of trial-and-error, so I’ll skip what didn’t work, and get straight to what actually does work. There are a few required components:
- winexe, a *nix program for remotely executing commands on Windows systems
- vshadow, a Windows program that creates and manages shadow copies
- dosdev, a Windows program that maps drive letters to volumes
- cwrsync, a Windows version of rsync (the “server” isn’t necessary)
Once all the pieces are assembled, I created a C:\BackupPC directory on the Windows box with all the necessary files. Note that rsync does not need to be installed as a service, it actually gets loaded on-the-fly. (Note that this directory is hard-coded in a lot of the files.) Here’s a listing of that directory:
Directory of C:\BackupPC 08/08/2008 07:11 PM 65 backuppc.cmd 08/10/2008 12:56 PM 1,928 cwrsync.cmd 07/22/2008 04:30 PM 1,082,368 cygcrypto-0.9.8.dll 04/11/2008 07:03 AM 999,424 cygiconv-2.dll 04/11/2008 07:03 AM 31,744 cygintl-3.dll 04/11/2008 07:03 AM 20,480 cygminires.dll 07/22/2008 04:30 PM 1,872,884 cygwin1.dll 04/11/2008 07:03 AM 66,048 cygz.dll 09/28/2004 02:07 PM 6,656 dosdev.exe 08/11/2008 11:08 PM 1,000 pre-cmd.vbs 08/11/2008 11:05 PM 44 pre-exec.cmd 07/22/2008 02:26 PM 348,160 rsync.exe 08/11/2008 10:12 PM 161 rsyncd.conf 08/11/2008 10:12 PM 22 rsyncd.secrets 08/11/2008 11:26 PM 1,177 sleep.vbs 06/08/2005 03:17 PM 294,912 vshadow.exe 08/11/2008 10:09 PM 581 vsrsync.cmd 08/11/2008 11:33 PM 308 vss-setvar.cmd
So, here’s how it works. Before each backup, BackupPC has an option to call a local script first, waiting for that script to finish. Here’s the execution chain:
- preusercmd.sh launches “pre-exec.cmd” on the Windows box
- preexec.cmd launches “pre-cmd.vbs”
- pre-cmd.vbs cleans up some files, launches “sleep.vbs” in the background (more on this later) and then launches “backuppc.cmd” in the background, and waits for the pid file to appear that signals that rsyncd has been launched
- backuppc.cmd launches vshadow, and tells it to execute vsrsync.cmd
- vsrsync.cmd maps the shadow volume to B:, and launches rsyncd — it sits and waits here, leaving vshadow and rsync open while the backup or rsync process runs — on the shadow copy on B:
Once the backup is completed, another local script is run — here’s its execution chain:
- postusercmd.sh puts a file called “wake.up” in the C:\BackupPC directory
- sleep.vbs wakes up, sees this file, reads rsyncd.pid, and kills the rsyncd process
- vsrsync.cmd now continues, since the rsync process is dead. It unmaps the B: drive. Once this script completes, vshadow automatically deletes the shadow volume.
Sure, it seems simple, but a lot of work went into that, since there are a lot of nuances to sort out. Here are the file listings:
preusercmd.sh
#!/bin/bash WINEXE=/usr/bin/winexe UNAME="Administrator" PWD="admin.password" WRKGRP="WORKGROUP" BOX=$1 $WINEXE --interactive=0 -U $UNAME -W $WRKGRP --password=$PWD //$BOX 'cmd /c c:\backuppc\pre-exec.cmd' sleep 5 echo "Rsync and shadow copy loaded" kill $$ # The script needs to be killed, otherwise, winexe waits for input
pre-exec.cmd
cd \backuppc @echo off cscript pre-cmd.vbs
pre-cmd.vbs
Const Flag = "C:\BackupPC\rsyncd.pid"
'
' Pid file shouldn't be there already
'
If DoesFileExist(Flag)=0 Then
Set fso = CreateObject("Scripting.FileSystemObject")
Set aFile = fso.GetFile(Flag)
aFile.Delete
End If
'
' Nor should "wake.up"
'
If DoesFileExist("C:\BackupPC\wake.up")=0 Then
Set fso = CreateObject("Scripting.FileSystemObject")
Set aFile = fso.GetFile("C:\BackupPC\wake.up")
aFile.Delete
End If
'
Set objShell = CreateObject("WScript.Shell")
objShell.Exec "cscript C:\BackupPC\sleep.vbs"
'
Set objShell = CreateObject("WScript.Shell")
objShell.Exec "C:\BackupPC\backuppc.cmd > C:\BackupPC\file.out"
'
' Just sleep until the file "rsyncd.pid" appears
'
While DoesFileExist(Flag)
wscript.sleep 10000
Wend
'
' functions
'
function DoesFileExist(FilePath)
Dim fso
Set fso = CreateObject("Scripting.FileSystemObject")
if not fso.FileExists(FilePath) then
DoesFileExist = -1
else
DoesFileExist = 0
end if
Set fso = Nothing
end function
sleep.vbs
Const Rsync = "C:\BackupPC\rsyncd.pid"
Const Flag = "C:\BackupPC\wake.up"
'
' Just sleep until the file "rsyncd.pid" appears
'
While DoesFileExist(Rsync)
wscript.sleep 10000
Wend
'
' Now sleep until the file "wake.up" appears
'
While DoesFileExist(Flag)
wscript.sleep 10000
Wend
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set aFile = fso.GetFile(Flag)
aFile.Delete
'
' It's time to kill Rsync
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set aReadFile = fso.OpenTextFile(Rsync, 1)
strContents = aReadFile.ReadLine
aReadFile.Close
'
Set objShell = CreateObject("WScript.Shell")
objShell.Run "taskkill /f /pid " & strContents, 0, true
'
' Wait for Rsync to let go
'
wscript.sleep 5000
'
' Delete PID file
'
If DoesFileExist(Rsync)=0 Then
Set objShell = CreateObject("WScript.Shell")
objShell.Run "cmd /c del C:\BackupPC\rsyncd.pid", 0, true
End If
'
' functions
'
function DoesFileExist(FilePath)
Dim fso
Set fso = CreateObject("Scripting.FileSystemObject")
if not fso.FileExists(FilePath) then
DoesFileExist = -1
else
DoesFileExist = 0
end if
Set fso = Nothing
end function
backuppc.cmd
cd \backuppc vshadow -script=vss-setvar.cmd -exec=vsrsync.cmd c:
vsrsync.cmd
REM @ECHO OFF call vss-setvar.cmd cd \BackupPC SET CWRSYNCHOME=\BACKUPPC SET CYGWIN=nontsec SET CWOLDPATH=%PATH% SET PATH=\BACKUPPC;%PATH% dosdev B: %SHADOW_DEVICE_1% REM Go into daemon mode, we'll kill it once we're done rsync -v -v --daemon --config=rsyncd.conf --no-detach --log-file=diagnostic.txt dosdev -r -d B:
rsyncd.conf
use chroot = false strict modes = false pid file = rsyncd.pid [C] path = /cygdrive/B/ auth users = Administrator secrets file = rsyncd.secrets
postusercmd.sh
#!/bin/bash WINEXE=/usr/bin/winexe UNAME="Administrator" PWD="admin.password" WRKGRP="WORKGROUP" BOX=$1 PID=$($WINEXE -U $UNAME -W $WRKGRP --password=$PWD //$BOX 'cmd /c echo '1' > c:\backuppc\wake.up') echo "Rsync and shadow copy unloaded"
Very nice how-to! I have a couple of questions regarding your setup. Where do you put the commands into backuppc? I know its in the config.pl file but is it the DumpPreUserCmd? Also, I ran into a problem (not with this solution) with the backuppc version of rsync and the cwrync, the backuppc version is to write the path c:\path\to\file, while the cwrsync version is to write it /cygdrive/c/path/to/file once i figured that out i could get backuppc to actually work, but i have no idea why they differ so dramatically like that. I guess my question is that i see sometimes you use “\” and other times the “/cygwin/path” like with the vbs and cmd scripts, is it that one is using dos and the other linux? (cygwin). And the last question, is the starting script the only one i have to put into backuppc? Since each of the other scripts calls or waits on another script to run, does it need a DumpPostUserCmd command?
thanks for all your help and sorry for the questions, i just stumbled upon your site and really want to give it a shot, ive been searching for months for a good way to do this.
Nick
Is there anyway to see whats going on? Where does it log to? How do you know if a part gets stuck or has an error?
I put in the PreUser and Post User like so:
$Conf{DumpPreUserCmd} = ‘/usr/tools/preusercmd.sh $host’;
$Conf{DumpPostUserCmd} = ‘/usr/tools/postusercmd.sh $host’
There are two logs that the scripts create: diagnostic.txt and file.out.
diagnostic.txt is the log from rsync, and file.out is the log from the script that turns on volume shadow copy and launches rsync.
In general, if anything goes wrong, rsync won’t launch, and BackupPC will therefore complain that it can’t reach rsync, and those two logs will tell you why.
To answer your question about Cygwin, both rsync and Cygwin view directories *nix style, with forward slashes. I use the convention of using forward slashes with rsync/Cygwin/*nix and backward slashes with Windows shells.
Well, i think i messed something up somewhere. It did the backup and it completed fine, backuppc didnt complain at all, but it left the rsync.exe running and didnt remove the shadow copy when it was finished. Ill have to check those log files you specified later to see if it will shed any light on this.
Do you actually install cwrsync? Is it necessary to install it once you have the files you need (aka the files in the BackupPC directory) Im planning on doing this on several 2003 servers and it would be nice if all i had to do was copy a directory to a server, add a host in backuppc and be off and running.
thanks for your help.
The volume shadow copy sits and waits until: a file called “wake.up” is placed in the C:\BackupPC directory, rsync is killed, or the system is rebooted. The *nix script “postusercmd.sh” drops a wake.up file in the directory (which causes everything to be cleaned up.)
There’s no need to install cwrsync; copying the directory from machine to machine is sufficient for everything to work (keeping in mind that there are different versions of vshadow for 2003 and XP.)
My eyes must have been playing tricks on me last night, i totally didnt see the postusercmd script last night, i also added $host after the commands to see if that has any effect. I do currently have cwrsync installed and it loads as a service, which id like to get away from. I tried stopping the service and running a backup to see if the backup would start, and it failed with a connection refused error. Once i started the service again the backup started as expected. Later i will try uninstalling it and just running it from the directory alone but do you think that will have any effect? It should of started rsync per the script right? I wonder if there is a conflict because it is also installed as a service that was stopped? Does the install of cwrsync make any kind of registry modifications? Sorry to bother you will all my questions, I really do appreciate all the help you have given. You did an amazing job putting this all together.
thanks again
The cwrsync service or a cwrsync server installation definitely conflicts with the methods outlined here; they attempt to use the same port, and rsync simply won’t load. As far as I know, cwrsync only makes registry modifications with regard to services that it loads.
As a troubleshooting step, you’ll want to start from the inside out, and check the diagnostic logs as you go, first starting with backuoppc.cmd from the command line, using taskmgr to verify that rsync is loaded and vshadow to confirm that you have a volume shadow copy, and work up to launching it from *nix.
ok uninstalled cwrsync, tried to run it just from the dir and get “connection refused” in backuppc.
get this in file.out:
ERROR: COM call “m_pVssObject->StartSnapshotSet(&m_latestSnapshotSetID)” failed.
– Returned HRESULT = 0x80042316
– Error text: VSS_E_SNAPSHOT_SET_IN_PROGRESS
– Please re-run VSHADOW.EXE with the /tracing option to get more details
just doesnt seem to be loading correcty.
before when i was getting the connection refused it was because cwrsync wasnt loaded as a service, and when i tried to reinstall the package it wont start the service anymore. says that it started and then stopped because there was nothing for it to do. very strange.
can you explain what you do when you set this up on a new server? you just copy the dir over to the server and it works? im trying to figure out where the problem is coming from.
thanks
Yes, I just copy it over, and it works.
From the description of the error, you already have a shadow copy open; most likely because it was unable to launch rsync due to the conflict.
As a general rule of thumb, you’ll want to reboot between attempts, or at least go to taskmgr and clean up any debris (command scripts, rsync, vbs, vshadow processes.) It’s also a handy diagnostic step to use vshadow from the command line to see if it’s complaining about anything (for example, it may complain that certain volume shadow providers aren’t properly registered, etc.)
im not getting a diagnostic.txt and its not launching the vshadow command, in the log i get:
rsync: chdir /cygdrive/S failed
: No such file or directory (2)
S is the drive i want to the vshadow commnad to attach the shadow to.
When i try to start a backup that is the error that i get. i have the pre/post command correctly in the backuppc config. what could be going wrong?
Rsync produces that error if cygwin doesn’t see an S: drive. That implies that you’re running a different rsync than the one that should be launched by vshadow. (Alternatively, vshadow isn’t creating a volume shadow copy, and by extension, you cannot map an S: drive to it.)
Rsync has to be launched *after* the drive is mapped, or it won’t work.
But im using your scripts, so im pretty sure the order has been worked out 😉 If i have to manually mount the shadow copy it would defeat the purpose of your wonderful scripts!
Ive made sure that i reboot every time i make a change just to make sure it doesnt mess up anything, so the server has been rebooted serveral times, all with the same result.
I added the /tracing command to the backuppc.cmd file to see if it will give me anymore details.
from what im gathering from your posts, vshadow thinks there is already a shadow copy open (doesnt a reboot clear this?) there is no drive assigned to a shadow copy, only the C: drive which isnt the copy.
and since vshadow cant do its thing the rest of the script is failing because it cant shadow to the S: drive.
I got an error with /tracing saying “ERROR: selected writeer ‘system writer’ is in a failed state!”
and then it exits. any ideas on this one?
Windows 2003 allows permanent volume shadow copies, so unless you’re using XP, a reboot may not clear them.
That being said, from the error, you’ve got a problem blocking the volume shadow copy service from working at all. You’ll want to track that down first. If a writer has an error, you’ll want to look up that specific writer to see what the problem is. VSSADMIN is a helpful tool.
That being said, Win2k3 has had a number of updates to VSS, so you’ll want to make sure you’re up to date:
http://support.microsoft.com/kb/833167
Ok, ive deleted all the shadow copies in 2k3 via vssadmin, tried to do a backup and go this error:
rsync: failed to create pid file /cygdrive/c/rsyncd/rsyncd.pid: File exists (17)
2008/09/25 20:59:11 [5756] rsync error: error in file IO (code 11) at clientserver.c(985) [receiver=3.0.4]
then i deleted the pid file and go the same IO error 11. then i registered the service again just to see what would happen and i go back to the same error about not being able to access drive S:
Im lost, why isnt it trying to shadow copy C to drive S? Im sure if i manually shadow copy C to S that it will work, but that kinda defeats the purpose of this whole thing.
The script will delete an rsyncd.pid left over from a dead rsync process… so the most logical conclusion to draw from the error you’re seeing is that rsyncd is already running. (Alternatively, the user launching the script doesn’t have permission to create/delete the file.) You seem to [still] have a conflict with a running copy of rsyncd. It shouldn’t be in the registry or running at all when the script is launched.
Meanwhile, I should explain that a volume shadow copy creates a device, and accessing that device directs the operating system to look back at earlier copies of the data. Mapping a drive to it is handy shorthand, so that the path stays short and for convenience. I chose drive B: because it’s generally reserved for floppies, and therefore nearly always available.
Well, unfortunately im still having a problem, im still getting connection refused which means it isnt launching the rsync process to do the backup. I started from scratch just to see if that had any effect and it didnt, at first i changed the drive letter and directory i wanted the stuff to be in, but i wanted to have a vanilla copy and see if what you had would work (maybe i screwed up the files somewhere) but even with no modifications to the scripts it still wont work. Its not creating log files or anything, i just get connection refused. The backuppc log doesnt give me anything but refused as well. I can get a little further if i install cwrsync as a service, but then it bombs out on cant find drive B, so it gets further but doesnt create the shadow copy. Am i suppose to see a drive B under my computer when it works? at least for the duration of the backup? Also, what is in the cwrsync.cmd script? I see the file listed in your list of files but the script isnt on your blog. Is that file important at all? Ive tested this over and over again on 2 separate machines, one 2k3 and one XP using the respective vshadow.exe files. Can you shed any more light on this for me? I appreciate all your help.
UPDATE: Got further in my troubleshooting.
The script wasnt actually running, i was getting a error:
/bin/bash^M: bad interpreter: no such file or directory
Which is caused by creating your scripts in notepad.exe in windows 😉 oops.
dos2unix on ubuntu (part of the tofrodos package) fixed that particular problem.
So now that i can actually run the scripts I get this error:
ERROR: Failed to open connection – NT_STATUS_IO_TIMEOUT
Which from what ive read is a samba type error with authentication.
Do you have to configure samba differently to use your solution? Did you have to modify samba in any way? I do have it installed. The machine are on a domain not a workgroup, Ive tried putting the domain name as the workgroup and that didnt work and ive also tried removing the workgroup flag altogether and that didnt work. The linux box isnt part of my domain, and it has some shares on it so i can access it from other domain computers. There has to be more to this setup that your original post. What did you to do your linux box in preparation for backuppc?
Im a little further, but hit another stumbling block. Any ideas?
Samba isn’t actually used at all. I recommend troubleshooting from the furthest-nested function outward. In other words, start in a command window in the c:\backuppc directory in Windows. In order, check:
rsync (once that works, kill it and try the next step, and so on)
vsrsync.cmd
backuppc.cmd
pre-cmd.vbs
If all that works, try launching it directly from winexe from *nix (again, from the command line)
For each step, if it doesn’t work, you should get useful information as to why, that could be hidden by the surrounding processes.
Also, if you want to compare your directory to mine, you can download it here:
http://www.houseofqueued.com/backuppc-client-scripts-XP.zip
Have you tested this over the internet? I can get this working (parts of it) but machines across the net i cant get connected to with winexe.
I’d generally expect everything to work over a VPN, though winexe uses named pipes, which some VPN’s don’t like.
I guess I’d expect a fully exposed box to work as well, though I haven’t got a good reason to try it.
Hi
I cant find the code for the following items?
rsyncd.secrets – can you provide an example? or how this file is generated
vss-setvar.cmd – can you provide the contents?
Thanks
Stephen
vss-setvar.cmd is generated automatically by vshadow
rsyncd.secrets is a list of usernames and passwords (you probably only need one) that match whatever’s contacting rsync, in this form:
username:password
@ Stephen – you also need a blank line at the end of the secrets file, so just hit return to give an extra space after your user/pass line.
@queued – i got it working successfully now, i used a program called hamachi from logmein that sets up a very simple vpn and have gotten good backups now. I do have another question, how would you change the scripts if the server has more than one drive? just add a line to the backuppc.cmd file to add the line for the next drive aka D:? would you have to modify any other scripts so that it knows to back up drive D: as well? Since vsrsync.cmd makes the volume shadow to B:, would you have to modify the script to first detach from B: once the backup is complete on the VS of C: and then remap the new VS of D: to B:? I would have no idea where to begin to code that. And i dont think you can setup 2 machines with the same IP address in backuppc and have one backup C: and the other D:, so how do you handle machines with more than one drive?
You can set up multiple machines with the same IP addresses in BackupPC, they just need to have different names. It’s pretty easy to manipulate a host file so that you can refer to the machine by “backup-c” and “backup-d,” for example, and BackupPC is happy enough to consider them completely independent. This is probably the simplest approach, since Volume Shadow Copies can’t cross volumes, and you’d essentially have to run one for each drive, anyway. For this approach, you’ll need separate \BackupPC directories (one per volume would do it) but all that would need to change is the launch script on the *nix side, and everywhere there’s an explicit reference to C: in the scripts. (You probably want to change the B: mapping, too, just in case both backups happen to run at the same time.)
The ugly alternative is that you launch a Volume Shadow Copy process for C:, have THAT launch the Volume Shadow Copy process for D:, then have that launch rsyncd. You’ll need to map multiple drives to your shadow copies, and then you’ll have two volume names in rsyncd. BackupPC would then see both volumes, and when rsyncd is killed, they’d (in theory) clean themselves up in reverse order. The inelegant part is that you’d be running multiple Volume Shadow Copies for the duration of the backup, which would generally make option #1 more appealing, which is inelegant only because BackupPC thinks it’s backing up two independent machines.
Thanks for the help
I think i have an rsync issue. Do i need to configure anything on the server for rsync?
I am not sure what passwords i should use for the secrets file.
How can i test rsync between the windows and linux box.
backuppc is set to:
xfermethod = rsync
rsyncsharename = B
Running backuppc.cmd using winexe stops at:
C:\BackupPC>rsync -v -v –daemon –config=rsyncd.conf –no-detach –log-file=diagnostic.txt
diagnostic.txt:
2008/10/03 22:44:29 [3652] rsyncd version 3.0.4 starting, listening on port 873
2008/10/03 22:55:16 [3992] rsyncd version 3.0.4 starting, listening on port 873
2008/10/03 22:59:42 [3992] _exit_cleanup(code=20, file=rsync.c, line=541): entered
2008/10/03 22:59:42 [3992] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(541) [receiver=3.0.4]
2008/10/03 22:59:42 [3992] _exit_cleanup(code=20, file=rsync.c, line=541): about to call exit(20)
2008/10/03 23:00:46 [948] rsyncd version 3.0.4 starting, listening on port 873
2008/10/03 23:01:24 [948] _exit_cleanup(code=20, file=rsync.c, line=541): entered
2008/10/03 23:01:24 [948] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(541) [receiver=3.0.4]
2008/10/03 23:01:24 [948] _exit_cleanup(code=20, file=rsync.c, line=541): about to call exit(20)
2008/10/03 23:14:47 [1152] rsyncd version 3.0.4 starting, listening on port 873
File.out:
C:\BackupPC>cd \backuppc
C:\BackupPC>vshadow -script=vss-setvar.cmd -exec=vsrsync.cmd c:
C:\BackupPC>REM @ECHO OFF
C:\BackupPC>call vss-setvar.cmd
[This script is generated by VSHADOW.EXE for the shadow set {12c90b7b-65ca-4e84-ad16-3ef29da005e1}]
C:\BackupPC>SET SHADOW_SET_ID={12c90b7b-65ca-4e84-ad16-3ef29da005e1}
C:\BackupPC>SET SHADOW_ID_1={2f9f0ac6-932a-479a-a4ea-0fb728d8448a}
C:\BackupPC>SET SHADOW_DEVICE_1=\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1
C:\BackupPC>cd \BackupPC
C:\BackupPC>SET CWRSYNCHOME=\BACKUPPC
C:\BackupPC>SET CYGWIN=nontsec
C:\BackupPC>SET CWOLDPATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem
C:\BackupPC>SET PATH=\BACKUPPC;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem
C:\BackupPC>dosdev B: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1
C:\BackupPC>REM Go into daemon mode, we’ll kill it once we’re done
C:\BackupPC>rsync -v -v –daemon –config=rsyncd.conf –no-detach –log-file=diagnostic.txt
C:\BackupPC>dosdev -r -d B:
VSHADOW.EXE 2.2 – Volume Shadow Copy sample client
Copyright (C) 2005 Microsoft Corporation. All rights reserved.
(Option: Generate SETVAR script ‘vss-setvar.cmd’)
(Option: Execute binary/script after shadow creation ‘vsrsync.cmd’)
(Option: Create shadow copy set)
(Gathering writer metadata…)
(Waiting for the asynchronous operation to finish…)
Initialize writer metadata …
Discover directly excluded components …
– Excluding writer ‘MSDEWriter’ since it has no selected components for restore.
Discover components that reside outside the shadow set …
Discover all excluded components …
Discover excluded writers …
Discover explicitly included components …
Verifying explicitly specified writers/components …
Select explicitly included components …
* Writer ‘Microsoft Writer (Bootable State)’:
– Add component \COM+ Registration Database
– Add component \Registry
* Writer ‘Microsoft Writer (Service State)’:
– Add component \Content Indexing Service
– Add component \Config Directory
– Add component \Event Logs
Creating shadow set {12c90b7b-65ca-4e84-ad16-3ef29da005e1} …
– Adding volume \\?\Volume{f8fc14d3-7449-11dd-b5df-806d6172696f}\ [C:\] to the shadow set…
Preparing for backup …
(Waiting for the asynchronous operation to finish…)
(Waiting for the asynchronous operation to finish…)
Creating the shadow (DoSnapshotSet) …
(Waiting for the asynchronous operation to finish…)
(Waiting for the asynchronous operation to finish…)
Shadow copy set succesfully created.
List of created shadow copies:
Querying all shadow copies with the SnapshotSetID {12c90b7b-65ca-4e84-ad16-3ef29da005e1} …
* SNAPSHOT ID = {2f9f0ac6-932a-479a-a4ea-0fb728d8448a} …
– Shadow copy Set: {12c90b7b-65ca-4e84-ad16-3ef29da005e1}
– Original count of shadow copies = 1
– Original Volume name: \\?\Volume{f8fc14d3-7449-11dd-b5df-806d6172696f}\ [C:\]
– Creation Time: 03/10/2008 22:44:22
– Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1
– Originating machine: (null)
– Service machine: (null)
– Not Exposed
– Provider id: {b5946137-7b9f-4925-af80-51abd60b20d5}
– Attributes: Auto_Release
Generating the SETVAR script (vss-setvar.cmd) …
– Executing command ‘vsrsync.cmd’ …
—————————————————–
—————————————————–
– Mark all writers as succesfully backed up…
Completing the backup (BackupComplete) …
(Waiting for the asynchronous operation to finish…)
(Waiting for the asynchronous operation to finish…)
Snapshot creation done.
You want to use the rsyncd backup method, not rsync.
Your logs look perfectly normal.
What goes in the rsyncd.secrets file is exactly the same username and password you specify as your rsyncdusername and rsyncdpassword in your BackupPC config file. It can be anything.
Thanks again for the help.
In the pc.pl config file am I backing up the B drive or another folder RsyncShareName = ???
$Conf{XferMethod} = ‘rsyncd’;
$Conf{RsyncdAuthRequired} = ‘1’;
$Conf{DumpPostUserCmd} =’/etc/backuppc/openfiles/postusercmd.sh $host’;
$Conf{DumpPreUserCmd} = ‘/etc/backuppc/openfiles/preusercmd.sh $host’;
$Conf{XferLogLevel} = ‘4’;
$Conf{RsyncdPasswd} = ‘*****;
$Conf{RsyncShareName} = [
‘\B’
];
The Volume Shadow is mapped to drive B:, which Cygwin considers to be /cygdrive/B, which is mapped to C by rsyncd.
In other words, your RsyncShareName should be ‘C’
I just wanted to point out that this layout is cutting off parts of your scripts. For example, both the bash shell scripts get cutt off on the line which calls winexe.
Copying / Pasting the scripts seems to work fine, but took me a long time reading through the scripts to figure out why I was missing something.
Thanks for the article though.
Thanks for all your help
Stephen
right now i use the ip addresses of the machines in backuppc’s host file. how would you list the same ip address for another host and have it know its a seperate drive? how can i name the hosts something like “backup-c” and still have it know what ip address it goes to? when i add a host in backuppc i use the “ip address” for the host dhcp=0 and then the backuppc user. if i wanted to use backup-c and backup-d for the host name, would i change the linux machines actual host file to be backup-c = ip address and the same with backup-d, so it can resolve it locally, or is there more steps to it?
thanks for the advice.
QUOTE
“You can set up multiple machines with the same IP addresses in BackupPC, they just need to have different names. It’s pretty easy to manipulate a host file so that you can refer to the machine by “backup-c†and “backup-d,†for example, and BackupPC is happy enough to consider them completely independent. This is probably the simplest approach, since Volume Shadow Copies can’t cross volumes, and you’d essentially have to run one for each drive, anyway. For this approach, you’ll need separate \BackupPC directories (one per volume would do it) but all that would need to change is the launch script on the *nix side, and everywhere there’s an explicit reference to C: in the scripts. (You probably want to change the B: mapping, too, just in case both backups happen to run at the same time.)”
In /etc/hosts:
192.168.0.100 backup-c backup-d
Now both backup-c and backup-d can be used to refer to the IP address 192.168.0.100.
DHCP complicates things, since IP addresses aren’t static… In that case, I’d probably add a few CNAMES to my DNS server to achieve the same goal.
Pingback: The Wiggling Fish » Blog Archive » Volume Shadow Copy Service SDK 7.2 - Compiling sample vshadow application
How did a restore work?
I would believe that this looked files can’t be restored?
BackupPC supports several types of restores; the kind where it creates a zip file of the requested files is pretty easy — while you can’t overwrite locked files, the trick, of course, is to restore the files when they’re not locked.
For most things, this is as simple as shutting down the program and/or user that has the files open — like shutting down Outlook or booting into safe mode. For files that are -always- open, and the only one that comes to mind is the registry, the recovery console seems to work perfectly well.
How hard would it be to add in some kind of check to see if the backup completed successfully? The problem im running into is that if a backup doesnt complete (connection drops, machine reboots etc) it deletes the volume shadow and the next time you run the backup, it makes a new volume shadow and restarts the backup. im mainly speaking of pst files or exchange data bases. I would like to see something that if it fails, it doesnt remove the current volume shadow, so when you restart the backup it can resume from the same shadow instead of making a new one. Would that be possible? Would you have any advice on how to do this (or possibly feel generous and do it) 🙂 Your solution is working great otherwise, just having this problem with a couple clients, having to restart 15GB files over a slow internet connection is making the backup take a couple weeks or more until by chance it actually completes without failure.
Hi,
Thanks for the nice tutorial and scripts. How should one go about with multiple drives ? If we do have multiple hosts created inside backuppc, will it not create problem when backuppc tries to initiate backup for both the hosts (being on the same pc) at the same time ?
Yes, it could be a problem if backuppc tries to back up two different drives, thinking they’re different hosts. A couple of solutions come immediately to mind:
What is a semaphore ?
A semaphore is a way of signaling that only one process should run at a time. For scripting, such as the pre backup command, I’ll often borrow a binary called “lockfile” from procmail.
lockfile -1 /tmp/my.lock
(the rest of the process runs)
rm -f /tmp/my.lock
(this allows the waiting process to run)
Really fantastic scripts, this adds a great deal of value to Backuppc, and it really didn’t take all that long to implement. You rock, thanks!
Did it work with Vista too?
i playing around with it on commandline but I have problems with the UAC.
Meaning: I only can run vshadow.exe if I have started the cmd as administrator. A normal user with administrator privileges seems not to be sufficient.
and I have a second issue in Vista.
dosdev B: %SHADOW_DEVICE_1%
do work, in cmd.
I also see the content of this volume shadow copy in B:
But in a cygwin shell I didn’t see it under /cygdrive and therefore I didn’t have access to the data.
Anyone knows something about that?
Thanks
I am sorry but i am confused here. Can you give more ideas on semaphone ? If i create a prebackup script which creates a lock file then what ? Do i also check in the other drives prebackup script that if the lock file exists ? And if the lock file exists, then what should the script do ?
Thanks for the help.
After much trial and error, I got the shadow copy and backuppc to work on multiple drives. Thanks to all the posts here. They got me started and thinking in the right direction. I hope this information is helpful to others. I don’t know if it something new in vshadow or what, but you can specify multiple drives on the vshadow line. It puts a different variable in the script file for each drive.
This is a copy of my backuppc.cmd file
cd \backuppc
vshadow -script=vss-setvar-c.cmd -p c: d:
call vss-setvar-c.cmd
vshadow -el=%SHADOW_ID_1%,B:
vshadow -el=%SHADOW_ID_2%,w:
REM Go into daemon mode, we’ll kill it once we’re done
rsync -v -v –daemon –config=rsyncd.conf –no-detach –log-file=diagnostic.txt
vshadow -da < y.bat
It works great and creates a record for each drive under each client.
After trying to implement Robert Phillips’ backuppc.cmd file, I found an interesting thing. There are different versions of vshadow for Server 2003 and Windows XP and windows Vista. Robert’s script argument for vshadow (the -el and the -p) will only work on Windows Server (and possibly on Vista). For Windows XP, I’ve successfully gotten the following to work:
backuppc.cmd:
cd \backuppc
vshadow -script=vss-setvar.cmd -exec=vsrsync.cmd c: d:
vrsync.cmd:
REM @ECHO OFF
call vss-setvar.cmd
cd \BackupPC
SET CWRSYNCHOME=\BACKUPPC
SET CYGWIN=nontsec
SET CWOLDPATH=%PATH%
SET PATH=\BACKUPPC;%PATH%
dosdev X: %SHADOW_DEVICE_1%
dosdev Y: %SHADOW_DEVICE_2%
REM Go into daemon mode, we’ll kill it once we’re done
rsync -v -v –daemon –config=rsyncd.conf –no-detach –log-file=diagnostic.txt
dosdev -r -d X:
dosdev -r -d Y:
rsyncd.conf:
use chroot = false
strict modes = false
pid file = rsyncd.pid
hosts allow = backuppc
hosts deny = 0.0.0.0/0
[C]
path = /cygdrive/X/
auth users = backuppc
secrets file = rsyncd.secrets
[D]
path = /cygdrive/Y/
auth users = backuppc
secrets file = rsyncd.secrets
Now to figure out some pre/post restore scripts to permit direct rsync restore to the clients!!
When executing preusercmd.sh on a WinXP x64, I was receiving “ERROR: Failed to install service winexesvc – NT_STATUS_INSUFF_SERVER_RESOURCES” . I was able to follow this KB to fix: http://support.microsoft.com/kb/177078 .
I pulled the dosdev from my tempdir after launching the MSReports tool @ http://www.microsoft.com/downloads/details.aspx?familyid=cebf3c7c-7ca5-408f-88b7-f9c79b7306c0&displaylang=en .
I also got the x64 vsshadow from the VSS SDK — google that.
From the Linux box, the preuser & postuser commands work from a shell, but BackupPC is telling me
2009-07-05 23:20:46 Backup failed on aos2 (DumpPreUserCmd returned error status 15) .
Any ideas on that?
I figured it out. websec was set to 770, and owned by root.root so the backuppc user couldn’t execute it.
Works beautifully now.
I saw on recent BackupPC users list that someone has compiled the VSS library header into a version of rsync.exe. http://www.consolejunky.net/cwrsync-vss/
Is there a greater advantage to using the compiled VSS rsync method versus this multi-file scripted method?
Pingback: Respaldo Con Linux - Foros de CHW
Thanks for the review! I want to say – thank you for this!