Backing Up Open Files on Windows with Rsync (and BackupPC)

Update:

Versions of the files below may be downloaded here.  This post is probably still useful as documentation.

 


 

This isn’t specific to BackupPC by any means, but I’ll preface this with a brief explanation:  BackupPC is a “set it and forget it” backup system driven from the server, that allows you to back up the entire network of *nix and Windows PCs.  It doesn’t require any software on the systems it backs up at all, since it relies upon rsync and smbclient, and optionally ssh.

For *nix, this works beautifully.  For Windows, this also works beautifully, except that “open files” can’t be backed up at all.  This problem isn’t unique to BackupPC, any attempt to back up or copy these files will fail, so most commercial backup systems have special “open file” clients to cope with it.

The official Windows solution for XP and later is something called a “volume shadow copy.”  It’s probably far more complex than it possibly needs to be, but essentially, it creates a pseudo-volume for any actual volume, with the difference being that you can actually back up files on it.  So, this can be handily used for rsync in order to make full backups, including every single file…  in theory, anyway.

My goals in getting this working:

  1. The solution should work with off-the-shelf components (i.e., no binaries or code)
  2. Installation and footprint should be minimal
  3. It should “just work” — if it’s too delicate, it’s not all that useful as a backup solution

It took quite a bit of trial-and-error, so I’ll skip what didn’t work, and get straight to what actually does work.  There are a few required components:

  1. winexe, a *nix program for remotely executing commands on Windows systems
  2. vshadow, a Windows program that creates and manages shadow copies
  3. dosdev, a Windows program that maps drive letters to volumes
  4. cwrsync, a Windows version of rsync (the “server” isn’t necessary)

Once all the pieces are assembled, I created a C:\BackupPC directory on the Windows box with all the necessary files.  Note that rsync does not need to be installed as a service, it actually gets loaded on-the-fly.  (Note that this directory is hard-coded in a lot of the files.) Here’s a listing of that directory:

Directory of C:\BackupPC
08/08/2008  07:11 PM                65 backuppc.cmd
08/10/2008  12:56 PM             1,928 cwrsync.cmd
07/22/2008  04:30 PM         1,082,368 cygcrypto-0.9.8.dll
04/11/2008  07:03 AM           999,424 cygiconv-2.dll
04/11/2008  07:03 AM            31,744 cygintl-3.dll
04/11/2008  07:03 AM            20,480 cygminires.dll
07/22/2008  04:30 PM         1,872,884 cygwin1.dll
04/11/2008  07:03 AM            66,048 cygz.dll
09/28/2004  02:07 PM             6,656 dosdev.exe
08/11/2008  11:08 PM             1,000 pre-cmd.vbs
08/11/2008  11:05 PM                44 pre-exec.cmd
07/22/2008  02:26 PM           348,160 rsync.exe
08/11/2008  10:12 PM               161 rsyncd.conf
08/11/2008  10:12 PM                22 rsyncd.secrets
08/11/2008  11:26 PM             1,177 sleep.vbs
06/08/2005  03:17 PM           294,912 vshadow.exe
08/11/2008  10:09 PM               581 vsrsync.cmd
08/11/2008  11:33 PM               308 vss-setvar.cmd

So, here’s how it works.  Before each backup, BackupPC has an option to call a local script first, waiting for that script to finish.  Here’s the execution chain:

  1. preusercmd.sh launches “pre-exec.cmd” on the Windows box
  2. preexec.cmd launches “pre-cmd.vbs”
  3. pre-cmd.vbs cleans up some files, launches “sleep.vbs” in the background (more on this later) and then launches “backuppc.cmd” in the background, and waits for the pid file to appear that signals that rsyncd has been launched
  4. backuppc.cmd launches vshadow, and tells it to execute vsrsync.cmd
  5. vsrsync.cmd maps the shadow volume to B:, and launches rsyncd — it sits and waits here, leaving vshadow and rsync open while the backup or rsync process runs — on the shadow copy on B:

Once the backup is completed, another local script is run — here’s its execution chain:

  1. postusercmd.sh puts a file called “wake.up” in the C:\BackupPC directory
  2. sleep.vbs wakes up, sees this file, reads rsyncd.pid, and kills the rsyncd process
  3. vsrsync.cmd now continues, since the rsync process is dead.  It unmaps the B: drive.  Once this script completes, vshadow automatically deletes the shadow volume.

Sure, it seems simple, but a lot of work went into that, since there are a lot of nuances to sort out.  Here are the file listings:

preusercmd.sh

#!/bin/bash
WINEXE=/usr/bin/winexe
UNAME="Administrator"
PWD="admin.password"
WRKGRP="WORKGROUP"
BOX=$1
$WINEXE --interactive=0 -U $UNAME -W $WRKGRP --password=$PWD //$BOX 'cmd /c c:\backuppc\pre-exec.cmd'
sleep 5
echo "Rsync and shadow copy loaded"
kill $$
# The script needs to be killed, otherwise, winexe waits for input

pre-exec.cmd

cd \backuppc
@echo off
cscript pre-cmd.vbs

pre-cmd.vbs

Const Flag = "C:\BackupPC\rsyncd.pid"
'
' Pid file shouldn't be there already
'
If DoesFileExist(Flag)=0 Then
   Set fso = CreateObject("Scripting.FileSystemObject")
   Set aFile = fso.GetFile(Flag)
   aFile.Delete
End If
'
' Nor should "wake.up"
'
If DoesFileExist("C:\BackupPC\wake.up")=0 Then
   Set fso = CreateObject("Scripting.FileSystemObject")
   Set aFile = fso.GetFile("C:\BackupPC\wake.up")
   aFile.Delete
End If
'
Set objShell = CreateObject("WScript.Shell")
objShell.Exec "cscript C:\BackupPC\sleep.vbs"
'
Set objShell = CreateObject("WScript.Shell")
objShell.Exec "C:\BackupPC\backuppc.cmd > C:\BackupPC\file.out"
'
' Just sleep until the file "rsyncd.pid" appears
'
While DoesFileExist(Flag)
   wscript.sleep 10000
Wend
'
' functions
'
function DoesFileExist(FilePath)
Dim fso
	Set fso = CreateObject("Scripting.FileSystemObject")
	if not fso.FileExists(FilePath) then
		DoesFileExist = -1
	else
		DoesFileExist = 0
	end if
	Set fso = Nothing
end function

sleep.vbs

Const Rsync = "C:\BackupPC\rsyncd.pid"
Const Flag = "C:\BackupPC\wake.up"
'
' Just sleep until the file "rsyncd.pid" appears
'
While DoesFileExist(Rsync)
   wscript.sleep 10000
Wend
'
' Now sleep until the file "wake.up" appears
'
While DoesFileExist(Flag)
   wscript.sleep 10000
Wend
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set aFile = fso.GetFile(Flag)
aFile.Delete
'
' It's time to kill Rsync
'
Set fso = CreateObject("Scripting.FileSystemObject")
Set aReadFile = fso.OpenTextFile(Rsync, 1)
strContents = aReadFile.ReadLine
aReadFile.Close
'
Set objShell = CreateObject("WScript.Shell")
objShell.Run "taskkill /f /pid " & strContents, 0, true
'
' Wait for Rsync to let go
'
wscript.sleep 5000
'
' Delete PID file
'
If DoesFileExist(Rsync)=0 Then
   Set objShell = CreateObject("WScript.Shell")
   objShell.Run "cmd /c del C:\BackupPC\rsyncd.pid", 0, true
End If
'
' functions
'
function DoesFileExist(FilePath)
Dim fso
	Set fso = CreateObject("Scripting.FileSystemObject")
	if not fso.FileExists(FilePath) then
		DoesFileExist = -1
	else
		DoesFileExist = 0
	end if
	Set fso = Nothing
end function

backuppc.cmd

cd \backuppc
vshadow -script=vss-setvar.cmd -exec=vsrsync.cmd c:

vsrsync.cmd

REM @ECHO OFF
call vss-setvar.cmd
cd \BackupPC
SET CWRSYNCHOME=\BACKUPPC
SET CYGWIN=nontsec
SET CWOLDPATH=%PATH%
SET PATH=\BACKUPPC;%PATH%
dosdev B: %SHADOW_DEVICE_1%
REM Go into daemon mode, we'll kill it once we're done
rsync -v -v --daemon --config=rsyncd.conf --no-detach --log-file=diagnostic.txt
dosdev -r -d B:

rsyncd.conf

use chroot = false
strict modes = false
pid file = rsyncd.pid
[C]
path = /cygdrive/B/
auth users = Administrator
secrets file = rsyncd.secrets

postusercmd.sh

#!/bin/bash
WINEXE=/usr/bin/winexe
UNAME="Administrator"
PWD="admin.password"
WRKGRP="WORKGROUP"
BOX=$1
PID=$($WINEXE -U $UNAME -W $WRKGRP --password=$PWD //$BOX 'cmd /c echo '1' > c:\backuppc\wake.up')
echo "Rsync and shadow copy unloaded"
Share
Tagged , , , , . Bookmark the permalink.

100 Responses to Backing Up Open Files on Windows with Rsync (and BackupPC)

  1. Neubian says:

    Is anyone else noticing very high CPU utilization for winexe during the backups ? I don’t see why it would be high at all, but when backing up 2 machines at the same time, Nagios load monitors are tripped on my Linux box.

    Anything I can do to curb that?

  2. queued says:

    I suppose you could always use nice or renice for the winexe process, but in my experience, it does consume a large amount of CPU while making the initial connection. I’m not sure why.

  3. Robert Phillips says:

    I now have one Windows 2008 server that needs to be backed up. I found out it uses something called DiskShadow not vshadow. I had to modify my files like the following.

    -backuppc.cmd-

    c:
    cd \backuppc
    diskshadow /s DiskShadowScript.txt

    del c:\backuppc\*.cab /q

    -DiskShadowScript.txt –

    #DiskShadow script file
    set context persistent
    set metadata c:\backuppc\example.cab
    set verbose on
    begin backup
    add volume c: alias SystemVolumeShadow

    create

    expose %SystemVolumeShadow% B:

    exec c:\backuppc\Serverbackup.cmd
    unexpose B:
    end backup
    delete shadows set %vss_shadow_set%

    #End of script

    -Serverbackup.cmd –

    c:
    cd \backuppc
    REM Go into daemon mode, we’ll kill it once we’re done
    rsync -v -v –daemon –config=rsyncd.conf –no-detach –log-file=diagnostic.txt

  4. Robert Phillips says:

    I was noticing alot of failed backups at different times. Not all of the systems were failing but several each night. (Child exited prematurely) I put “port = ” in the rsync.conf file and made sure the config file in Backuppc matched. I used a different port number for each server and I don’t seem to have as many failures now.

  5. Neubian says:

    I found a way to deal with the excessive CPU utilization of winexe. I still don’t know why it goes bonkers when BackupPC launches it, because I couldn’t reproduce it on the command line, even sudo’ing as backuppc .

    AOS2 is an XP x64 machine.

    The key was to send winexe to the background so that it could be killed later. Otherwise the bash script never gets past the winexe line.

    It’s fugly, but it works.

    preusercmd.sh
    #!/bin/bash
    WINEXE=/usr/bin/winexe
    UNAME=”backuppc”
    PWD=””
    WRKGRP=”WORKINIT”
    BOX=$1
    if [ $BOX == “aos2″ ] ; then
    $WINEXE –interactive=0 –ostype=2 -U $UNAME -W $WRKGRP –password=$PWD //$BOX ‘cmd /c c:\backuppc\pre-exec.cmd’ &
    else
    $WINEXE –interactive=0 -U $UNAME -W $WRKGRP –password=$PWD //$BOX ‘cmd /c c:\backuppc\pre-exec.cmd’ &
    fi
    WINEXEPID=”$pids $!”
    sleep 5
    echo “Rsync and shadow copy loaded\n”
    kill $WINEXEPID
    kill $$
    # The script needs to be killed, otherwise, winexe waits for input

  6. Trevor says:

    I don’t understand the point of a backup that does not restore easily. The way this is configured the connection is to the shadowed drive or nothing. Why?

    Shouldn’t the actual backup use the shadowed drive and the restore connect to the regular (not shadowed) drive/path so that files can be restored – easily?

    Sure, locked files won’t restore, but those are the exception, not the norm. For me, the usual restore is because somebody deleted a file accidentally , or overwrote a file accidentally, or otherwise corrupted a needed file.

    I strongly suggest creating restore scripts that link to an unshadowed, non read-only drive(path) so that the built-in restore feature works for the majority of scenarios.

    P.S. You cant restore a zipped file this way either because the whole path/drive (when shadowed) is read-only!

  7. queued says:

    There’s nothing in the backup that implies how it must be restored; the techniques above are generally used for backing up, not recovery — and recovery is usually via smb, rsync, or zip files to an unshadowed drive. Aside from the registry, most files can be unlocked by shutting down the application that controls them (e.g., Outlook) and recovering normally. It’s possible to recover the entire OS, registry and all, with the help of the recovery console — but as you note, most recoveries are for a file here and there, not the entire drive.

  8. Pingback: 飄狂山莊 | [筆記] 利用robocopy , vshadow,dosdev 來達成備份windows open file backup的目標

  9. Pingback: [筆記] 利用robocopy , vshadow,dosdev 來達成備份windows open file backup的目標 « 飄狂山莊

  10. Pingback: Restoring Windows XP with BackupPC | Good Job Sucking

  11. higuita says:

    Hi, i just implemented this and its working fine… but i had some pitfalls and blank spaces that made this a littler harder, so this update might be useful for others:

    1-the backuppc host config file… this are mostly the needed things. Of course, tune the log level after testing

    # cat /etc/backuppc/test.pl
    $Conf{XferMethod} = ‘rsyncd’;
    $Conf{ClientCharset} = ‘cp1252’;
    $Conf{RsyncShareName} = [
    ‘users’
    ];
    $Conf{DumpPostUserCmd} = ‘/etc/backuppc/bin/postusercmd.sh $host’;
    $Conf{DumpPreUserCmd} = ‘/etc/backuppc/bin/preusercmd.sh $host’;
    $Conf{XferLogLevel} = ‘3’;
    $Conf{RsyncdPasswd} = ‘rsync-pass’;
    $Conf{RsyncdUserName} = ‘backuppc’;

    2- after some search, i found a small zip with all this files, that helps alot… it is in here:

    http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/backuppc-21/backuppc-file-rsyncp-issues-100060/index-30.html

    sadly the included vshadow is old, XP only and doesnt work in vista and windows7, so it needs the latest version (might not work in XP, but use the older one for that).
    But to this this latest version i had to download 1.3GB of a MS dev ISO to get that damn file (about 221KB)… luckily cabextract manage to get the file without having to install that huge SDK just to get the vshadow.exe

    i updated the zip file, add a some new files (see below) and uploaded to here:
    http://caravela.homelinux.net/~higuita/backuppc/backuppc-rsyncd+vss-windows.tar.gz

    ps: dont forget to change the usernames, password and domains

    4-a example rsyncd.secret, useful for people that dont know rsync, its just a “username:password” plain file, one user in each line

    5-a rsyncd.conf example… please note that the backuppc do the backup to the rsyncd repository name, it doesnt accept any other path (ie: rsyncd defines the [home], backuppc makes the backup to “home”, “/home/user1/Documents” isnt valid, create a repository directly to that path if you need it)

    use chroot = false
    strict modes = false
    pid file = rsyncd.pid
    UID = 0
    GID = 0

    [home]
    path = /cygdrive/B/Users/
    auth users = backuppc
    secrets file = rsyncd.secrets
    # if you want to exclude files on the windows site, use this… useful for global temporary dirs, windows junction points, etc
    # no example here, because i dont use it yet, testing the exclude on the backuppc side
    #exclude from = /cygdrive/c/admin/backuppc/exclude_user.txt

    6-A small install script for the backuppc machine, to install the client in windows:

    #
    #set -x
    host=$1

    if [ ! -z $host ] ; then
    cd /root/windows
    nano BackupPC/rsyncd.conf
    smbclient //$host/c$ -U backup-admin%passdword -W domain <<EOF
    mkdir admin
    cd admin
    mkdir backuppc
    cd backuppc
    prompt
    lcd BackupPC
    mput *
    dir
    exit
    EOF

    else
    echo install-pc.sh [hostname]
    echo install on c:\\admin\\backuppc
    fi

    7-for the winexe to work, you need a admin user, the firewall disables (or at least open for the backuppc server) and domain policy/local policy allowing the remote access for that user… and for my install script, the c$ share enabled

    8-if you use my tar.gz package, please check the scripts and scan the binaries for virus… you shouldn't trust me when those scripts will have access to windows admin accounts

    good luck
    higuita

  12. egrimisu says:

    Hi higuita, i folowed you install note from the archive file http://caravela.homelinux.net/~higuita/backuppc/backuppc-rsyncd+vss-windows.tar.gz, i have changed the usernames password and domain names, i made sure that no firewall is blocking the backuppc server, i used domain admin user in preusercmd.sh and postusercmd.sh to make sure it can acces all pc-s in the domain, used cp1252 charset, the commands in backuppc webgui for post and pre look like /etc/backuppc/bin/post(an pre)usercmd.sh $host but i got the folowing message form backuppc 2010-05-27 09:46:33 Backup aborted (inet connect: Connection refused). I believe this message is returned when the connection between backuppc and the clinet pc is not made via rsync. What have i done wrong?

    i’ll paste the setting files:

    PREUSERCMD.SH
    #!/bin/bash
    #set -x
    WINEXE=/etc/backuppc/bin/winexe
    UNAME=”baltagul”
    PWD=”miorita”
    WRKGRP=”DDOO”
    BOX=$1
    $WINEXE –interactive=0 -U $UNAME -W $WRKGRP –password=$PWD //$BOX ‘cmd /c c:\admin\backuppc\pre-exec.cmd’

    #if winexe uses too much CPU, try to background it and kill it later
    #$WINEXE –interactive=0 -U $UNAME -W $WRKGRP –password=$PWD //$BOX ‘cmd /c c:\admin\backuppc\pre-exec.cmd’ &
    #WINEXEPID=”$pids $!”
    #sleep
    #kill $WINEXEPID

    echo “Rsync and shadow copy loaded”
    kill $$
    # The script needs to be killed, otherwise, winexe waits for input

    POSTUSERCMD.SH
    #!/bin/bash
    #set -x
    WINEXE=/etc/backuppc/bin/winexe
    UNAME=”baltagul”
    PWD=”miorita”
    WRKGRP=”DDOO”
    BOX=$1
    $WINEXE –interactive=0 -U $UNAME -W $WRKGRP –password=$PWD //$BOX ‘cmd /c c:\admin\backuppc\pre-exec.cmd’
    #if winexe uses too much CPU, try to background it and kill it later
    #$WINEXE –interactive=0 -U $UNAME -W $WRKGRP –password=$PWD //$BOX ‘cmd /c c:\admin\backuppc\pre-exec.cmd’ &
    #WINEXEPID=”$pids $!”
    #sleep
    #kill $WINEXEPID
    echo “Rsync and shadow copy loaded”
    kill $$
    # The script needs to be killed, otherwise, winexe waits for input

    rsyncd.secrets
    backupp:miorita

  13. queued says:

    You need to open your firewall — in short, “rsync” needs to be added to programs allowed to make connections through the firewall.

  14. higuita says:

    egrimisu:

    that mean that rsync didnt start or that you cant connect to the machine… for the later, its the firewall, for the first its probably a problem with the VSS service on the windows machine.

    try to run the scripts in the command line, specially the c:\admin\backuppc\pre-exec.cmd and check the windows event viewer.

    sadly, VSS is hard to debug, for about 100 windows machine, i have right now 3 with VSS problems that i’m still trying to fix.
    this url is useful for most problems:

    http://www.gfi.com/blog/vss-troubleshooting/

    if the VSS still doesn’t work, you can try to enable the debug

    http://support.microsoft.com/default.aspx?scid=kb;EN-US;887013

    and/or contact MS or reinstall.

    good luck

  15. falko says:

    Thank you very much for the detailed explanations and downloads, works great.

    My question: what can I do if not only C:, but rather C: and D: partitions should be backed up? We have our “users” folder at D: partition …

    Thanx in advance

  16. falko says:

    Ok, I’ve got it. It seems as easy as:

    In backuppc.cmd
    vshadow -script=vss-setvar.cmd -exec=vsrsync.cmd c: d:

    In vsrrsync.cmd:

    dosdev A: %SHADOW_DEVICE_1%
    dosdev B: %SHADOW_DEVICE_2%

    dosdev -r -d A:
    dosdev -r -d B:

    In rsyncd.conf:
    [system]
    path = /cygdrive/A/

    [users]
    path = /cygdrive/B/

    $Conf{RsyncShareName} = [
    ‘system’,
    ‘users’
    ];

  17. higuita says:

    I recently started to have problems with winexe and some windows machines, the winexe failed with:
    winexe: relocation error: /lib32/libresolv.so.2: symbol strlen, version GLIBC_2.0 not defined in file libc.so.6 with link time reference

    i was using the static build, version 0.8 IIRC, but the webpage is no more. luckily, the author moved to sourceforge and update the license to GPL3:
    http://winexe.sourceforge.net
    So i grabbed the source, version 0.91, compiled (tar zvxf winexe-*.tar.gz ; cd winexec ; cd source4; ./autogen.sh && ./configure && make ; sudo cp bin/winexe /etc/backuppc/bin ) and its now working fine again. i will update my package later, but my “new” package will be dynamic build against debian testing… might not work in other distros… but recompile is easy enough to do it yourself.

    also, i will add the vshadow for windows64, as the 32bit one doesnt work… so now there are 3 vshadow: vshadow-xp, vshadow-32 and vshadow-64 and their use should be autodetected by the script

  18. Olivier says:

    I couldn’t get winexe to connect to a Windows 7 computer (had a “NT_STATUS_ACCESS_DENIED” error). Thanks to Wireshark, I found out that winexe couldn’t access the ADMIN$ share.

    On this forum : http://help.lockergnome.com/vista/access-administrative-share-enabling-admin-accoun–ftopict25338.html , Jimmy Brush gives the solution. You have to add the “LocalAccountTokenFilterPolicy” key to the registry. This soution should also works for Windows Vista. Now I can backup my Outlook files 🙂

    Also, thanks for this great blog post, helped me a lot !

  19. Karlos says:

    @higuita
    I am only interested if you have already added vshadow for windows64, and if where can be downloaded?

    Thanks

  20. queued says:

    The 64-bit version of vshadow is in the Windows 7 SDK, in the “Developer’s tools” section, which is installed in %ProgramFiles%\Microsoft SDKs\Windows\v7.0\Bin\x64\vsstools\vshadow.exe.

  21. higuita says:

    A info that might be useful of others as there is no info about this error in the net.
    its also good future reference for me 😉

    When trying to run a windows backup, it failed, checking by hand the /etc/backuppc/bin/preusercmd.sh i got this error from winexe:

    ERROR: Failed to install service winexesvc – NT_STATUS_CANT_WAIT

    After some research, i found that winexe was trying to install (or at least use) the winexesvc service, but a 64bit one (?!). No idea how that was installed on a 32bit machine!

    To fix, i run:

    /etc/backuppc/bin/winexe –reinstall –interactive=0 –ostype=0 -U (user) -W (domain) –password=(pass) //(machine) ‘cmd’

    ie: force the -reinstall and force the –ostype=0 (0=32bit, 1=64bit, default 2=autodetect that was wrongly detecting in this case)

    So if the winexec fails, try to reinstall the service, and of course, check the permissions/domain of the remote windows computer

    I had another problem, with a machine failling to backup, but manually running the script via a winexe cmd i got this:

    c:\admin\backuppc\> cscript pre-cmd.vbs
    C:\admin\backuppc\pre-cmd.vbs(35, 2) Microsoft VBScript runtime error: ActiveX component can’t create object: ‘Scripting.FileSystemObject’

    the fix is running from the c:\windows\system32\ :
    regsvr32.exe vbscript.dll
    regsvr32.exe jscript.dll
    regsvr32.exe ScrRun.dll

    finally, i also had some other errors when manually debuging:

    the VSS_E_PROVIDER_VETO, that from the docs its the computer that is out of disk space, or the HD is not NTFS (run convert.exe to convert vfat to ntfs) or the user is using too much IO, not allowing enough room for vss to run

    other VSS errors are mostly fixed by following this MS doc: http://support.microsoft.com/kb/940184

    I didnt have it, but if you had installed in the past visio, there is a known problem with it when uninstall and vss (Error: 0x8000FFFF on vss): http://support.microsoft.com/kb/907574

    Finally, i found that winexe doesnt timeout when he connected to a machine, but then it was turned off, leaving a active winexe that never dies… i made a simple script that i add in the cron to kill this zombie sessions…
    cron:
    */10 * * * * /etc/backuppc/bin/clean-winexe.sh

    /etc/backuppc/bin/clean-winexe.sh :
    #!/bin/bash

    # some winexe never die when the remote host shutdown after the connect
    # so try to detect and kill them
    # please note that the time is cpu time, so 6min of CPU is a long time
    ps xua | awk ‘ /backuppc/ && /winexe/ && ! /awk/ && ! /SCREEN/ {
    time=$10
    gsub(/.*:0?/,””,time)
    if( time >= 6 ) { system (“kill ” $2) }
    }’

    All this erros might look like this gives all sort of problems, but they are normal for a big number of machines and they happen mostly when installing/first backup.
    I have right now about 690 machines in backuppc on about 6TB of disks with Mac(ssh+rsync), Linux (ssh+rsync), Windows 2000/XP/Vista/7 (rsyncd), both 32bit and 64bit machines, doing backups every hour, every day and everything is working fine

    I update my package on http://caravela.homelinux.net/~higuita/backuppc/backuppc-rsyncd+vss-windows.tar.gz and you can view/grab individual files by browsing the URL http://caravela.homelinux.net/~higuita/backuppc/
    It now supports windows 2000 and above (except windows XP 64bit, but no one uses that anyway), workaround most problems i have found and several scripts that might help managing backuppc

    have fun 🙂

  22. Karlos says:

    Really great infos!

    If someone else will need winexe for FreeNAS, here is my compiled version

    Winexe 0.91 for FreeBSD 7.3 – FreeNAS 0.7.1
    http://www.karlosp.net/blog/wp-content/uploads/2010/10/Winexe-FreeBSD.zip

  23. Pingback: File Backup Scheme – How to make effective backups

  24. Marco says:

    Hi, thanks for that great addon to BackupPC !
    i use the modifed solution made by higuita

    . i’m glad to finally being able to backup open file from my windows box (so far testing ok on vista and XP clients),but for some reason , now, my backup files lost their French special characters when trying to restore files. if use cp1252 charset, the backup file names are ok in the web interface but not ok when restoring. if i leave the charset empty, the names of the backup files looks ok in the web interface but still not ok when trying to restore…
    everything was working fine before when using the cygwin-rsync package from “backuppc.sourceforge.net”. any advice ?

    my server is Debian 5.04 using BackupPC 3.1-0-4lenny.

    Thanks

  25. Marco says:

    Oups, typo 🙂

    what i meant was: if use cp1252 charset, the backup file names are “BAD” in the web interface and bad when restoring. if i leave the charset empty, the names of the backup files looks ok in the web interface but still bad when trying to restore…

    i tried to replace rsync and and cygwin1.dll with other versions i found on the net but in this case, the backup never start..it block at “backup Starting..”

    i also found this error message when i replace rsync or the Dll on the host: ”
    Can’t call method “getStats” on an undefined value at /usr/share/backuppc/bin/BackupPC_dump line 1143,”
    don’t know if that help ?

  26. Marco says:

    Ok, to respond to myself and to others:

    there will be no problem with special characters when restoring files if you use Firefox and forget IE (like i should i’ve done a long time ago..)

    I LUV BACKUPPC ! (i need my BackupPC t-shirt 😉 )

  27. higuita says:

    one small update…

    winexe requires also the admin$ share… one developer decided to disable the default windows shares and winexe failed to connect… to fix, just reboot (if it was just removed from the share list) or do the reverse of this site (if the user disabled permanently the shares):
    http://www.windowsnetworking.com/kbase/WindowsTips/WindowsXP/AdminTips/Network/DisableWindowsNTW2KXPHiddenAdministrativeShares.html

    good luck

  28. higuita says:

    Yet Another Update 🙂

    People with problems doing the shadow copy due to big virtual machines or other files on low HD space, use this doc to exclude files from the VSS snapshot

    http://msdn.microsoft.com/en-us/library/aa819132(v=vs.85).aspx

    example:

    Key: VMDK (REG_MULTI_SZ)
    Value: “C:\My VMWare Virtual Machines\XPSP3\*.vmdk”

    on both the registry trees:

    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Controlaos\BackupRestore\FilesNotToSnapshot
    HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\BackupRestore\FilesNotToBackup

    good luck

  29. Pingback: Green backups with BackupPC and WOL | Good Job Sucking

  30. Драган says:

    Is there official webpage where DOSDEV can be downloaded?

  31. paperAdvocate says:

    I’ve been using your VSS solution via winexe and your scripts for BackupPC for a couple of years, it’s still a thing of beauty.

    I’ve started updating my CentOS boxes from 5 to 6. The winexe that you referenced doesn’t compile on 6 and the other versions from sourceforge or even rpm packages won’t work properly (the 2nd rsync instance is never launched). So I’m stuck with CentOS 5.x and the winexe package you referenced. That’s ok, but I’m concerned about the future. What distro and winexe source are you using today?

    BTW, you’re solution is simple and wonderful.

  32. higuita says:

    for Драган:
    DOSDEV homepage is this one http://www.ltr-data.se/opencode.html/

    for paperAdvocate:

    i’m using debian with a PPA with winexe:

    deb http://ppa.launchpad.net/jdthood/winexe/ubuntu natty main

    choose the correct version in here:

    http://ppa.launchpad.net/jdthood/winexe/ubuntu/

    this should work in any debian/ubuntu based distro

    for other distros, you can try the winexe homepage:
    http://sourceforge.net/projects/winexe/files/

  33. Vladac says:

    I’m getting error:
    Got fatal error during xfer (inet connect: Connection timed out)

    How do I start rsync on windows ? Do I have to have admin shares (c$, d$, admin$, …) ?

  34. queued says:

    No, but you have to have sharing on in order for winexe to work.
    If you use these scripts, winexe will launch rsync, but you need to make sure the firewall is open.
    An updated set of scripts is available for download from http://www.michaelstowe.com/backuppc

  35. Vladac says:

    winexe is not working.

    I started rsyncd as service (downloaded from main backuppc site)
    Now I’m getting error: Backup aborted (auth failed on module home)
    Strange as I have correct username and pass on both sides

  36. queued says:

    That error means you don’t have a module named “home” in your rsyncd.conf. The scripts here use a module named “C”. The module name is unimportant, but it has to match between rsync and BackupPC.

  37. Vladac says:

    I created c:\rsyncd and backups are working.

    Now I’m trying to use these scripts. But when I disable rsync on Windows XP services I get this error
    Backup aborted (inet connect: Connection refused)
    As if winexe is not working.

  38. Vladac says:

    Ok got it working…

    Now i have to kill winexe service on windows xp to let backuppc transfer files.
    IF I let winexe service to run it will run for ages and do nothing. Tried with crontab but same thing.

  39. queued says:

    That’s a known issue with some versions of winexe — apparently the solution is to add <&- to the end of the command line, as is done on the most recent versions of the scripts.

  40. upen says:

    Dear Michael,
    Thank you for the efforts. You have made life real easier for people. Gotta love the click and install in windows, lol.

  41. Bullett says:

    Firstly, thanks for the neat little download package for my Windows box. It’s been very helpful.

    Unfortunately, I’m having difficulty getting backups to complete. I’m getting the following error:


    2013-04-29 12:15:56 Output from DumpPreUserCmd: warning: commands will be executed using /bin/sh
    2013-04-29 12:15:56 Output from DumpPreUserCmd: job 31 at Mon Apr 29 12:15:00 2013
    2013-04-29 12:15:56 incr backup started back to 2013-04-11 02:00:01 (backup #7) for directory /BackupArea
    2013-04-29 12:15:57 Output from DumpPostUserCmd: Rsync and shadow copy unloaded
    2013-04-29 12:15:57 Got fatal error during xfer (Unknown module '/BackupArea/')
    2013-04-29 12:16:02 Backup aborted (Unknown module '/BackupArea/')

    As I can’t find any other errors, I’m assuming this is something to do with the trailing slash after the module name ‘BackupArea’ on the ‘Got fatal error’ line.

    The thing is, I’ve not added a trailing slash anywhere. My rsync.conf module name is set with [BackupArea] and my BackupPC Xfer settings have the module set to ‘/BackupArea’, so I’ve no idea where the trailing slash is coming from, but it is stopping anything from working!

    Does anyone have any suggestions where I should look?
    Thanks,
    Bullett

  42. Bullett says:

    Well, I’ve managed to make it work…

    I removed the leading slash from ‘/BackupArea’ and the backups have been running successfully ever since. I find this odd because I’d tried this previously, before my question, and it failed with auth errors!

    I’m now not sure if the leading slash is required. I had it in place from when I’d previously used a regular rsync via cygwin, before picking up the VSS version from this page.

    Having duplicated the client setup to another machine, I can confirm this does not require the leading slash in the BackupPC config either.

  43. WhilelM says:

    I can’t get it to work. I’m stuck with ” auth failed on module C” message.
    I checked login/pass on my auth.sh.

    But i don’t understand why rsyncd.secret remain ’empty’ with your installer. I have entered rsync username and password but rsyncd.secret contain only the ‘:’ caracter.
    And even if get it with the line “user:password”, it still don’t work.
    Any hint ?

  44. queued says:

    The installer has a bug where it won’t save what goes in rsyncd.secret properly, but any text editor should work.

    Check rsyncd.conf to make sure you have a module C and that the credentials in rsyncd.secrets match what’s in BackupPC for the rsync transfer.

    To be clear: auth.sh needs to correspond to the Windows account which launches vshadow/rsync
    BackupPC needs to correspond to rsyncd.secrets.

  45. Computer King says:

    I keep getting these errors NT_STATUS_CONNECTION_REFUSED it works fine with a backup server on my local network of 10.22.20.0 but not on my vpn network of 127.16.1.0 below is the full error.

    NOTE i can ping both to and from the remote backup server and have windows firewall disabled for testing

    Executing DumpPreUserCmd: /etc/backuppc/scripts/preusercmd.sh 127.16.1.2
    ERROR: Failed to open connection – NT_STATUS_CONNECTION_REFUSED
    warning: commands will be executed using /bin/sh
    job 187 at Fri Dec 13 15:16:00 2013
    full backup started for directory C
    Error connecting to rsync daemon at 127.16.1.2:873: inet connect: Connection refused
    Executing DumpPostUserCmd: /etc/backuppc/scripts/postusercmd.sh 127.16.1.2 0
    ERROR: Failed to open connection – NT_STATUS_CONNECTION_REFUSED
    Rsync and shadow copy unloaded
    Got fatal error during xfer (inet connect: Connection refused)
    Backup aborted (inet connect: Connection refused)
    Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 0)

  46. Marco says:

    since I upgraded one of my windows 8 box to windows 8.1, Backuppc stopped working. the error is “connection refused”.

    Since all my original settings seems ok and admin share still working, I suspected compatibility issues with winexe and Windows 8.1. that URL appear to validate my suspicions:
    http://sourceforge.net/p/winexe/bugs/30/?page=1

    so the developer of Winexe mention that a beta patch is available (winexe 1.1) but he only published high level recipe for building your own patch version of winexe 1.1.

    is there someone with enough skills to help me to build a patch version of winexe 1.1 ? (i’m green in Linux apps tools..)

    i’m using debian 5.0.4 32 bits as backuppc server.

    thanks !
    sorry for my English..
    Marco

  47. higuita says:

    As many people found already, pre-compiled winexe files all over the internet are too old now, you need to compile it your self. but don’t worry, is not that hard.

    Go to this githut project, clone it and read the docs:

    https://github.com/opinkerfi/winexe-waf

    i use debian and followed the “Instructions for Samba shared library build”. After building, just copy the winexe to the /usr/bin or other place you want (don’t forget to update the backuppc scripts to use the correct version)

    when things fail again, just update the distro, update the git and build it again.

    Any distro with samba4 may use source to build, so don’t be scared by the debian instructions, just adapt to your distro.

  48. Majales says:

    Hi. Is there any chance to download from anywhere above mentioned files from now non existant website http://caravela.homelinux.net/~higuita/backuppc/ ??
    I’ve just hit this website and I want to use backuppc to backup our windows servers.

  49. queued says:

    The link at the top of the article should be working again — it appears that a DDOS attack of a number of name servers is causing some sites (like that one and this one) to be unable to be resolved temporarily, which was affecting those, and is most likely affecting other sites as well.

  50. higuita says:

    @Majales

    Sorry, dyndns terminating the free dyndns forced me to change the domain…

    new address is:
    http://caravela.motaleite.net/~higuita/backuppc/

    Also check the http://www.michaelstowe.com/backuppc/

Leave a Reply

Your email address will not be published. Required fields are marked *

By submitting this form, you accept the Mollom privacy policy.