{"id":62,"date":"2008-08-11T23:55:30","date_gmt":"2008-08-12T05:55:30","guid":{"rendered":"http:\/\/www.goodjobsucking.com\/?p=62"},"modified":"2012-06-05T13:44:22","modified_gmt":"2012-06-05T19:44:22","slug":"backing-up-open-files-on-windows-with-rsync-and-backuppc","status":"publish","type":"post","link":"http:\/\/www.goodjobsucking.com\/?p=62","title":{"rendered":"Backing Up Open Files on Windows with Rsync (and BackupPC)"},"content":{"rendered":"<p>Update:<\/p>\n<p>Versions of the files below may be downloaded <a title=\"here\" href=\"http:\/\/www.michaelstowe.com\/backuppc\">here<\/a>.\u00a0 This post is probably still useful as documentation.<\/p>\n<p>&nbsp;<\/p>\n<hr \/>\n<p>&nbsp;<\/p>\n<p>This isn&#8217;t specific to <a title=\"BackupPC\" href=\"http:\/\/backuppc.sourceforge.net\/\">BackupPC<\/a> by any means, but I&#8217;ll preface this with a brief explanation:\u00a0 <a title=\"BackupPC\" href=\"http:\/\/backuppc.sourceforge.net\/\">BackupPC<\/a> is a &#8220;set it and forget it&#8221; backup system driven from the server, that allows you to back up the entire network of *nix and Windows PCs.\u00a0 It doesn&#8217;t require any software on the systems it backs up at all, since it relies upon rsync and smbclient, and optionally ssh.<\/p>\n<p>For *nix, this works beautifully.\u00a0 For Windows, this also works beautifully, except that &#8220;open files&#8221; can&#8217;t be backed up at all.\u00a0 This problem isn&#8217;t unique to BackupPC, any attempt to back up or copy these files will fail, so most commercial backup systems have special &#8220;open file&#8221; clients to cope with it.<\/p>\n<p>The official Windows solution for XP and later is something called a &#8220;<a title=\"How Volume Shadow Copy Works\" href=\"http:\/\/technet.microsoft.com\/en-us\/library\/cc785914.aspx\">volume shadow copy<\/a>.&#8221;\u00a0 It&#8217;s probably far more complex than it possibly needs to be, but essentially, it creates a pseudo-volume for any actual volume, with the difference being that you can actually back up files on it.\u00a0 So, this can be handily used for rsync in order to make full backups, including every single file&#8230;\u00a0 in theory, anyway.<\/p>\n<p>My goals in getting this working:<\/p>\n<ol>\n<li>The solution should work with off-the-shelf components (i.e., no binaries or code)<\/li>\n<li>Installation and footprint should be minimal<\/li>\n<li>It should &#8220;just work&#8221; &#8212; if it&#8217;s too delicate, it&#8217;s not all that useful as a backup solution<\/li>\n<\/ol>\n<p>It took quite a bit of trial-and-error, so I&#8217;ll skip what didn&#8217;t work, and get straight to what actually <em>does<\/em> work.\u00a0 There are a few required components:<\/p>\n<ol>\n<li><a title=\"winexe\" href=\"http:\/\/eol.ovh.org\/winexe\/\">winexe<\/a>, a *nix program for remotely executing commands on Windows systems<\/li>\n<li><a title=\"vshadow\" href=\"http:\/\/msdn.microsoft.com\/en-us\/library\/bb530725.aspx\">vshadow<\/a>, a Windows program that creates and manages shadow copies<\/li>\n<li><a title=\"dosdev\" href=\"http:\/\/blogs.msdn.com\/adioltean\/archive\/2005\/10\/04\/477164.aspx\">dosdev<\/a>, a Windows program that maps drive letters to volumes<\/li>\n<li><a title=\"cwrsync\" href=\"http:\/\/www.itefix.no\/i2\/node\/10650\">cwrsync<\/a>, a Windows version of rsync (the &#8220;server&#8221; isn&#8217;t necessary)<\/li>\n<\/ol>\n<p>Once all the pieces are assembled, I created a C:\\BackupPC directory on the Windows box with all the necessary files.\u00a0 Note that rsync does <em>not<\/em> need to be installed as a service, it actually gets loaded on-the-fly.\u00a0 (Note that this directory is hard-coded in a lot of the files.) Here&#8217;s a listing of that directory:<\/p>\n<pre style=\"padding-left: 30px;\">Directory of C:\\BackupPC\r\n08\/08\/2008  07:11 PM                65 backuppc.cmd\r\n08\/10\/2008  12:56 PM             1,928 cwrsync.cmd\r\n07\/22\/2008  04:30 PM         1,082,368 cygcrypto-0.9.8.dll\r\n04\/11\/2008  07:03 AM           999,424 cygiconv-2.dll\r\n04\/11\/2008  07:03 AM            31,744 cygintl-3.dll\r\n04\/11\/2008  07:03 AM            20,480 cygminires.dll\r\n07\/22\/2008  04:30 PM         1,872,884 cygwin1.dll\r\n04\/11\/2008  07:03 AM            66,048 cygz.dll\r\n09\/28\/2004  02:07 PM             6,656 dosdev.exe\r\n08\/11\/2008  11:08 PM             1,000 pre-cmd.vbs\r\n08\/11\/2008  11:05 PM                44 pre-exec.cmd\r\n07\/22\/2008  02:26 PM           348,160 rsync.exe\r\n08\/11\/2008  10:12 PM               161 rsyncd.conf\r\n08\/11\/2008  10:12 PM                22 rsyncd.secrets\r\n08\/11\/2008  11:26 PM             1,177 sleep.vbs\r\n06\/08\/2005  03:17 PM           294,912 vshadow.exe\r\n08\/11\/2008  10:09 PM               581 vsrsync.cmd\r\n08\/11\/2008  11:33 PM               308 vss-setvar.cmd<\/pre>\n<p>So, here&#8217;s how it works.\u00a0 Before each backup, BackupPC has an option to call a local script first, waiting for that script to finish.\u00a0 Here&#8217;s the execution chain:<\/p>\n<ol>\n<li>preusercmd.sh launches &#8220;pre-exec.cmd&#8221; on the Windows box<\/li>\n<li>preexec.cmd launches &#8220;pre-cmd.vbs&#8221;<\/li>\n<li>pre-cmd.vbs cleans up some files, launches &#8220;sleep.vbs&#8221; in the background (more on this later) and then launches &#8220;backuppc.cmd&#8221; in the <em>background<\/em>, and waits for the pid file to appear that signals that rsyncd has been launched<\/li>\n<li>backuppc.cmd launches vshadow, and tells it to execute vsrsync.cmd<\/li>\n<li>vsrsync.cmd maps the shadow volume to B:, and launches rsyncd &#8212; it sits and waits here, leaving vshadow and rsync open while the backup or rsync process runs &#8212; on the shadow copy on B:<\/li>\n<\/ol>\n<p>Once the backup is completed, another local script is run &#8212; here&#8217;s its execution chain:<\/p>\n<ol>\n<li>postusercmd.sh puts a file called &#8220;wake.up&#8221; in the C:\\BackupPC directory<\/li>\n<li>sleep.vbs wakes up, sees this file, reads rsyncd.pid, and kills the rsyncd process<\/li>\n<li>vsrsync.cmd now continues, since the rsync process is dead.\u00a0 It unmaps the B: drive.\u00a0 Once this script completes, vshadow automatically deletes the shadow volume.<\/li>\n<\/ol>\n<p>Sure, it seems simple, but a lot of work went into that, since there are a lot of nuances to sort out.\u00a0 Here are the file listings:<\/p>\n<p><strong>preusercmd.sh<\/strong><\/p>\n<pre style=\"padding-left: 30px;\">#!\/bin\/bash\r\nWINEXE=\/usr\/bin\/winexe\r\nUNAME=\"Administrator\"\r\nPWD=\"admin.password\"\r\nWRKGRP=\"WORKGROUP\"\r\nBOX=$1\r\n$WINEXE --interactive=0 -U $UNAME -W $WRKGRP --password=$PWD \/\/$BOX 'cmd \/c c:\\backuppc\\pre-exec.cmd'\r\nsleep 5\r\necho \"Rsync and shadow copy loaded\"\r\nkill $$\r\n# The script needs to be killed, otherwise, winexe waits for input<\/pre>\n<p><strong>pre-exec.cmd<\/strong><\/p>\n<pre style=\"padding-left: 30px;\">cd \\backuppc\r\n@echo off\r\ncscript pre-cmd.vbs<\/pre>\n<p><strong>pre-cmd.vbs<\/strong><\/p>\n<pre style=\"padding-left: 30px;\">Const Flag = \"C:\\BackupPC\\rsyncd.pid\"\r\n'\r\n' Pid file shouldn't be there already\r\n'\r\nIf DoesFileExist(Flag)=0 Then\r\n   Set fso = CreateObject(\"Scripting.FileSystemObject\")\r\n   Set aFile = fso.GetFile(Flag)\r\n   aFile.Delete\r\nEnd If\r\n'\r\n' Nor should \"wake.up\"\r\n'\r\nIf DoesFileExist(\"C:\\BackupPC\\wake.up\")=0 Then\r\n   Set fso = CreateObject(\"Scripting.FileSystemObject\")\r\n   Set aFile = fso.GetFile(\"C:\\BackupPC\\wake.up\")\r\n   aFile.Delete\r\nEnd If\r\n'\r\nSet objShell = CreateObject(\"WScript.Shell\")\r\nobjShell.Exec \"cscript C:\\BackupPC\\sleep.vbs\"\r\n'\r\nSet objShell = CreateObject(\"WScript.Shell\")\r\nobjShell.Exec \"C:\\BackupPC\\backuppc.cmd &gt; C:\\BackupPC\\file.out\"\r\n'\r\n' Just sleep until the file \"rsyncd.pid\" appears\r\n'\r\nWhile DoesFileExist(Flag)\r\n   wscript.sleep 10000\r\nWend\r\n'\r\n' functions\r\n'\r\nfunction DoesFileExist(FilePath)\r\nDim fso\r\n\tSet fso = CreateObject(\"Scripting.FileSystemObject\")\r\n\tif not fso.FileExists(FilePath) then\r\n\t\tDoesFileExist = -1\r\n\telse\r\n\t\tDoesFileExist = 0\r\n\tend if\r\n\tSet fso = Nothing\r\nend function<\/pre>\n<p><strong>sleep.vbs<\/strong><\/p>\n<pre style=\"padding-left: 30px;\">Const Rsync = \"C:\\BackupPC\\rsyncd.pid\"\r\nConst Flag = \"C:\\BackupPC\\wake.up\"\r\n'\r\n' Just sleep until the file \"rsyncd.pid\" appears\r\n'\r\nWhile DoesFileExist(Rsync)\r\n   wscript.sleep 10000\r\nWend\r\n'\r\n' Now sleep until the file \"wake.up\" appears\r\n'\r\nWhile DoesFileExist(Flag)\r\n   wscript.sleep 10000\r\nWend\r\n'\r\nSet fso = CreateObject(\"Scripting.FileSystemObject\")\r\nSet aFile = fso.GetFile(Flag)\r\naFile.Delete\r\n'\r\n' It's time to kill Rsync\r\n'\r\nSet fso = CreateObject(\"Scripting.FileSystemObject\")\r\nSet aReadFile = fso.OpenTextFile(Rsync, 1)\r\nstrContents = aReadFile.ReadLine\r\naReadFile.Close\r\n'\r\nSet objShell = CreateObject(\"WScript.Shell\")\r\nobjShell.Run \"taskkill \/f \/pid \" &amp; strContents, 0, true\r\n'\r\n' Wait for Rsync to let go\r\n'\r\nwscript.sleep 5000\r\n'\r\n' Delete PID file\r\n'\r\nIf DoesFileExist(Rsync)=0 Then\r\n   Set objShell = CreateObject(\"WScript.Shell\")\r\n   objShell.Run \"cmd \/c del C:\\BackupPC\\rsyncd.pid\", 0, true\r\nEnd If\r\n'\r\n' functions\r\n'\r\nfunction DoesFileExist(FilePath)\r\nDim fso\r\n\tSet fso = CreateObject(\"Scripting.FileSystemObject\")\r\n\tif not fso.FileExists(FilePath) then\r\n\t\tDoesFileExist = -1\r\n\telse\r\n\t\tDoesFileExist = 0\r\n\tend if\r\n\tSet fso = Nothing\r\nend function<\/pre>\n<p><strong>backuppc.cmd<\/strong><\/p>\n<pre style=\"padding-left: 30px;\">cd \\backuppc\r\nvshadow -script=vss-setvar.cmd -exec=vsrsync.cmd c:<\/pre>\n<p><strong>vsrsync.cmd<\/strong><\/p>\n<pre style=\"padding-left: 30px;\">REM @ECHO OFF\r\ncall vss-setvar.cmd\r\ncd \\BackupPC\r\nSET CWRSYNCHOME=\\BACKUPPC\r\nSET CYGWIN=nontsec\r\nSET CWOLDPATH=%PATH%\r\nSET PATH=\\BACKUPPC;%PATH%\r\ndosdev B: %SHADOW_DEVICE_1%\r\nREM Go into daemon mode, we'll kill it once we're done\r\nrsync -v -v --daemon --config=rsyncd.conf --no-detach --log-file=diagnostic.txt\r\ndosdev -r -d B:<\/pre>\n<p><strong>rsyncd.conf<\/strong><\/p>\n<pre style=\"padding-left: 30px;\">use chroot = false\r\nstrict modes = false\r\npid file = rsyncd.pid\r\n[C]\r\npath = \/cygdrive\/B\/\r\nauth users = Administrator\r\nsecrets file = rsyncd.secrets<\/pre>\n<p><strong>postusercmd.sh<\/strong><\/p>\n<pre style=\"padding-left: 30px;\">#!\/bin\/bash\r\nWINEXE=\/usr\/bin\/winexe\r\nUNAME=\"Administrator\"\r\nPWD=\"admin.password\"\r\nWRKGRP=\"WORKGROUP\"\r\nBOX=$1\r\nPID=$($WINEXE -U $UNAME -W $WRKGRP --password=$PWD \/\/$BOX 'cmd \/c echo '1' &gt; c:\\backuppc\\wake.up')\r\necho \"Rsync and shadow copy unloaded\"<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Update: Versions of the files below may be downloaded here.\u00a0 This post is probably still useful as documentation. &nbsp; &nbsp; This isn&#8217;t specific to BackupPC by any means, but I&#8217;ll preface this with a brief explanation:\u00a0 BackupPC is a &#8220;set it and forget it&#8221; backup system driven from the server, \u2026 <a class=\"continue-reading-link\" href=\"http:\/\/www.goodjobsucking.com\/?p=62\"> Continue reading <span class=\"meta-nav\">&rarr; <\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[39,36,12,37,38],"_links":{"self":[{"href":"http:\/\/www.goodjobsucking.com\/index.php?rest_route=\/wp\/v2\/posts\/62"}],"collection":[{"href":"http:\/\/www.goodjobsucking.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.goodjobsucking.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.goodjobsucking.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.goodjobsucking.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=62"}],"version-history":[{"count":12,"href":"http:\/\/www.goodjobsucking.com\/index.php?rest_route=\/wp\/v2\/posts\/62\/revisions"}],"predecessor-version":[{"id":71,"href":"http:\/\/www.goodjobsucking.com\/index.php?rest_route=\/wp\/v2\/posts\/62\/revisions\/71"}],"wp:attachment":[{"href":"http:\/\/www.goodjobsucking.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=62"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.goodjobsucking.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=62"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.goodjobsucking.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=62"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}