Home Synchronization and Transfer Scheme This is based in Wirth's Rule... Never store data in more than one place, sooner or later you will update one and not the other. And my own corollary Only store and update data in one place, but have all copies generate from that one primary source. Note this scheme involves the transfer of the 'one place' from one computer to another. As such it can still suffer the problems of Wirth's rule, if I fail to do the transfer/update before modifing files. Caution is required. UPDATE: This scheme was later superseeded by the "Unison" command for my Home directory. http://www.cis.upenn.edu/~bcpierce/unison/ However I still use a rsync transfer scheme for mail transfers due to the the same reasons as given above. ------------------------------------------------------------------------------- Rsync Transfer Scheme (Home) Transfering file updates between Work Computer and my Home Laptop It is based on the program "rsync", as it will only copy/delete the files that need to be modified. It was designed specifically to minimise network/modem transfers for file coping. It can figure this out by file size/date or in a more intensive checksum method to discover if a file has changed. It even handle partial transfers (if wanted) and log files that only has appended data. Transfer Scheme.... 1/ One of the machines is always regarded by me as my current primary home. 2/ I only modify or change my primary home. Though I can freely use and read from the secondary home, I never modify it as any modifications will be lost when... 3/ To swap my primary and secondary homes I synchronize them by transfering all changes I have made to my primary to the other machine. This includes file deletions (--delete) and date changes (--archive). 4/ After syncronization I can then pick from either of the two now syncronized homes as the new primary, generally where whereever I will now be working. Whatever I pick is now primary. NOTE: That I don't always synchronize my whole home. I sometimes only synchronize a single subdirectory (such as mail). But I still keep track of what is the 'primary site'. Some directories may only be primary on one machine with only mirror copys transfered to the other machine(s). That is because I only work on these directories only that specific machine, with the other machine as backup. Also note that only the differences are syncronised thanks to rsync. Only time big changes happen is usally when I rename a large sub-directory or something like that. This means it generally works well over a slow link such as a modem. Personal situation note... My own home transfer scheme has over time become complex. Some directories syncronise in both directions, others only mirror in one direction. And others transfer hapazardally as needed. I keep track of it with a list, of the directories and files and the directions in which they can syncronise. I then set up aliases (only used from home)... dl to download primary to my laptop and ul to upload primary to my work machine I also setup a few other aliases... dlm incorperate new mail then download new mail (checksum style) to laptop ulm upload the mail read/delete changes back to to my work machine With these aliases I can download my mail for reading, then upload those changes, before downloading again to retreive any new mail. The next section explains this method further... ---- Mail Transfer Scheme -- a practical example of the above One directory is also on its own separate transfer scheme of the same time. My mail sub-directory (using the MH system and read with EXMH). I chose the mh system instead of POP or some other method as after `incorperation' of new mail, which is a specific action, ALL my mail is stored in one special directory in my home area, and not elsewhere on the system. I use a simular system to the above `home transfer' except it can work independant to which is my primary home. EG I could transfer my primary mail directory to my laptop independant of wether my primary home is at work or on my laptop. Thus using the above scheme form mail... the various actions I do is as follows.. Transfer mail to laptop at home to read mail shell alias dlm or ``download mail'' (start of evening, weekend or vacation) ( cd; # go home in a sub-shell # incorperate new mail into mail directory, # and purge deleted mail (for fast transfer) rsh {work_host} 'inc; rm -f mail/*/,*;' # transfer mail to linux laptop rsync -azRcv --delete --rsh ssh {work_host}:mail . ) Transfer my mail reading and deletions from laptop back to my work host shell alias ulm or ``upload mail'' ( cd; # go home in a sub-shell rm -f mail/*/,* # purge deleted mail (for fast transfer) rsync -azRcv --delete --rsh ssh mail {work_host}: ) Note that I use a checksum on these files as the filename have a tendency to change and be re-used for different mail messages. As such date checking is problematical and error prone. During the evening, a simple xbiff flag indicates new mail on my work machines system mailbox, I transfer it to my laptop for reading and replying. To do this I run ulm to "upload mail changes" then I run dlm to incorperate the new mail and "download the new mail" again. This is fast as only the mail directory (and not my whole home), and only my mail directory changes to transfered each time. Also I can reply to mail when 'offline' or still read and reply to mail from the 'secondary', as long as I don't modify the secondary (as the modification will be lost). More importantly the same mail is on BOTH machines and at no time do end up with mail on one machine with out it easilly and automatically transfering to the other as I swap the primary and secondary machines. ---- This transfer method is reasonably simple. and could be used in a LOT of situations. A flag could also be setup to indicate which machine is currently the primary, and programs could transfer primary, automatically if required. Sort of a 'token ring' update scheme. The important point is writes are only performed into the current primary copy of the file system, which can be synchronized and transfered to other machines. Anthony Thyssen ( System Programmer ) ------------------------------------------------------------------------------- From: Andrew Tridgell Date: Wed, 30 Sep 1998 13:59:59 +1000 > This transfer method is reasonably simple. Simple? well ... I actually use a somewhat similar scheme of transfers for various things between home and work (source trees, mail etc) but I generally don't recommend it to people as it is so fragile. One error or mis-directed change and you end up losing data. I use it, but I don't mind living a bit dangerously. > The important point is writes are only performed by the current primary > copy of the file system. -- Appling Wirth's Rule Yes! If you wanted to make this scheme robust then you could do that by putting the data in a separate partition and remounting it read-only when it isn't the primary. Or maybe even use chattr -R on the tree and set the immutable flag on the files you don't want touched. I don't do that for myself, but if I was setting up something like this for end users then I'd want a safety net. Cheers, Tridge ------------------------------------------------------------------------------- From: "Manuel M. T. Chakravarty" Date: Wed, 30 Sep 1998 17:00:00 +1000 I use absolutely identical home directories on my desktop and my laptop (except .forward, .rhosts, a critical file written by my window manager, and netscape's disk cache). When I leave work, I sync to my laptop, and when I come to work, I sync to my desktop. To make this reasonable convenient, I wrote a small shell script to do the synchronization. It uses a rather verbatim set of options, which makes it difficult to cause harvoc by a typo. Furthermore, the script doesn't --delete by default, only when explicitly asked to. On most days, I actually don't have to delete anything and this way it become really difficult to screw up. This scheme works quite well for both me and my wife. Manuel -------------------------------------------------------------------------------