------------------------------------------------------------------------------- Remote File Coping Methods for Remotely coping files from one machine to another are many and varied.. The following are some methods I have seen or have succedded in using for varous purposes. Few of the following methods are good or really secure network wise. More specific techniques and information follows. Anthony Thyssen ------------------------------------------------------------------------------- Rcp Command For coping between two account owned by the same person, a ".rhosts" file needs to be setup to allow coping (and logins!!!) without a password from one account to another. rcp file dest_mach:file This is is not a good method as it is open to abuse. EG: 1/ a craker on one account can go to the outher without problems 2/ hostname spoofing will allow breakins 3/ the file trasfer could be listened to. --------------------- RDist If you can use rcp you can also use rdist to only copy files or whole directories of files which have been modified since the last time it was transfered. I use this for keeping my 30 odd unix accounts up to date. Also see http://www.cit.gu.edu.au/~anthony/info/misc/rdist.hints --------------------- Ftp this is what FTP was designed for originally. However you need to know the password of the remote account you are transfering to/from. This is fine if you own the account, but if it isn't yours you have a problem. Also as a password is required it can NOT be shell scripted without comprimising the password (see next). Also FTP does NOT preserve permissions/ownership/dates (in general) or transfer whole directories recursivally. For that it is best to create a TAR archive of the files to transfer, and FTP that. This will also preserve the file information FTP fails to do. --------------------- Ftp via anonymous ftp server Put the file on some anonymous ftp server and create ftp script to upload/download the file. You can automatically download files with ftp -n -i << END_FTP open FTP_SERVER_WITH_FILE user anonymous YOUR_EMAIL_ADDRESS get /path/to/some/file local_file bye END_FTP You could also password protect a special ftp access area for this use only. The passowrd would only be used for this purpose. --------------------- Scp (Secure Shell (Ssh) Copy Command) This is like the rcp command but is secure, the channel is encrypted from evedroppers, and host/account spoofing isn't posible. However if the private key is encrypted (recomended) then a ssh-agent will have to be setup to hold the unencrypted password for the transfer, or your public key access password will have to be entered. scp file dest_mach:file Secure shell ssh will also provide encrypted X windows channel accross machines automatically (set DISPLAY and Xauth stuff on remote machine). Environment variables set is also usful to keep track of the inital start machine of ssh links and the exect command being asked to be performed on the remote machine (before the shell meta-characters have been performed). --------------------- Rsync A bit like rdist but faster due to the ability to perform incremental updates. That is it can update log files with just the new data, without needing to transfer the whole file. Unlike rdist you can specify the command to use to connect to the remote machine. This program is improving in capabilities and performance all the time. (see ssh-rdist security below) --------------------- SDist -- The GNU Rdist A separate program to the Ssh stuff which will do rdists over Ssh instead of a rsh like channel. (I have not tryed this yet) This has been merged into the Gnu Rdist command, by allowing it to specify the remote connection command). NOTE This is different and incompatable with the rdist on most commercial UNIX operating systems. --------------------- Mail To accounts owned by different users or where .rhost and ssh connections can't be performed, you could send the file via mail. UUencode will encode binary and text files to allow easy and exact decode at the other end.. EG: on client machine uuencode filename filename | sed 's/^$/`/' |\ /usr/ucb/mail -s "MAIL of file filename" user@dest.machine on recieving machine a special mail filter can be setup or "procmail" can be used to send mail with a specific subject though a decode script. cd /path/to/save/dir uudecode < mail or uudecode -p save_filename See the man pages on uuencode uudecode and procmail This system unless protected somehow (EG: PGP mail encryption ) could also be abused by anyone sending mail of the appropraite configuration to your filter. NOTE: I have more info on my trials of this in ../sendmail/hints.mail also code I developed for retrival of reports, from my research assistant period, using parellel procing on hundreds of machines around the world. --- Anthony --------------------- Interactive Shell Connection Use a interactive rsh or telnet to do things on the remote machine See the software expect if installed, also the file. "../shell/interactive.hints". NOTE: the program "mconnect" on Sun machines is equivelent to telnet but is non-interactive, allowing you to pipe data in and out normally. --------------------- Roll your own Daemon Create your own network service to transfer information and files between the hosts. This is very DANGEROUS unless you know what you are doing, especially with regard to security. =============================================================================== Coping lots of small files.. Peter Chun wrote: I have a machine that contains  a heap of small files. number of files (inodes) = 200,000 (50,000 directories and 150000 files) file size = 10kB , 3kB and 30B in 1 directory. total size = 6 GB reside on 10000rpm SCSI in RAID 1 format ( mirroring across 2 disks ) machines have 100Mbps NIC on 1Gb switch both Solaris 7 Results... backup tapes, 1hr to tape, 3.5 hr from tape straight rcp from host1 to host2, > 3 hours rsync (only copy changes) - too long > 3 hrs ufsdump - ufsrestore 90 mintutes NB: changing block size of 126 only serves to increase the time taken tar - untar 120 minutes A few things: - If you're using vxvm, is vxfs an option? It should be much better than ufs, even with logging - Ufs with logging has slow write throughput - Normal ufs is slow creating lots of small files & directories; each is a synchronous operation - tar/cpio, et al will not backup ACLs - ufsdump | ufsrestore will backup ACLs - There is a hack available that makes a ufs filesystem async on solaris, which is good when extracting files from a backup, but not for production!. I've attached it below. Dunno if it works on Solaris 8 though. `dd' is ok if you're happy to accept that the restored file system will have all the same characteristics as the previous. It's a good quick way to backup a direct file system image. Depending on CPU speed vs disk speed, I find that something like: dd if=/dev/rXXXX bs=32k | gzip -1 > /image/foo.gz [ do your stuff ] gunzip -c /image/foo.gz | dd bs=32k of=/dev/rXXXX Here's a program once posted by someone whose name I can't find back. I since made it compile on Solaris 2.x as well. Requires SunOS 4.1.2 or later. /* * This programs turns on/off delayed I/O on a filesystem. * * Usage: fastfs filesystem fast|slow|status * * Note that it is intended for use with restore(8) * to speed up full filesystem restores. Remember * that if a filesystem is running with delayed I/O * enabled when the system crashes it can result in * fsck being unable to "fix" the filesystem on reboot * without manual intervention. * * Typical use is * * fastfs /home fast * cd /home; restore rf /dev/rst5 * fastfs /home slow * * The above gives about a 500% increase in the speed of * the restore (your milage may vary). * * Its also good for /tmp giving most of the benifits of tmpfs * without the problems. * * In rc.local * * fastfs /tmp fast * * but you may need to add fsck -y /tmp into /etc/rc.boot * before the real fsck to ensure the machine always boots * (I have not done this yet). */ #include #include #include #include #include #ifndef FIODIO #define FIODIO _FIOSDIO #define FIODIOS _FIOGDIO #endif extern char *sys_errlist[]; int main(argc, argv) int argc; char **argv; { int fd; int flag; if (geteuid() != 0) { fprintf(stderr,"%s: Must run as root.\n", argv[0]); exit(1); } if (argc != 3) { fprintf(stderr,"Usage: %s path {fast|slow|status}\n",argv[0]); exit(2); } if ((fd = open(argv[1], O_RDONLY)) < 0) { fprintf(stderr,"%s: Unable to open \"%s\"\n%s\n", argv[0], argv[1], sys_errlist[errno]); exit(3); } if (!strcasecmp(argv[2], "status")) { if (ioctl(fd, FIODIOS, &flag) != 0) { fprintf(stderr,"%s: ioctl failed%s\n", argv[0], sys_errlist[errno]); exit(4); } if (flag == 0) printf("%s is slow.\n", argv[1]); else printf("%s is fast\n", argv[1]); exit(0); } flag = (strcasecmp(argv[2], "fast") == 0) ? 1 : 0; if (ioctl(fd, FIODIO, &flag) != 0) { fprintf(stderr,"%s: ioctl failed\n%s\n", argv[0], sys_errlist[errno]); exit(5); } printf("%s is now %s\n",argv[1], flag ? "fast" : "slow"); exit(0); } ===============================================================================