Archive for May 5th, 2009
Tar over SSH (Pipe Tar)
by z3n on May.05, 2009, under Linux Happyness, Tips & Hints
Problem:
Copy over 10 million files to a different server perserving the permissions, ownership, paths etc.
Solution:
tar -zcf – /path/to/local/dir | ssh remoteuser@remotehost.tld tar -C /path/to/remote/dir -zxf -
this will send a compressed (z parameter) stream to stdout (-) wich will be sent by ssh and uncompressed on the remote server. If you are on a local network you can supress the z parameter in order to keep it faster.
Many people would say that rsync is a better solution, however, rsync don’t work well with many files, specially million files. rsync tends to do a checksum of each file and time/date comparsions which slowdown the process and create lots of disk I/Os, in my case, it was impossible to rsync, the remote server loads were so high that it eventually got stuck and crashed. Tar wasn’t only faster but more effective, making the server able to serve pages while doing it.
Tar will also preserve the ownership and permissions of the files, just what you need to have an exact copy.
Sources:
