Copying directory trees with rsync

You can use cp -a to copy directory trees, but rsync can do the same and give you more flexibility. rsync supports a syntax for filter rules which specify which files and directories should and should not be copied.

Examples

Copy only the directory structure without copying any files:

$ rsync -a -f"+ */" -f"- *" source/ destination/

The two -f arguments mean, respectively, "copy all directories" and then "do not copy anything else".

Copy only directories and Python files:

$ rsync -a -f"+ */" -f"+ *.py" -f"- *" source/ destination/

This is really handy for replicating the general directory structure but only copying a subset of the files.

Copy everything but exclude .git directories:

$ rsync -a -f"- .git/" -f"+ *" source/ destination/

Conclusion

Of course, rsync also works great for copying files between machines, and it knows better than to transfer files that already exist on the destination. I use something similar to the above to do backups, copying my homedir but excluding stuff like caches that are not even worth copying.

19 comments:

  1. Very good examples. Thank you.

    ReplyDelete
  2. First, thanks :)
    Second, how can I have it copy all directories under /var/www but exclude all files under /var/www ; recursively...

    ReplyDelete
  3. PERFECT. Been looking this for a while.

    ReplyDelete
  4. Thanks for the tip! Just used this today.

    ReplyDelete
  5. Shai, you replace "source/" and "destination/" with:

    /var/www/ /some/directory/

    /var/www/ somemachine:/some/directory/

    ReplyDelete
  6. thnx for tip very helpful

    ReplyDelete
  7. Excellent. Rsyncing the directory structure only was exactly what I needed!

    ReplyDelete
  8. Thanks for the tip Phil!

    For those who have an older version of rsync (one that doesn't support -f), you can do the same thing with --include and --exclude.

    So Phil's example becomes:

    $ rsync -a --include=*/ --exclude=* source/ destination/

    -klam

    ReplyDelete
  9. I have a question. I tried to use the syntax above under rsync 3.0.9 with the following results (source and dest names changed to protect the not-so-innocent):

    rsync -v -f"+ */" -f"- *" /source desthost:/dest

    skipping directory source


    rsync -v -f"+ */" -f"- *" /source/ desthost:/dest/

    skipping directory .


    Am I missing something, or did the behavior change in 3.0.9?

    ReplyDelete
  10. Found the trick. I had to use -r to recurse the tree.

    rsync -v -a -r -f"+ */" -f"- *"

    ReplyDelete
  11. -v is verbose output
    -r will recurse through the directories
    -a is "archive" mode which includes -r
    FROM "rsync --help":
    -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)

    So rsync -rv would have worked as well.

    ReplyDelete
  12. Very good examples, thanks. However, I cannot figure out how to include not only *.py files, but, for example *.py and *.log files. Of course, one can do it with two subsequent commands but still I wonder how to do it in one.

    ReplyDelete
  13. helped me to copy my personal structure to /etc/skel/ _

    # $1 source/
    # $2 destination/
    rsync -av -f"+ */" -f"- *" $1 $2

    ReplyDelete
  14. Very slow method in case of milions of files as rsync stat()s every file in source tree (instead of just reading directory entries) :/

    ReplyDelete
  15. Another possible solution (whithout rsync):
    (cd /home/user/source/; find -type d -print0) | xargs -0 mkdir -p

    ReplyDelete
  16. Im using Rsync inside the gui of OMV(mediavault) and even if the command has a special tab for extra commands i cant manage to copy the folder also instead of just the inside files. The way it works inside OmV is you make a share from the inside nas hdd and then plug in an external hdd (the source of files) and mount it and make a share with one of its folders.
    Problem is that rsync creates to the destination folder the files only of the source folder without the external folder which includes the original files. I can't mange this to work no matter what I try. Any help would eb appreciated

    ReplyDelete