Jay Taylor's notes

back to listing index

rsync exclude according to .gitignore & .hgignore & svn:ignore like --filter=:C

[web search]
Original source (stackoverflow.com)
Tags: git rsync gitignore stackoverflow.com
Clipped on: 2016-11-01

Rsync includes a nifty option --cvs-exclude to “ignore files in the same way CVS does”, but CVS has been obsolete for years. Is there any way to make it also exclude files which would be ignored by modern version control systems (Git, Mercurial, Subversion)?

For example, I have lots of Maven projects checked out from GitHub. Typically they include a .gitignore listing at least target, the default Maven build directory (which may be present at top level or in submodules). Since the contents of these directories are entirely disposable, and they can be far larger than source code, I would like to exclude them when using rsync for backups.

Of course I can explicitly --exclude=target/ but that will accidentally suppress unrelated directories that just happen to be named target and are not supposed to be ignored.

And I could supply a complete list of absolute paths for all file names and patterns mentioned in any .gitignore, .hgignore, or svn:ignore property on my disk, but this would be a huge list that would have to be produced by some sort of script.

Since rsync has no built-in support for VCS checkouts other than CVS, is there any good trick for feeding it their ignore patterns? Or some kind of callback system whereby a user script can be asked whether a given file/directory should be included or not?

Update: --filter=':- .gitignore' as suggested by LordJavac seems to work as well for Git as --filter=:C does for CVS, at least on the examples I have found, though it is unclear if the syntax is an exact match. --filter=':- .hgignore' does not work very well for Mercurial; e.g. an .hgignore containing a line like ^target$ (the Mercurial equivalent of Git /target/) is not recognized by rsync as a regular expression. And nothing seems to work for Subversion, for which you would have to parse .svn/dir-prop-base for a 1.6 or earlier working copy, and throw up your hands in dismay for a 1.7 or later working copy.

asked Dec 4 '12 at 22:30
Image (Asset 2/23) alt=
Jesse Glick
9,83222952
9 upvote
  flag
Sounds a bit like it would be a good idea to submit a patch for rsync that adds support for .gitignore, .hgignore, etc. – ThiefMaster♦ Dec 9 '12 at 11:48
3 upvote
  flag
@ThiefMaster: I filed bugzilla.samba.org/show_bug.cgi?id=9744 as a starting point. – Jesse Glick Mar 27 '13 at 14:30
2 upvote
  flag
just a note for others, the .gitignore needs to be in the folder hierarchy being rysnc'd, not in the directory the command is being executed – myol Nov 20 '14 at 12:40

As mentioned by luksan, you can do this with the --filter switch to rsync. I achieved this with --filter=':- .gitignore' (there's a space before ".gitignore") which tells rsync to do a directory merge with .gitignore files and have them exclude per git's rules. You may also want to add your global ignore file, if you have one. To make it easier to use, I created an alias to rsync which included the filter.

answered Mar 12 '13 at 22:48
Image (Asset 4/23) alt=
LordJavac
56953
   upvote
  flag
A good start, though I hesitate to “accept” this answer as it only covers Git. – Jesse Glick Mar 27 '13 at 14:31
7 upvote
  flag
A more verbose version which also excludes .git files: --exclude='/.git' --filter="dir-merge,- .gitignore" – VasyaNovikov Dec 22 '14 at 22:29
1 upvote
  flag
I have something like this now: rsync -rvv --exclude='.git*' --exclude='/rsync-to-dev.sh' --filter='dir-merge,-n /.gitignore' $DIR/ development.foobar.com:~/test/ .. but although it says [sender] hiding file .gitignore because of pattern .git*, the file still is sent to the desintation – rolandow Jun 30 '15 at 7:53

You can use git ls-files to build the list of files excluded by the repository's .gitignore files. https://git-scm.com/docs/git-ls-files

Options:

  • --exclude-standard Consider all .gitignore files.
  • -o Don't ignore unstaged changes.
  • -i Only output ignored files.
  • --directory Only output the directory path if the entire directory is ignored.

The only thing I left to ignore was .git.

rsync -azP --exclude=.git --exclude=`git -C <SRC> ls-files --exclude-standard -oi --directory` <SRC> <DEST>
answered Mar 10 at 21:21
Image (Asset 5/23) alt=

I had a number of very large .gitignore files and none of the "pure rsync" solutions worked for me. I wrote this rsync wrapper script, it fully respects .gitignore rules (include !-style exceptions and .gitignore files in subdirectories) and has worked like a charm for me.

answered Sep 23 '15 at 22:01
Image (Asset 6/23) alt=
cobbzilla
931711
   upvote
  flag
Trying this via locate -0e .gitignore | (while read -d '' x; do process_git_ignore "$x"; done), but has a lot of issues. Files in the same directory as .gitignore not correctly separated from the directory name with /. Blank lines and comments misinterpreted. Chokes on .gitignore files in paths with spaces (never mind the fiendish /opt/vagrant/embedded/gems/gems/rb-fsevent-0.9.4/spec/fixtur‌​es/custom 'path/.gitignore from the vagrant package for Ubuntu). Perhaps better done as a Perl script. – Jesse Glick Sep 24 '15 at 13:20
   upvote
  flag
@JesseGlick I'm not sure why you're calling the function within the script. it's intended to be used as a drop-in replacement for rsync, for the specific reason that handling quoting/whitespace is such a pain. If you have an example of a gsync command line that is failing, and the .gitignore files associated with it, I would be happy to take a closer look. – cobbzilla Sep 25 '15 at 22:05
   upvote
  flag
I need to rsync an entire filesystem, with various Git repositories scattered around it. Perhaps your script works fine for the case of synchronizing a single repository. – Jesse Glick Sep 27 '15 at 15:30
1 upvote
  flag
yes, definitely. sorry I did not make that clear. With this script, you'd have to invoke it once per git repo, from within the repo directory. – cobbzilla Sep 27 '15 at 21:37

how about rsync --exclude-from='path/.gitignore' --exclude-from='path/myignore.txt' source destination?
It worked for me.
I believe you can have more --exclude-from parameters too.

answered Sep 30 '15 at 6:09
Image (Asset 7/23) alt=
eric
1,70032353
1 upvote
  flag
This will work insofar as your .gitignore files happen to use a syntax compatible with rsync. – Jesse Glick Sep 30 '15 at 21:50

Check out the MERGE-FILES FILTER RULES section in rsync(1).

It looks like it's possible to create a rsync --filter rule that will include .gitignore files as traverses the directory structure.

answered Dec 9 '12 at 11:46
Image (Asset 8/23) alt=
luksan
361

Per the rsync man page, in addition to the standard list of file patterns:

files listed in a $HOME/.cvsignore are added to the list and any files listed in the CVSIGNORE environment variable

So, my $HOME/.cvsignore file looks like this:

.git/
.sass-cache/

to exclude .git and the files generated by Sass.

answered Jul 15 '13 at 13:49
Image (Asset 9/23) alt=
Doug Harris
1,00321126
2 upvote
  flag
To the contrary, I definitely want to include .git/ directories, perhaps even more strongly than the working copy. What I want to exclude are build products. – Jesse Glick Jul 26 '13 at 14:01
   upvote
  flag
Also, this setting is not portable. It's per-user, not per-project. – VasyaNovikov Dec 22 '14 at 21:37
   upvote
  flag
@JesseGlick I second you about keeping .git/ dirs included. Git being a distributed SCM, it's important to backup the whole local repository. – Johan Boule Jul 23 '15 at 14:51

Your Answer

asked

3 years ago

viewed

8708 times

active

7 months ago
Blog
Featured on Meta

Hot Network Questions

Technology Life / Arts Culture / Recreation Science Other
  1. Stack Overflow
  2. Server Fault
  3. Super User
  4. Web Applications
  5. Ask Ubuntu
  6. Webmasters
  7. Game Development
  8. TeX - LaTeX
  1. Software Engineering
  2. Unix & Linux
  3. Ask Different (Apple)
  4. WordPress Development
  5. Geographic Information Systems
  6. Electrical Engineering
  7. Android Enthusiasts
  8. Information Security
  1. Database Administrators
  2. Drupal Answers
  3. SharePoint
  4. User Experience
  5. Mathematica
  6. Salesforce
  7. ExpressionEngine® Answers
  8. Cryptography
  1. Code Review
  2. Magento
  3. Signal Processing
  4. Raspberry Pi
  5. Programming Puzzles & Code Golf
  6. more (7)
  1. Photography
  2. Science Fiction & Fantasy
  3. Graphic Design
  4. Movies & TV
  5. Music: Practice & Theory
  6. Seasoned Advice (cooking)
  7. Home Improvement
  8. Personal Finance & Money
  1. Academia
  2. more (8)
  1. English Language & Usage
  2. Skeptics
  3. Mi Yodeya (Judaism)
  4. Travel
  5. Christianity
  6. English Language Learners
  7. Japanese Language
  8. Arqade (gaming)
  1. Bicycles
  2. Role-playing Games
  3. Anime & Manga
  4. Motor Vehicle Maintenance & Repair
  5. more (17)
  1. MathOverflow
  2. Mathematics
  3. Cross Validated (stats)
  4. Theoretical Computer Science
  5. Physics
  6. Chemistry
  7. Biology
  8. Computer Science
  1. Philosophy
  2. more (3)
  1. Meta Stack Exchange
  2. Stack Apps
  3. Area 51
  4. Stack Overflow Talent
site design / logo © 2016 Stack Exchange Inc; user contributions licensed under cc by-sa 3.0 with attribution required
rev 2016.11.1.4163