How to extract a git subdirectory and make a submodule out of it?

I started a project some months ago and stored everything within a main directory. In my main directory "Project" there are several subdirectories containing different things: Project/paper contains a document written in LaTeX Project/sourcecode/RailsApp contains my rails app.

"Project" is GITified and there have been a lot of commits in both "paper" and "RailsApp" directory. Now, as I'd like to use cruisecontrol.rb for my "RailsApp" I wonder if there is a way to make a submodule out of "RailsApp" without losing the history.

Nowadays there's a much easier way to do it than manually using git filter-branch: git subtree

Installation

NOTE git-subtree is now part of git (if you install contrib) as of 1.7.11, so you might already have it installed. You may check by executing git subtree.

---

To install git-subtree from source (for older versions of git):

git clone https://github.com/apenwarr/git-subtree.git

cd git-subtree
sudo rsync -a ./git-subtree.sh /usr/local/bin/git-subtree

Or if you want the man pages and all

make doc
make install

Usage

Split a larger into smaller chunks:

# Go into the project root
cd ~/my-project

# Create a branch which only contains commits for the children of 'foo'
git subtree split --prefix=foo --branch=foo-only

# Remove 'foo' from the project
git rm -rf ./foo

# Create a git repo for 'foo' (assuming we already created it on github)
mkdir foo
pushd foo
git init
git remote add origin [email protected]:my-user/new-project.git
git pull ../ foo-only
git push origin -u master
popd

# Add 'foo' as a git submodule to `my-project`
git submodule add [email protected]:my-user/new-project.git foo

For detailed documentation (man page), please read git-subtree.txt.

Checkout git filter-branch.

The Examples section of the man page shows how to extract a sub-directory into it's own project while keeping all of it's history and discarding history of other files/directories (just what you're looking for).

> To rewrite the repository to look as if foodir/ had been its project root, and discard all other history: > > git filter-branch --subdirectory-filter foodir -- --all > > Thus you can, e.g., turn a library subdirectory into a repository of its own.
Note the -- that separates filter-branch options from revision options, and the --all to rewrite all branches and tags.

One way of doing this is the inverse - remove everything but the file you want to keep.

Basically, make a copy of the repository, then use git filter-branch to remove everything but the file/folders you want to keep.

For example, I have a project from which I wish to extract the file tvnamer.py to a new repository:

git filter-branch --tree-filter 'for f in *; do if [ $f != "tvnamer.py" ]; then rm -rf $f; fi; done' HEAD

That uses git filter-branch --tree-filter to go through each commit, run the command and recommit the resulting directories content. This is extremely destructive (so you should only do this on a copy of your repository!), and can take a while (about 1 minute on a repository with 300 commits and about 20 files)

The above command just runs the following shell-script on each revision, which you'd have to modify of course (to make it exclude your sub-directory instead of tvnamer.py):

for f in *; do
    if [ $f != "tvnamer.py" ]; then
        rm -rf $f;
    fi;
done

The biggest obvious problem is it leaves all commit messages, even if they are unrelated to the remaining file. The script git-remove-empty-commits, fixes this..

git filter-branch --commit-filter 'if [ z$1 = z`git rev-parse $3^{tree}` ]; then skip_commit "$@"; else git commit-tree "$@"; fi'

You need to use the -f force argument run filter-branch again with anything in refs/original/ (which basically a backup)

Of course this will never be perfect, for example if your commit messages mention other files, but it's about as close a git current allows (as far as I'm aware anyway).

Again, only ever run this on a copy of your repository! - but in summary, to remove all files but "thisismyfilename.txt":

git filter-branch --tree-filter 'for f in *; do if [ $f != "thisismyfilename.txt" ]; then rm -rf $f; fi; done' HEAD
git filter-branch -f --commit-filter 'if [ z$1 = z`git rev-parse $3^{tree}` ]; then skip_commit "$@"; else git commit-tree "$@"; fi'

Both CoolAJ86 and apenwarr answers are very similar. I went back and forth between the two trying to understand bits that were missing from either one. Below is a combination of them.

First navigate Git Bash to the root of the git repo to be split. In my example here that is ~/Documents/OriginalRepo (master)

# move the folder at prefix to a new branch
git subtree split --prefix=SubFolderName/FolderToBeNewRepo --branch=to-be-new-repo

# create a new repository out of the newly made branch
mkdir ~/Documents/NewRepo
pushd ~/Documents/NewRepo
git init
git pull ~/Documents/OriginalRepo to-be-new-repo

# upload the new repository to a place that should be referenced for submodules
git remote add origin [email protected]:myUsername/newRepo.git
git push -u origin master
popd

# replace the folder with a submodule
git rm -rf ./SubFolderName/FolderToBeNewRepo
git submodule add [email protected]:myUsername/newRepo.git SubFolderName/FolderToBeNewRepo
git branch --delete --force to-be-new-repo

Below is a copy of above with the customize-able names replaced and using https instead. Root folder is now ~/Documents/_Shawn/UnityProjects/SoProject (master)

# move the folder at prefix to a new branch
git subtree split --prefix=Assets/SoArchitecture --branch=so-package

# create a new repository out of the newly made branch
mkdir ~/Documents/_Shawn/UnityProjects/SoArchitecture
pushd ~/Documents/_Shawn/UnityProjects/SoArchitecture
git init
git pull ~/Documents/_Shawn/UnityProjects/SoProject so-package

# upload the new repository to a place that should be referenced for submodules
git remote add origin https://github.com/Feddas/SoArchitecture.git
git push -u origin master
popd

# replace the folder with a submodule
git rm -rf ./Assets/SoArchitecture
git submodule add https://github.com/Feddas/SoArchitecture.git
git branch --delete --force so-package

If you want to transfer some subset of files to a new repository but keep the history, you're basically going to end up with a completely new history. The way this would work is basically as follows:

1. Create new repository.
2. For each revision of your old repository, merge the changes to your module into the new repository. This will create a "copy" of your existing project history.

It should be somewhat straightforward to automate this if you don't mind writing a small but hairy script. Straightforward, yes, but also painful. People have done history rewriting in Git in the past, you can do a search for that.

Alternatively: clone the repository, and delete the paper in the clone, delete the app in the original. This would take one minute, it's guaranteed to work, and you can get back to more important things than trying to purify your git history. And don't worry about the hard drive space taken up by redundant copies of history.