Restore git submodules from .gitmodules

GitGit Submodules

Git Problem Overview

I have a folder, which was a git repo. It contains some files and .gitmodules file. Now, when I do git init and then git submodule init, the latter command output is nothing. How can I help git to see submodules, defined in .gitmodules file without running git submodule add by hand again?

Update: this is my .gitmodules file:

[submodule "vim-pathogen"]
	path = vim-pathogen
	url = git://github.com/tpope/vim-pathogen.git
[submodule "bundle/python-mode"]
	path = bundle/python-mode
	url = git://github.com/klen/python-mode.git
[submodule "bundle/vim-fugitive"]
	path = bundle/vim-fugitive
	url = git://github.com/tpope/vim-fugitive.git
[submodule "bundle/ctrlp.vim"]
	path = bundle/ctrlp.vim
	url = git://github.com/kien/ctrlp.vim.git
[submodule "bundle/vim-tomorrow-theme"]
	path = bundle/vim-tomorrow-theme
	url = git://github.com/chriskempson/vim-tomorrow-theme.git

and here is listing of this dir:

drwxr-xr-x  4 evgeniuz 100 4096 июня  29 12:06 .
drwx------ 60 evgeniuz 100 4096 июня  29 11:43 ..
drwxr-xr-x  2 evgeniuz 100 4096 июня  29 10:03 autoload
drwxr-xr-x  7 evgeniuz 100 4096 июня  29 12:13 .git
-rw-r--r--  1 evgeniuz 100  542 июня  29 11:45 .gitmodules
-rw-r--r--  1 evgeniuz 100  243 июня  29 11:18 .vimrc

so, definitely, it is in top level. the git directory is not changed, only git init is done

Git Solutions

Solution 1 - Git

git submodule init only considers submodules that already are in the index (i.e. "staged") for initialization. I would write a short script that parses .gitmodules, and for each url and path pair runs:

git submodule add <url> <path>

For example, you could use the following script:

#!/bin/sh

set -e

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        url_key=$(echo $path_key | sed 's/\.path/.url/')
        url=$(git config -f .gitmodules --get "$url_key")
        git submodule add $url $path
    done

This is based on how the git-submodule.sh script itself parses the .gitmodules file.

Solution 2 - Git

Expanding on @Mark Longair's answer, I wrote a bash script to automate steps 2 & 3 of the following process:

  1. Clone a 'boilerplate' repo to begin a new project
  2. Remove the .git folder and re-initialize as a new repo
  3. Re-initialize the submodules, prompting for input before deleting folders
#!/bin/bash

set -e
rm -rf .git
git init

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' > tempfile

while read -u 3 path_key path
do
    url_key=$(echo $path_key | sed 's/\.path/.url/')
    url=$(git config -f .gitmodules --get "$url_key")

    read -p "Are you sure you want to delete $path and re-initialize as a new submodule? " yn
    case $yn in
        [Yy]* ) rm -rf $path; git submodule add $url $path; echo "$path has been initialized";;
        [Nn]* ) exit;;
        * ) echo "Please answer yes or no.";;
    esac
    
done 3<tempfile

rm tempfile

Note: the submodules will be checked out at the tip of their master branch instead of the same commit as the boilerplate repo, so you'll need to do that manually.

Piping the output from git config into the read loop was causing problems with the prompt for input, so it outputs it to a temp file instead. Any improvements on my first bash script would be very welcome :)

Big thanks to Mark, https://stackoverflow.com/a/226724/193494, https://stackoverflow.com/q/11704353/193494, and tnettenba @ chat.freenode.net for helping me arrive at this solution!

Solution 3 - Git

Extending excellent @Mark Longair's answer to add submodule respecting branch and repo name.

#!/bin/sh

set -e

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        name=$(echo $path_key | sed 's/\submodule\.\(.*\)\.path/\1/')
        url_key=$(echo $path_key | sed 's/\.path/.url/')
        branch_key=$(echo $path_key | sed 's/\.path/.branch/')
        url=$(git config -f .gitmodules --get "$url_key")
        branch=$(git config -f .gitmodules --get "$branch_key" || echo "master")
        git submodule add -b $branch --name $name $url $path || continue
    done

Solution 4 - Git

I had a similar issue. git submodule init was failing silently.

When I did:

git submodule add <url> <path>

I got:

The following path is ignored by one of your .gitignore files: ...

I'm thinking that .gitignore(d) paths might be the cause.

Solution 5 - Git

An updated version of the script by @mark-longair . This one also supports branches, handles the case where some of the submodules already exist in .git/config, and when necessary backs up existing directories of the same name as the submodule paths.

git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
    while read path_key path
    do
        url_key=$(echo $path_key | sed 's/\.path/.url/');
        branch_key=$(echo $path_key | sed 's/\.path/.branch/');
        # If the url_key doesn't yet exist then backup up the existing
        # directory if necessary and add the submodule
        if [ ! $(git config --get "$url_key") ]; then
            if [ -d "$path" ] && [ ! $(git config --get "$url_key") ]; then
                mv "$path" "$path""_backup_""$(date +'%Y%m%d%H%M%S')";
            fi;
            url=$(git config -f .gitmodules --get "$url_key");
            # If a branch is specified then use that one, otherwise
            # default to master
            branch=$(git config -f .gitmodules --get "$branch_key");
            if [ ! "$branch" ]; then branch="master"; fi;
            git submodule add -f -b "$branch" "$url" "$path";
        fi;
    done;

# In case the submodule exists in .git/config but the url is out of date

git submodule sync;

# Now actually pull all the modules. I used to use this...
#
# git submodule update --init --remote --force --recursive
# ...but the submodules would check out in detached HEAD state and I 
# didn't like that, so now I do this...

git submodule foreach --recursive 'git checkout $(git config -f $toplevel/.gitmodules submodule.$name.branch || echo master)';

Solution 6 - Git

I know its been a while, but I want to share this version that calls git config only once, doesn't requires a script and also handles branches:

git config -f .gitmodules --get-regexp '^submodule\.' | perl -lane'
$conf{$F[0]} = $F[1]}{
@mods = map {s,\.path$,,; $_} grep {/\.path$/} keys(%conf);
sub expand{$i = shift; map {$conf{$i . $_}} qw(.path .url .branch)}
for $i (@mods){
    ($path, $url, $branch) = expand($i);
    print(qq{rm -rf $path});
    print(qq{git submodule add -b $branch $url $path});
}
'

The only side effect is the output of the commands, nothing gets executed, so you can audit before committing to them.

This works with a simple copy and paste at the console, but should be trivial to put in a shell script.

example output:

rm -rf third-party/dht
git submodule add -b post-0.25-transmission https://github.com/transmission/dht third-party/dht
rm -rf third-party/libutp
git submodule add -b post-3.3-transmission https://github.com/transmission/libutp third-party/libutp
rm -rf third-party/libb64
git submodule add -b post-1.2.1-transmission https://github.com/transmission/libb64 third-party/libb64
rm -rf third-party/libnatpmp
git submodule add -b post-20151025-transmission https://github.com/transmission/libnatpmp third-party/libnatpmp
rm -rf third-party/miniupnpc
git submodule add -b post-2.0.20170509-transmission https://github.com/transmission/miniupnpc third-party/miniupnpc
rm -rf third-party/libevent
git submodule add -b post-2.0.22-transmission https://github.com/transmission/libevent third-party/libevent

Solution 7 - Git

For zsh users, try my function which has DRY_RUN=1 support to see what commands will be ran and only uses git to parse the file instead of sed.

gsub_file() {(
  set -eu

  cd "$(git rev-parse --show-toplevel)"

  submodule_paths=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "path" | awk '{ print $2 }')}"
  )
  submodule_urls=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "url" | awk '{ print $2 }')}"
  )
  submodule_branches=(
    "${(@f)$(git config --file ./.gitmodules --get-regexp "branch" | awk '{ print $2 }')}"
  )

  sh_c() {
    echo + "$*"
    if [ "${DRY_RUN-}" ]; then
      return
    fi
    eval "$@"
  }

  for (( i=1; i <= ${#submodule_paths[@]}; i++ )) do
    p="${submodule_paths[$i]}"
    if [ -d "$p" ]; then
      continue
    fi

    url="${submodule_urls[$i]}"
    unset b
    if [ "${submodule_branches[$i]-}" ]; then
      b="-b ${submodule_branches[$i]}" 
    fi
    sh_c git submodule add "${b-}" "$url" "$p"
  done
)}
Categories
Recommended Git Solutions
Recommended Git Submodules Solutions

Previous Solution:

Create array of n items based on integer value

Ruby

Next Solution:

How to share code between node.js apps?

node.jsNpm

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionevgeniuzView Question on Stackoverflow
Solution 1 - GitMark LongairView Answer on Stackoverflow
Solution 2 - GitKevin C.View Answer on Stackoverflow
Solution 3 - Gitmauron85View Answer on Stackoverflow
Solution 4 - GitHenryView Answer on Stackoverflow
Solution 5 - GitbokovView Answer on Stackoverflow
Solution 6 - GitalexgiraoView Answer on Stackoverflow
Solution 7 - GitnhooyrView Answer on Stackoverflow