Gitlab repository mirroring
GitVersion ControlMirroringGitlabGit Problem Overview
Is it possible to have gitlab set up to automatically sync (mirror) a repository hosted at another location?
At the moment, the easiest way I know of doing this involves manually pushing to the two (gitlab and the other) repository, but this is time consuming and error prone.
The greatest problem is that a mirror can resynchronize if two users concurrently push changes to the two different repositories. The best method I can come up with to prevent this issue is to ensure users can only push to one of the repositories.
Git Solutions
Solution 1 - Git
Update Dec 2016: Mirroring is suported with GitLAb EE 8.2+: see "Repository mirroring".
As commented by Xiaodong Qi:
> This answer can be simplified without using any command lines (just set it up on Gitlab repo management interface)
Original answer (January 2013)
If your remote mirror repo is a bare repo, then you can add a post-receive hook to your gitlab-managed repo, and push to your remote repo in it.
#!/bin/bash
git push --mirror [email protected]:/path/to/repo.git
As Gitolite (used by Gitlab) mentions:
> if you want to install a hook in only a few specific repositories, do it directly on the server.
which would be in:
~git/repositories/yourRepo.git/hook/post-receive
Caveat (Update Ocotober 2014)
Ciro Santilli points out in the comments:
> Today (Q4 2014) this will fail because GitLab automatically symlinks github.com/gitlabhq/gitlab-shell/tree/… into every repository it manages.
So if you make this change, every repository you modify will try to push.
Not to mention possible conflicts when upgrading gitlab-shell
, and that the current script is a ruby script, not bash (and you should not remove it!).
> You could correct this by reading the current directory name and ensuring bijection between that and the remote, but I recommend people to stay far far away from those things
See (and vote for) feeadback "Automatic push to remote mirror repo after push to GitLab Repo".
Update July 2016: I see this kind of feature added for GitLab EE (Enterprise Edition): MR 249
- Add ability to enter remote push URL under Mirror Repository settings
- Add implementation code to push to remote repository
- Add new background worker
- Show latest update date and sync errors if they exists.
- Sync remote mirror every hour.
Note that the recent Remote Mirror Repository
(issues 17940) can be tricky:
> I'm currently trying to shift main development of the Open Source npm modules of my company Lossless GmbH (https://www.npmjs.com/~lossless) from GitHub.com to GitLab.com
> I'm importing all the repos from GitHub, however when I try to switch off Mirror Repository
and switch on Remote Mirror Repository
with the original GitHub URL I get an error saying:
Remote mirrors url is already in use
Here is one of the repos this fails with: https://gitlab.com/pushrocks/npmts Edited 2 months ago
> turns out, it just requires multiple steps:
> - disable the Mirror Repository
-
press save
-
remove the URl
-
press save
-
then add the Remote Mirror
Solution 2 - Git
If not hosting your own GitLab, it's worth knowing GitLab.com has introduced this feature directly, without any workarounds.
- From within a project use the gear icon to select Mirror Repository
- Scroll down to Push to a remote repository
- Checkmark Remote mirror repository: Automatically update the remote mirror's branches, tags, and commits from this repository every hour.
- Enter the repository you want to update; for GitHub at least you can include your username and password in the URL, like so:
https://yourgithubusername:[email protected]/agaric/guts_discuss_resource.git
Note that if you are pulling from a remote repository, it will still push on to the remote repository set here. I haven't tried it, but you should be able to push to and pull from the same repository.
Solution 3 - Git
A quick summary of configure mirroring repositories with GitLab and GitHub
- GitHub: Settings > Developer settings > Personal access tokens
- You need enable:
public_repo
- You need enable:
- GitLab: GitLab project > Settings > Repository > Mirroring repositories
Push
- Git repository URL:
https://[email protected]/username/repo.git
- Mirror direction:
Push
- Authentication method:
Password
- Password:
your_github_token
- Password:
Pull
- Git repository URL:
https://github.com/username/repo.git
- Mirror direction:
Pull
- Authentication method:
leave_empty
Solution 4 - Git
I also created a project to mirror repositories in GitLab 6 through the API (API mostly used on project creation only).
Solution 5 - Git
The best option today is to use GitLab CI. It is essentially an already implemented server for the webhooks, which automatically clones for you and let's you run arbitrary shell commands: all you have to do then is to push.
services are a the best option if someone implements them: they live in the source tree, would do a single push, and require no extra deployment overhead.
The key implementation difficulty now is how to store the push credentials safely: likely the best option for GitHub is to get a key somehow (Oauth on UI through the service would be perfect) and store that plaintext.
Another option which has just been added are custom hooks.
Solution 6 - Git
UPDATE 2020
> As you all know about Free Gitlab Mirror Support Ended on March 22 for external private Repositories.
With cron automation here is little code in php:-
Solution 7 - Git
You can use hooks to customize a script that runs after some commit. With that you can send the new changes to another repository. Look for more information about hook in the following page: http://git-scm.com/book/en/Customizing-Git-Git-Hooks
Solution 8 - Git
The best option is not to use post-receive hooks, but deployments commands that perform synchronisation via rsync, using Capistrano if you like Ruby, using Shipit if you prefer Javascript (Grunt).
Solution 9 - Git
GitLab Enterprise Edition now supports repository mirroring as of version 8.2. Information about how to configure this is in the Repository Mirroring help topic.
Solution 10 - Git
GitBitLabHub allows you automatically mirror repositories between Bitbucket / Gitlab / Github using simple webhooks.
- Generate an ssh key for source and destination repositories:
ssh-keygen -t rsa -f ~/.ssh/project_id_rsa
- It will generate 2 keys, the PRIVATE key to ~/.ssh/project_id_rsa and the PUBLIC key to ~/.ssh/project_id_rsa.pub.
- The PUBLIC key is used as Deploy Key while the PRIVATE key should be used in SRC_DEPLOY_KEY and DEST_DEPLOY_KEY env variables.
- Add this PUBLIC key as a Deploy Key for the source and destination repositories. Depending on the platform it's called Deploy Key or Access key. See how to add Deploy/Access Key to bitbucket, bitbucket access keys, gitlab, github.
- Add the PUBLIC key to the source repo with read-only access.
- Add the PUBLIC key to the destination repo with write access.
- Run this container in docker with all environment variables configured properly.
docker run -it \
-e SRC_REPO[email protected]:vendor/src_repo.git \
-e DEST_REPO[email protected]:2222/vendor/dest_repo.git \
-e SRC_DEPLOY_KEY=base64_encoded_key== \
-e DEST_DEPLOY_KEY=base64_encoded_key== \
-p 8181:8080/tcp \
karser/gitbitlabhub
- Use the PRIVATE key for SRC_DEPLOY_KEY and DEST_DEPLOY_KEY env variables. Don't forget to encode it with base64:
base64 -w 0 < ~/.ssh/project_id_rsa
- If you configured everything properly it should mirror the repo after the first launch. If it didn't see the logs in the container output.