Two way sync with rsync

FileSyncRsyncTwo Way

File Problem Overview


I have a folder a/ and a remote folder A/. I now run something like this on a Makefile:

get-music:
 rsync -avzru server:/media/10001/music/ /media/Incoming/music/

put-music:
 rsync -avzru /media/Incoming/music/ server:/media/10001/music/

sync-music: get-music put-music

when I make sync-music, it first gets all the diffs from server to local and then the opposite, sending all the diffs from local to server.

This works very well only if there are just updates or new files on the future. If there are deletions, it doesn't do anything.

In rsync there is --delete and --delete-after options to help accomplish what I want but thing is, it doesn't work on a 2-way-sync.

If I want to delete server files on a syn, when local files have been deleted, it works, but if, for some reason (explained after) I have some files that aren't in the server but exist locally and they were deleted, I want locally to remove them and not server copied (as it happens).

Thing is I have 3 machines in context:

  1. desktop
  2. notebook
  3. home-server

So, sometimes, server will have files that were deleted with a notebook sync, for example and then, when I run a sync with my desktop (where the deleted server files still exist on) I want these files to be deleted and not to be copied again to the server.

I guess this is only possible with a database and track of operations :P

Any simpler solutions? Thank you.

File Solutions


Solution 1 - File

Try Unison: http://www.cis.upenn.edu/~bcpierce/unison/

Syntax:

unison dirA/ dirB/

Unison asks what to do when files are different, but you can automate the process by using the following which accepts default (nonconflicting) options:

unison -auto dirA/ dirB/

unison -batch dirA/ dirB/ asks no questions at all, and writes to output how many files were ignored (because they conflicted).

Note: I am no longer using Unison (I use NextCloud, which doesn't address the original use case). However, note that rsync is not designed for bidirectional sync, while unison is. unison may have its bugs (as any other piece of software) and its wrinkles. I am surprised it seems to be actively maintained now (last time I looked I think I thought it looked dead), but I'm not sure what's the state nowadays. I haven't had the need to have a two-way file synchronizer, so there may be better options, though.

Solution 2 - File

Since the original question also involves a desktop and laptop and example involving music files (hence he's probably using a GUI), I'd also mention one of the best bi-directional, multi-platform, free and open source programs to date: FreeFileSync.

It's GUI based, very fast and intuitive, comes with filtering and many other options, including the ability to remote connect, to view and interactively manage "collisions" (in example, files with similar timestamps) and to switch between bidirectional transfer, mirroring and so on.

FreeFileSync can easily sync two computers on the same network and also sync two computers on different and remote networks.

  • On same network: have FreeFileSync use the local file system on one side and a shared network drive / path on the other. On Windows systems you enable file / disk sharing on one computer and access that share from the other. I use FreeFileSync this way to keep my main development PC source code synced with my 2 laptops. I have also synced one of these laptops with a Linux server with Samba installed and sharing one of its directories.
  • Across networks: create a VPN and do the same as above. FreeFileSync will see the remote disk as it was on the local network. Or buy one router that allows you to connect a USB disk to it and share over the internet. I have installed a VPN on a remote Linux server and used it through the OpenVPN Windows client.

Solution 3 - File

You could also try bitpocket: https://github.com/sickill/bitpocket

Solution 4 - File

Try this,

get-music:
 rsync -avzru --delete-excluded server:/media/10001/music/ /media/Incoming/music/

put-music:
 rsync -avzru --delete-excluded /media/Incoming/music/ server:/media/10001/music/

sync-music: get-music put-music

I just test this and it worked for me. I'm doing a 2-way sync between Windows7 (using cygwin with the rsync package installed) and FreeNAS fileserver (FreeNAS runs on FreeBSD with rsync package pre-installed).

Solution 5 - File

You might use Osync: http://www.netpower.fr/osync , which is rsync based with intelligent deletion propagation. it has also multiple options like resuming a halted execution, soft deletion, and time control.

Solution 6 - File

You could try csync, it is the sync engine under the hood of owncloud.

Solution 7 - File

I'm surprised no one has mentioned Syncthing yet. I have been using it for years to synchronize my phone, my tablet and my two laptops. One time I also used it to send 10 GB of photos to my family ~600 km away, straight from my machine to their machine, and it was incredibly fast (despite the data getting routed through Syncthing's discovery server to work around NAT issues). I also tried OwnCloud/NextCloud at some point but Syncthing has been much more reliable and, also, much faster.

Solution 8 - File

I'm now using SparkleShare https://www.sparkleshare.org/

works on mac, linux and windows.

Solution 9 - File

I'm not sure whether it works with two syncing but for the --delete to work you also need to add the --recursive parameter as well.

Solution 10 - File

Rclone is what you are looking for. Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers including local filesystems. Rclone was previously known as Swiftsync and has been available since 2013.

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionmwmView Question on Stackoverflow
Solution 1 - FilealexView Answer on Stackoverflow
Solution 2 - FileDario FumagalliView Answer on Stackoverflow
Solution 3 - FilesaifView Answer on Stackoverflow
Solution 4 - FileChris CollinsView Answer on Stackoverflow
Solution 5 - FileOrsiris de JongView Answer on Stackoverflow
Solution 6 - Filemarton78View Answer on Stackoverflow
Solution 7 - FilebaluView Answer on Stackoverflow
Solution 8 - FilemwmView Answer on Stackoverflow
Solution 9 - FileBoardyView Answer on Stackoverflow
Solution 10 - Fileuser1461607View Answer on Stackoverflow