We need to transfer `15TB` of data from one server to another as fast as we can. We&#39;re currently using `rsync` but we&#39;re only getting speeds of around `150Mb/s`, when our network is capable of `900+Mb/s` (tested with `iperf`). I&#39;ve done tests of the disks, network, etc and figured it&#39;s just that rsync is only transferring one file at a time which is causing the slowdown.

I found a script to run a different rsync for each folder in a directory tree (allowing you to limit to x number), but I can&#39;t get it working, it still just runs one rsync at a time.

I found the `script` [here][1] (copied below).

Our directory tree is like this:

    /main
       - /files
          - /1
             - 343
                - 123.wav
                - 76.wav
             - 772
                - 122.wav
             - 55
                - 555.wav
                - 324.wav
                - 1209.wav
             - 43
                - 999.wav
                - 111.wav
                - 222.wav
          - /2
             - 346
                - 9993.wav
             - 4242
                - 827.wav
          - /3
             - 2545
                - 76.wav
                - 199.wav
                - 183.wav
             - 23
                - 33.wav
                - 876.wav
             - 4256
                - 998.wav
                - 1665.wav
                - 332.wav
                - 112.wav
                - 5584.wav

So what I&#39;d like to happen is to create an rsync for each of the directories in /main/files, up to a maximum of, say, 5 at a time. So in this case, 3 rsyncs would run, for `/main/files/1`, `/main/files/2` and `/main/files/3`.

I tried with it like this, but it just runs 1 rsync at a time for the `/main/files/2` folder:

    #!/bin/bash
    
    # Define source, target, maxdepth and cd to source
    source=&quot;/main/files&quot;
    target=&quot;/main/filesTest&quot;
    depth=1
    cd &quot;${source}&quot;
    
    # Set the maximum number of concurrent rsync threads
    maxthreads=5
    # How long to wait before checking the number of rsync threads again
    sleeptime=5
    
    # Find all folders in the source directory within the maxdepth level
    find . -maxdepth ${depth} -type d | while read dir
    do
    	# Make sure to ignore the parent folder
    	if [ `echo &quot;${dir}&quot; | awk -F&#39;/&#39; &#39;{print NF}&#39;` -gt ${depth} ]
    	then
    		# Strip leading dot slash
    		subfolder=$(echo &quot;${dir}&quot; | sed &#39;s@^\./@@g&#39;)
    		if [ ! -d &quot;${target}/${subfolder}&quot; ]
    		then
    			# Create destination folder and set ownership and permissions to match source
    			mkdir -p &quot;${target}/${subfolder}&quot;
    			chown --reference=&quot;${source}/${subfolder}&quot; &quot;${target}/${subfolder}&quot;
    			chmod --reference=&quot;${source}/${subfolder}&quot; &quot;${target}/${subfolder}&quot;
    		fi
    		# Make sure the number of rsync threads running is below the threshold
    		while [ `ps -ef | grep -c [r]sync` -gt ${maxthreads} ]
    		do
    			echo &quot;Sleeping ${sleeptime} seconds&quot;
    			sleep ${sleeptime}
    		done
    		# Run rsync in background for the current subfolder and move one to the next one
    		nohup rsync -a &quot;${source}/${subfolder}/&quot; &quot;${target}/${subfolder}/&quot; &lt;/dev/null &gt;/dev/null 2&gt;&amp;1 &amp;
    	fi
    done
    
    # Find all files above the maxdepth level and rsync them as well
    find . -maxdepth ${depth} -type f -print0 | rsync -a --files-from=- --from0 ./ &quot;${target}/&quot;


  [1]: http://www.krazyworks.com/making-rsync-faster/

All Simultaneous Solutions on Gang of Coders

Total of 1 Simultaneous Solutions

Speed up rsync with Simultaneous/Concurrent File Transfers?