Moving large number of files

Bash

Bash Problem Overview


If I run the command mv folder2/*.* folder, I get "argument list too long" error.

I find some example of ls and rm, dealing with this error, using find folder2 -name "*.*". But I have trouble applying them to mv.

Bash Solutions


Solution 1 - Bash

find folder2 -name '*.*' -exec mv {} folder \;

-exec runs any command, {} inserts the filename found, \; marks the end of the exec command.

Solution 2 - Bash

The other find answers work, but are horribly slow for a large number of files, since they execute one command for each file. A much more efficient approach is either to use + at the end of find, or use xargs:

# Using find ... -exec +
find folder2 -name '*.*' -exec mv --target-directory=folder '{}' +

# Using xargs
find folder2 -name '*.*' | xargs mv --target-directory=folder

Solution 3 - Bash

find folder2 -name '*.*' -exec mv \{\} /dest/directory/ \;

Solution 4 - Bash

First, thanks to Karl's answer. I have only minor correction to this.

My scenario:

Millions of folders inside /source/directory, containing subfolders and files inside. Goal is to copy it keeping the same directory structure.

To do that I use such command:

find /source/directory -mindepth 1 -maxdepth 1 -name '*' -exec mv {} /target/directory \;

Here:

  • -mindepth 1 : makes sure you don't move root folder
  • -maxdepth 1 : makes sure you search only for first level children. So all it's content is going to be moved too, but you don't need to search for it.

Commands suggested in answers above made result directory structure flat - and it was not what I looked for, so decided to share my approach.

Solution 5 - Bash

This one-liner command should work for you. Yes, it is quite slow, but works even with millions of files.

for i in /folder1/*; do mv "$i" /folder2; done

It will move all the files from folder /folder1 to /folder2.

Solution 6 - Bash

find doesn't work with really long lists of files, it will give you the same error "Argument list too long". Using a combination of ls, grep and xargs worked for me:

$ ls|grep RadF|xargs mv -t ../fd/

It did the trick moving about 50,000 files where mv and find alone failed.

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionDrXChengView Question on Stackoverflow
Solution 1 - BashKarl BielefeldtView Answer on Stackoverflow
Solution 2 - BashIdelicView Answer on Stackoverflow
Solution 3 - BashInternetSeriousBusinessView Answer on Stackoverflow
Solution 4 - BashGadgetView Answer on Stackoverflow
Solution 5 - BashSidView Answer on Stackoverflow
Solution 6 - Bashuser2309000View Answer on Stackoverflow