AWS S3 sync --delete, removed new files in local

Amazon Web-ServicesAmazon S3Aws Cli

Amazon Web-Services Problem Overview


aws s3 sync --delete removed some new files.

For example:

There is a file in the bucket - S3://my-bucket/images/1.jpg

Then, I uploaded a file to the server: 2.jpg

There are 2 files in the server: 1.jpg and 2.jpg

Start running the sync cronjob:

aws s3 sync s3://my-bucket/ ./ --delete
aws s3 sync  ./ s3://my-bucket/ --delete

Why do we add --delete - we want to delete the files in s3 and sync it to the server.

We will upload files to the server and remove the files in s3.

Is there any way to fix it?

Amazon Web-Services Solutions


Solution 1 - Amazon Web-Services

By default, the aws sync command (see documentation) does not delete files. It simply copies new or modified files to the destination.

Using the --delete option deletes files that exist in the destination but not in the source.

So, if your source contains: 1.jpg and 2.jpg and the destination contains 1.jpg, 2.jpg and 3.jpg, then using the --delete option will delete 3.jpg from the destination.

I see that you are running the sync command in both directions. Your first command (that syncs S3 to a local directory) will delete any local files that are not in S3.

If your goal is to copy all local files to S3 and all S3 files to the local directory, without deleting any files, then do not use the --delete option.

Solution 2 - Amazon Web-Services

--delete option to remove files or objects from the target not present in the source.

Solution 3 - Amazon Web-Services

For the new s3cmd you can use the option --delete-removed to remove extra files in the destination, that are not present the source.

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionKeith KongView Question on Stackoverflow
Solution 1 - Amazon Web-ServicesJohn RotensteinView Answer on Stackoverflow
Solution 2 - Amazon Web-Servicessarat chandraView Answer on Stackoverflow
Solution 3 - Amazon Web-ServicesSeffView Answer on Stackoverflow