How can I delete folder on s3 with node.js?
node.jsAmazon Web-ServicesAmazon S3node.js Problem Overview
Yes, I know. There is no folder concept on s3 storage. but I really want to delete a specific folder from s3 with node.js. I tried two solutions, but both didn't work. My code is below: Solution 1: Deleting folder directly.
var key='level/folder1/folder2/';
var strReturn;
var params = {Bucket: MyBucket};
var s3 = new AWS.S3(params);
s3.client.listObjects({
Bucket: MyBucket,
Key: key
}, function (err, data) {
if(err){
strReturn="{\"status\":\"1\"}";
}else{
strReturn=+"{\"status\":\"0\"}";
}
res.send(returnJson);
console.log('error:'+err+' data:'+JSON.stringify(data));
});
Actually, I have a lot of files under folder2. I can delete single file from folder2 if I define key like this: var key='level/folder1/folder2/file1.txt', but it didn't work when I deleted a folder(key='level/folder1/folder2/'). Solution 2: I tried to set expiration to an object when I uploaded this file or folder to s3. code is below:
s3.client.putObject({
Bucket: Camera_Bucket,
Key: key,
ACL:'public-read',
Expires: 60
}
But it didn't either. After finishing uploading, I checked the properties of that file. it showed there was nothing value for expiry date:
Expiry Date:none
Expiration Rule:N/A
How can I delete folder on s3 with node.js?
node.js Solutions
Solution 1 - node.js
Here is an implementation in ES7 with an async
function and using listObjectsV2
(the revised List Objects API):
async function emptyS3Directory(bucket, dir) {
const listParams = {
Bucket: bucket,
Prefix: dir
};
const listedObjects = await s3.listObjectsV2(listParams).promise();
if (listedObjects.Contents.length === 0) return;
const deleteParams = {
Bucket: bucket,
Delete: { Objects: [] }
};
listedObjects.Contents.forEach(({ Key }) => {
deleteParams.Delete.Objects.push({ Key });
});
await s3.deleteObjects(deleteParams).promise();
if (listedObjects.IsTruncated) await emptyS3Directory(bucket, dir);
}
To call it:
await emptyS3Directory(process.env.S3_BUCKET, 'images/')
Solution 2 - node.js
You can use aws-sdk module for deleting folder. Because you can only delete a folder when it is empty, you should first delete the files in it. I'm doing it like this :
function emptyBucket(bucketName,callback){
var params = {
Bucket: bucketName,
Prefix: 'folder/'
};
s3.listObjects(params, function(err, data) {
if (err) return callback(err);
if (data.Contents.length == 0) callback();
params = {Bucket: bucketName};
params.Delete = {Objects:[]};
data.Contents.forEach(function(content) {
params.Delete.Objects.push({Key: content.Key});
});
s3.deleteObjects(params, function(err, data) {
if (err) return callback(err);
if (data.IsTruncated) {
emptyBucket(bucketName, callback);
} else {
callback();
}
});
});
}
Solution 3 - node.js
According to accepted answer I created promise returned function, so you can chain it.
function emptyBucket(bucketName){
let currentData;
let params = {
Bucket: bucketName,
Prefix: 'folder/'
};
return S3.listObjects(params).promise().then(data => {
if (data.Contents.length === 0) {
throw new Error('List of objects empty.');
}
currentData = data;
params = {Bucket: bucketName};
params.Delete = {Objects:[]};
currentData.Contents.forEach(content => {
params.Delete.Objects.push({Key: content.Key});
});
return S3.deleteObjects(params).promise();
}).then(() => {
if (currentData.Contents.length === 1000) {
emptyBucket(bucketName, callback);
} else {
return true;
}
});
}
Solution 4 - node.js
listObjectsV2 list files only with current dir Prefix not with subfolder Prefix. If you want to delete folder with subfolders recursively this is the source code: https://github.com/tagspaces/tagspaces-common/blob/master/common-aws/io-objectstore.js#L826
deleteDirectoryPromise = async (path: string): Promise<Object> => {
const prefixes = await this.getDirectoryPrefixes(path);
if (prefixes.length > 0) {
const deleteParams = {
Bucket: this.config.bucketName,
Delete: { Objects: prefixes }
};
return this.objectStore.deleteObjects(deleteParams).promise();
}
return this.objectStore
.deleteObject({
Bucket: this.config.bucketName,
Key: path
})
.promise();
};
/**
* get recursively all aws directory prefixes
* @param path
*/
getDirectoryPrefixes = async (path: string): Promise<any[]> => {
const prefixes = [];
const promises = [];
const listParams = {
Bucket: this.config.bucketName,
Prefix: path,
Delimiter: '/'
};
const listedObjects = await this.objectStore
.listObjectsV2(listParams)
.promise();
if (
listedObjects.Contents.length > 0 ||
listedObjects.CommonPrefixes.length > 0
) {
listedObjects.Contents.forEach(({ Key }) => {
prefixes.push({ Key });
});
listedObjects.CommonPrefixes.forEach(({ Prefix }) => {
prefixes.push({ Key: Prefix });
promises.push(this.getDirectoryPrefixes(Prefix));
});
// if (listedObjects.IsTruncated) await this.deleteDirectoryPromise(path);
}
const subPrefixes = await Promise.all(promises);
subPrefixes.map(arrPrefixes => {
arrPrefixes.map(prefix => {
prefixes.push(prefix);
});
});
return prefixes;
};
Solution 5 - node.js
A much simpler way is to fetch all objects (keys) at that path & delete them. In each call fetch 1000 keys & s3 deleteObjects can delete 1000 keys in each request too. Do that recursively to achieve the goal
> Written in typescript
/**
* delete a folder recursively
* @param bucket
* @param path - without end /
*/
deleteFolder(bucket: string, path: string) {
return new Promise((resolve, reject) => {
// get all keys and delete objects
const getAndDelete = (ct: string = null) => {
this.s3
.listObjectsV2({
Bucket: bucket,
MaxKeys: 1000,
ContinuationToken: ct,
Prefix: path + "/",
Delimiter: "",
})
.promise()
.then(async (data) => {
// params for delete operation
let params = {
Bucket: bucket,
Delete: { Objects: [] },
};
// add keys to Delete Object
data.Contents.forEach((content) => {
params.Delete.Objects.push({ Key: content.Key });
});
// delete all keys
await this.s3.deleteObjects(params).promise();
// check if ct is present
if (data.NextContinuationToken) getAndDelete(data.NextContinuationToken);
else resolve(true);
})
.catch((err) => reject(err));
};
// init call
getAndDelete();
});
}
Solution 6 - node.js
According to Emi's answer I made a npm package so you don' t need to write the code yourself. Also the code is written in typescript.
See https://github.com/bingtimren/s3-commons/blob/master/src/lib/deleteRecursive.ts
Solution 7 - node.js
You can try this:
import { s3DeleteDir } from '@zvs001/s3-utils'
import { S3 } from 'aws-sdk'
const s3Client = new S3()
await s3DeleteDir(s3Client, {
Bucket: 'my-bucket',
Prefix: `folder/`,
})
Solution 8 - node.js
The accepted answer throws an error when used in typescript, and it is do Objects array in deleteParams. I made it work by modifying the code in the following way. I'm very new to Typescript but at least it is working now.
async function emptyS3Directory(prefix: string) {
const listParams = {
Bucket: "bucketName",
Prefix: prefix, // ex. path/to/folder
};
const listedObjects = await s3.listObjectsV2(listParams).promise();
if (listedObjects.Contents.length === 0) return;
const deleteParams = {
Bucket: bucketName,
Delete: { Objects: [] as any },
};
listedObjects.Contents.forEach((content: any) => {
deleteParams.Delete.Objects.push({ Key: content.Key });
});
await s3.deleteObjects(deleteParams).promise();
if (listedObjects.IsTruncated) await emptyS3Directory(prefix);
}
Solution 9 - node.js
I like the list objects and then delete approach, which is what the aws cmd line does behind the scenes btw. But I didn't want to await the list (few seconds) before deleting them. So I use this 1 step (background) process, I found it slightly faster. You can await the child process if you really want to confirm deletion, but I found that took around 10 seconds, so I don't bother I just fire and forget and check logs instead. The entire API call with other stuff now takes 1.5s which is fine for my situation.
var CHILD = require("child_process").exec;
function removeImagesAndTheFolder(folder_name_str, callback){
var cmd_str = "aws s3 rm s3://"
+ IMAGE_BUCKET_STR
+ "/" + folder_name_str
+ "/ --recursive";
if(process.env.NODE_ENV === "development"){
//When not on an EC2 with a role I use my profile
cmd_str += " " + "--profile " + LOCAL_CONFIG.PROFILE_STR;
}
// In my situation I return early for the user. You could make them wait tho'.
callback(null, {"msg_str": "Check later that these images were actually removed."});
//do not return yet still stuff to do
CHILD(cmd_str, function(error, stdout, stderr){
if(error || stderr){
console.log("Problem removing this folder with a child process:" + stderr);
}else{
console.log("Child process completed, here are the results", stdout);
}
});
}
Solution 10 - node.js
You can delete an empty folder the same way you delete a file. In order to delete a non-empty folder on AWS S3, you'll need to empty it first by deleting all files and folders inside. Once the folder is empty, you can delete it as a regular file. The same applies to the bucket deletion. We've implemented it in this app called Commandeer so you can do it from a GUI.