Best way to call an asynchronous function within map?
Javascriptnode.jsAsynchronousJavascript Problem Overview
I'm mapping over an array and for one of the return values of the new object, I need to make an asynchronous call.
var firebaseData = teachers.map(function(teacher) {
return {
name: teacher.title,
description: teacher.body_html,
image: urlToBase64(teacher.summary_html.match(/src="(.*?)"/)[1]),
city: metafieldTeacherData[teacher.id].city,
country: metafieldTeacherData[teacher.id].country,
state: metafieldTeacherData[teacher.id].state,
studioName: metafieldTeacherData[teacher.id].studioName,
studioURL: metafieldTeacherData[teacher.id].studioURL
}
});
The implementation of that function will look something like
function urlToBase64(url) {
request.get(url, function (error, response, body) {
if (!error && response.statusCode == 200) {
return "data:" + response.headers["content-type"] + ";base64," + new Buffer(body).toString('base64');
}
});
}
I'm not clear what's the best approach to do this... promises? Nested callbacks? Use something in ES6 or ES7 and then transpile with Babel?
What's the current best way to implement this?
Javascript Solutions
Solution 1 - Javascript
update in 2018: Promise.all
async function within map callback is easier to implement:
let firebaseData = await Promise.all(teachers.map(async teacher => {
return {
name: teacher.title,
description: teacher.body_html,
image: await urlToBase64(teacher.summary_html.match(/src="(.*?)"/)[1]),
city: metafieldTeacherData[teacher.id].city,
country: metafieldTeacherData[teacher.id].country,
state: metafieldTeacherData[teacher.id].state,
studioName: metafieldTeacherData[teacher.id].studioName,
studioURL: metafieldTeacherData[teacher.id].studioURL
}
}));
async function urlToBase64(url) {
return request.get(url, function (error, response, body) {
if (!error && response.statusCode == 200) {
return "data:" + response.headers["content-type"] + ";base64," + new Buffer(body).toString('base64');
}
});
}
Edit@2018/04/29
: I put the general example for everyone:
Edit@2019/06/19
: async/await should have try/catch to handle error: it would throw a warning message if error occur;
let data = await Promise.all(data.map(async (item) => {
try {
item.fetchItem = await fetchFunc(item.fetchParams);
return item;
} catch(err) {
throw err;
}
}));
Solution 2 - Javascript
One approach is Promise.all
(ES6).
This answer will work in Node 4.0+. Older versions will need a Promise polyfill or library. I have also used ES6 arrow functions, which you could replace with regular function
s for Node < 4.
This technique manually wraps request.get
with a Promise. You could also use a library like request-promise.
function urlToBase64(url) {
return new Promise((resolve, reject) => {
request.get(url, function (error, response, body) {
if (!error && response.statusCode == 200) {
resolve("data:" + response.headers["content-type"] + ";base64," + new Buffer(body).toString('base64'));
} else {
reject(response);
}
});
})
}
// Map input data to an Array of Promises
let promises = input.map(element => {
return urlToBase64(element.image)
.then(base64 => {
element.base64Data = base64;
return element;
})
});
// Wait for all Promises to complete
Promise.all(promises)
.then(results => {
// Handle results
})
.catch(e => {
console.error(e);
})
Solution 3 - Javascript
In 2020 we now have the for await...of
syntax of ECMAScript2021 that significantly simplifies things:
So you can now simply do this:
//return an array of promises from our iteration:
let promises = teachers.map(async m => {
return await request.get(....);
});
//simply iterate those
//val will be the result of the promise not the promise itself
for await (let val of promises){
....
}
Solution 4 - Javascript
You can use async.map.
var async = require('async');
async.map(teachers, mapTeacher, function(err, results){
// results is now an array of stats for each file
});
function mapTeacher(teacher, done) {
// computing stuff here...
done(null, teacher);
}
note that all teachers will be processed in parallel - you can use also this functions:
mapSeries(arr, iterator, [callback])
maps one by one
mapLimit(arr, limit, iterator, [callback])
maps limit
at same time
Solution 5 - Javascript
I had a similar problem and found this to be easier (I'm using Kai's generic template). Below, you only need to use one await
. I was also using an ajax function as my async function:
function asyncFunction(item) {
return $.ajax({
type: "GET",
url: url,
success: response => {
console.log("response received:", response);
return response;
}, error: err => {
console.log("error in ajax", err);
}
});
}
let data = await Promise.all(data.map(item => asyncFunction(item)));
Solution 6 - Javascript
I am using an async function over the array. And not using array.map, but a for function. It's something like this:
const resultingProcessedArray = async function getSomeArray() {
try {
let { data } = await axios({url: '/myUrl', method:'GET'}); //initial array
let resultingProcessedArray = [];
for (let i = 0, len = data.items.length; i < len; i++) {
let results = await axios({url: `/users?filter=id eq ${data.items[i].someId}`, method:'GET'});
let domainName = results.data.items[0].domainName;
resultingProcessedArray.push(Object.assign(data.items[i], {domainName}));
}
return resultingProcessedArray;
} catch (err) {
console.error("Unable to fetch the data", err);
return [];
}
};
Solution 7 - Javascript
I had to write this, for the sake of convenience. Otherwise, I might need <https://github.com/mcollina/make-promises-safe>
export async function mapAsync<T, U>(
arr: T[],
callbackfn: (value: T, index: number, array: T[]) => Promise<U>,
thisArg?: any
) {
return await Promise.all(arr.map(async (value, index, array) => {
try {
return await callbackfn(value, index, array);
} catch(e) {
throw e;
}
}, thisArg));
}
Solution 8 - Javascript
For production purposes you probably want to use a lib like lodasync, you should not reinvent the wheel:
import { mapAsync } from 'lodasync'
const result = await mapAsync(async(element) => {
return 3 + await doSomething(element)
}, array)
It uses promises, has no dependencies, and is as fast as it gets.
Solution 9 - Javascript
By using Promise.all you can make map and forEach work with async functions (i.e. Promises).
To make filter, some and every work you can first use an async map (that in turn uses Promise.all) and then go through the true/false values and synchronously do the filtering/evaluation.
To make reduce and reduceRight work with async functions you can wrap the original function in a new one that waits for the accumulator to resolve.
Using this knowledge it is possible to modify the original array methods in a way so that they continue to work "as usual" with normal/synchronous functions but will also work with async functions.
// a 'mini library' (save it somewhere and import it once/project)
(() => {
let AsyncFunction = Object.getPrototypeOf(async e => e).constructor;
['map', 'forEach'].forEach(method => {
let orgMethod = Array.prototype[method];
Array.prototype[method] = function (func) {
let a = orgMethod.call(this, func);
return func instanceof AsyncFunction ? Promise.all(a) : a;
};
});
['filter', 'some', 'every'].forEach(method => {
let orgMethod = Array.prototype[method];
Array.prototype[method] = function (func) {
if (func instanceof AsyncFunction) {
return (async () => {
let trueOrFalse = await this.map(func);
return orgMethod.call(this, (_x, i) => trueOrFalse[i]);
})();
}
else {
return orgMethod.call(this, func);
}
};
});
['reduce', 'reduceRight'].forEach(method => {
let orgMethod = Array.prototype[method];
Array.prototype[method] = function (...args) {
if (args[0] instanceof AsyncFunction) {
let orgFunc = args[0];
args[0] = async (...args) => {
args[0] = await args[0];
return orgFunc.apply(this, args);
};
}
return orgMethod.apply(this, args);
};
});
})();
// AND NOW:
// this will work
let a = [1, 2, 3].map(x => x * 3); // => [3, 6, 9]
let b = [1, 2, 3, 4, 5, 6, 7].filter(x => x > 3); // [4, 5, 6, 7]
let c = [1, 2, 3, 4, 5].reduce((acc, val) => acc + val); // => 15
// this will also work
let x = await [1, 2, 3].map(async x => x * 3);
let y = await [1, 2, 3, 4, 5, 6, 7].filter(async x => x > 3);
let z = await [1, 2, 3, 4, 5].reduce(async (acc, val) => acc + val);
Solution 10 - Javascript
If you'd like to map over all elements concurrently:
function asyncMap(arr, fn) {
return Promise.all(arr.map(fn));
}
If you'd like to map over all elements non-concurrently (e.g. when your mapping function has side effects or running mapper over all array elements at once would be too resource costly):
Option A: Promises
function asyncMapStrict(arr, fn) {
return new Promise((resolve) => {
const result = [];
arr.reduce(
(promise, cur, idx) => promise
.then(() => fn(cur, idx, arr)
.then((res) => {
result.push(res);
})),
Promise.resolve(),
).then(() => resolve(result));
});
}
Option B: async/await
async function asyncMapStrict(arr, fn) {
const result = [];
for (let idx = 0; idx < arr.length; idx += 1) {
const cur = arr[idx];
result.push(await fn(cur, idx, arr));
}
return result;
}
Solution 11 - Javascript
Try amap(), the asynchronous map function below:
async function amap(arr,fun) {
return await Promise.all(arr.map(async v => await fun(v)))
}
Or, written in a more concise way:
let amap = async (arr,fun) => await Promise.all(arr.map(async v => await fun(v)))
Usage:
let arr = await amap([1,2,3], async x => x*2)
console.log(arr) // [2, 4, 6]
Solution 12 - Javascript
The best way to call an async function within map
is to use a map
created expressly for async functions.
For a function to be async, it must return a Promise.
function urlToBase64(url) {
return new Promise((resolve, reject) => {
request.get(url, function (error, response, body) {
if (error) {
reject(error)
} else if (response && response.statusCode == 200) {
resolve(
"data:" + response.headers["content-type"] + ";base64," + new Buffer(body).toString('base64');
)
} else {
reject(new Error('invalid response'))
}
});
})
}
Now, we can map:
const { pipe, map, get } = require('rubico')
const metafieldTeacherData = {} // { [teacher_id]: {...}, ... }
const parseTeacher = teacher => ({
name: teacher.title,
description: teacher.body_html,
image: urlToBase64(teacher.summary_html.match(/src="(.*?)"/)[1]),
city: metafieldTeacherData[teacher.id].city,
country: metafieldTeacherData[teacher.id].country,
state: metafieldTeacherData[teacher.id].state,
studioName: metafieldTeacherData[teacher.id].studioName,
studioURL: metafieldTeacherData[teacher.id].studioURL
})
const main = async () => {
const teachers = [] // array full of teachers
const firebaseData = await map(pipe([
parseTeacher,
get('studioURL'),
urlToBase64,
]))(teachers)
console.log(firebaseData) // > ['data:application/json;base64,...', ...]
}
main()
rubico's map worries about Promise.all
so you don't have to.
Solution 13 - Javascript
Using IIFE and Promise.all, can make a simple use cases.
await Promise.all(arr.map(el=>(async _=>{
// some async code
})()))
This IIFE can return a promise which is used as the return value for map function.
(async _=>{
// some async code
})()
So arr.map will return a list of promise to Promise.all to handle.
Example
const sleep = (ms) => {
return new Promise((resolve, reject) => {
setTimeout(_ => {
resolve()
}, ms)
});
}
await Promise.all([1000,2000,3000].map(el=>(async _=>{
await sleep(el)
console.log(el)
return el
})()))
Solution 14 - Javascript
Here is a simple function that will let you choose to await
each mapping operation (serial
) or run all mappings in parallel
.
The mapper function need not return a promise
either.
async function asyncMap(items, mapper, options = {
parallel: true
}) {
const promises = items.map(async item => options.parallel ? mapper(item) : await mapper(item))
return Promise.all(promises)
}
Typescript Version
async function asyncMap<I, O>(items: I[], mapper: (item: I) => O, options = {
parallel: true
}): Promise<O[]> {
const promises = items.map(async item => options.parallel ? mapper(item) : await mapper(item))
return Promise.all(promises)
}
A working snippet for science
async function asyncMap(items, mapper, options = {
parallel: true
}) {
const promises = items.map(async item => options.parallel ? mapper(item) : await mapper(item))
return Promise.all(promises)
}
// A test to multiply number by 2 after 50 milliseconds
function delay(num) {
return new Promise((resolve) => {
setTimeout(() => {
resolve(num * 2)
}, 50)
})
}
;
(async() => {
const data = [1, 2, 3, 4]
const resParallel = await asyncMap(data, it => delay(it))
const resSerial = await asyncMap(data, it => delay(it), {
parallel: false
})
console.log(data)
console.log(resParallel)
console.log(resSerial)
})();