How to remove all duplicates from an array of objects?
JavascriptArraysObjectDuplicatesJavascript Problem Overview
I have an object that contains an array of objects.
obj = {};
obj.arr = new Array();
obj.arr.push({place:"here",name:"stuff"});
obj.arr.push({place:"there",name:"morestuff"});
obj.arr.push({place:"there",name:"morestuff"});
I'm wondering what is the best method to remove duplicate objects from an array. So for example, obj.arr
would become...
{place:"here",name:"stuff"},
{place:"there",name:"morestuff"}
Javascript Solutions
Solution 1 - Javascript
How about with some es6
magic?
obj.arr = obj.arr.filter((value, index, self) =>
index === self.findIndex((t) => (
t.place === value.place && t.name === value.name
))
)
A more generic solution would be:
const uniqueArray = obj.arr.filter((value, index) => {
const _value = JSON.stringify(value);
return index === obj.arr.findIndex(obj => {
return JSON.stringify(obj) === _value;
});
});
Using the above property strategy instead of JSON.stringify
:
const isPropValuesEqual = (subject, target, propNames) =>
propNames.every(propName => subject[propName] === target[propName]);
const getUniqueItemsByProperties = (items, propNames) =>
items.filter((item, index, array) =>
index === array.findIndex(foundItem => isPropValuesEqual(foundItem, item, propNames))
);
You can add a wrapper if you want the propNames
property to be either an array or a value:
const getUniqueItemsByProperties = (items, propNames) => {
const propNamesArray = Array.from(propNames);
return items.filter((item, index, array) =>
index === array.findIndex(foundItem => isPropValuesEqual(foundItem, item, propNamesArray))
);
};
allowing both getUniqueItemsByProperties('a')
and getUniqueItemsByProperties(['a']);
Explanation
- Start by understanding the two methods used:
- Next take your idea of what makes your two objects equal and keep that in mind.
- We can detect something as a duplicate, if it satisfies the criterion that we have just thought of, but it's position is not at the first instance of an object with the criterion.
- Therefore we can use the above criterion to determine if something is a duplicate.
Solution 2 - Javascript
One liners with filter ( Preserves order )
Find unique id
's in an array.
arr.filter((v,i,a)=>a.findIndex(v2=>(v2.id===v.id))===i)
If the order is not important, map solutions will be faster: Solution with map
Unique by multiple properties ( place
and name
)
arr.filter((v,i,a)=>a.findIndex(v2=>['place','name'].every(k=>v2[k] ===v[k]))===i)
Unique by all properties (This will be slow for large arrays)
arr.filter((v,i,a)=>a.findIndex(v2=>(JSON.stringify(v2) === JSON.stringify(v)))===i)
Keep the last occurrence by replacing findIndex
with findLastIndex
.
arr.filter((v,i,a)=>a.findLastIndex(v2=>(v2.place === v.place))===i)
Solution 3 - Javascript
Using ES6+ in a single line you can get a unique list of objects by key:
const unique = [...new Map(arr.map((item, key) => [item[key], item])).values()]
It can be put into a function:
function getUniqueListBy(arr, key) {
return [...new Map(arr.map(item => [item[key], item])).values()]
}
Here is a working example:
const arr = [
{place: "here", name: "x", other: "other stuff1" },
{place: "there", name: "x", other: "other stuff2" },
{place: "here", name: "y", other: "other stuff4" },
{place: "here", name: "z", other: "other stuff5" }
]
function getUniqueListBy(arr, key) {
return [...new Map(arr.map(item => [item[key], item])).values()]
}
const arr1 = getUniqueListBy(arr, 'place')
console.log("Unique by place")
console.log(JSON.stringify(arr1))
console.log("\nUnique by name")
const arr2 = getUniqueListBy(arr, 'name')
console.log(JSON.stringify(arr2))
How does it work
First the array is remapped in a way that it can be used as an input for a Map.
> arr.map(item => [item[key], item]);
which means each item of the array will be transformed in another array with 2 elements; the selected key as first element and the entire initial item as second element, this is called an entry (ex. array entries, map entries). And here is the official doc with an example showing how to add array entries in Map constructor.
Example when key is place:
[["here", {place: "here", name: "x", other: "other stuff1" }], ...]
Secondly, we pass this modified array to the Map constructor and here is the magic happening. Map will eliminate the duplicate keys values, keeping only last inserted value of the same key. Note: Map keeps the order of insertion. (check difference between Map and object)
> new Map(entry array just mapped above)
Third we use the map values to retrieve the original items, but this time without duplicates.
> new Map(mappedArr).values()
And last one is to add those values into a fresh new array so that it can look as the initial structure and return that:
> return [...new Map(mappedArr).values()]
Solution 4 - Javascript
A primitive method would be:
const obj = {};
for (let i = 0, len = things.thing.length; i < len; i++) {
obj[things.thing[i]['place']] = things.thing[i];
}
things.thing = new Array();
for (const key in obj) {
things.thing.push(obj[key]);
}
Solution 5 - Javascript
Simple and performant solution with a better runtime than the 70+ answers that already exist:
const ids = array.map(o => o.id)
const filtered = array.filter(({id}, index) => !ids.includes(id, index + 1))
Example:
const arr = [{id: 1, name: 'one'}, {id: 2, name: 'two'}, {id: 1, name: 'one'}]
const ids = arr.map(o => o.id)
const filtered = arr.filter(({id}, index) => !ids.includes(id, index + 1))
console.log(filtered)
How it works:
Array.filter()
removes all duplicate objects by checking if the previously mapped id-array includes the current id ({id}
destructs the object into only its id). To only filter out actual duplicates, it is using Array.includes()
's second parameter fromIndex
with index + 1
which will ignore the current object and all previous.
Since every iteration of the filter
callback method will only search the array beginning at the current index + 1, this also dramatically reduces the runtime because only objects not previously filtered get checked.
This obviously also works for any other key that is not called id
, multiple or even all keys.
Solution 6 - Javascript
If you can use Javascript libraries such as underscore or lodash, I recommend having a look at _.uniq
function in their libraries. From lodash
:
_.uniq(array, [isSorted=false], [callback=_.identity], [thisArg])
Basically, you pass in the array that in here is an object literal and you pass in the attribute that you want to remove duplicates with in the original data array, like this:
var data = [{'name': 'Amir', 'surname': 'Rahnama'}, {'name': 'Amir', 'surname': 'Stevens'}];
var non_duplidated_data = _.uniq(data, 'name');
UPDATE: Lodash now has introduced a .uniqBy
as well.
Solution 7 - Javascript
I had this exact same requirement, to remove duplicate objects in a array, based on duplicates on a single field. I found the code here: Javascript: Remove Duplicates from Array of Objects
So in my example, I'm removing any object from the array that has a duplicate licenseNum string value.
var arrayWithDuplicates = [
{"type":"LICENSE", "licenseNum": "12345", state:"NV"},
{"type":"LICENSE", "licenseNum": "A7846", state:"CA"},
{"type":"LICENSE", "licenseNum": "12345", state:"OR"},
{"type":"LICENSE", "licenseNum": "10849", state:"CA"},
{"type":"LICENSE", "licenseNum": "B7037", state:"WA"},
{"type":"LICENSE", "licenseNum": "12345", state:"NM"}
];
function removeDuplicates(originalArray, prop) {
var newArray = [];
var lookupObject = {};
for(var i in originalArray) {
lookupObject[originalArray[i][prop]] = originalArray[i];
}
for(i in lookupObject) {
newArray.push(lookupObject[i]);
}
return newArray;
}
var uniqueArray = removeDuplicates(arrayWithDuplicates, "licenseNum");
console.log("uniqueArray is: " + JSON.stringify(uniqueArray));
The results:
uniqueArray is:
[{"type":"LICENSE","licenseNum":"10849","state":"CA"},{"type":"LICENSE","licenseNum":"12345","state":"NM"},{"type":"LICENSE","licenseNum":"A7846","state":"CA"},{"type":"LICENSE","licenseNum":"B7037","state":"WA"}]
Solution 8 - Javascript
One liner using Set
var things = new Object();
things.thing = new Array();
things.thing.push({place:"here",name:"stuff"});
things.thing.push({place:"there",name:"morestuff"});
things.thing.push({place:"there",name:"morestuff"});
// assign things.thing to myData for brevity
var myData = things.thing;
things.thing = Array.from(new Set(myData.map(JSON.stringify))).map(JSON.parse);
console.log(things.thing)
Explanation:
new Set(myData.map(JSON.stringify))
creates a Set object using the stringified myData elements.- Set object will ensure that every element is unique.
- Then I create an array based on the elements of the created set using Array.from.
- Finally, I use JSON.parse to convert stringified element back to an object.
Solution 9 - Javascript
ES6 one liner is here
let arr = [
{id:1,name:"sravan ganji"},
{id:2,name:"pinky"},
{id:4,name:"mammu"},
{id:3,name:"avy"},
{id:3,name:"rashni"},
];
console.log(Object.values(arr.reduce((acc,cur)=>Object.assign(acc,{[cur.id]:cur}),{})))
Solution 10 - Javascript
The simplest way is use filter
:
var uniq = {};
var arr = [{"id":"1"},{"id":"1"},{"id":"2"}];
var arrFiltered = arr.filter(obj => !uniq[obj.id] && (uniq[obj.id] = true));
console.log('arrFiltered', arrFiltered);
Solution 11 - Javascript
Here's another option to do it using Array iterating methods if you need comparison only by one field of an object:
function uniq(a, param){
return a.filter(function(item, pos, array){
return array.map(function(mapItem){ return mapItem[param]; }).indexOf(item[param]) === pos;
})
}
uniq(things.thing, 'place');
Solution 12 - Javascript
This is a generic way of doing this: you pass in a function that tests whether two elements of an array are considered equal. In this case, it compares the values of the name
and place
properties of the two objects being compared.
ES5 answer
function removeDuplicates(arr, equals) {
var originalArr = arr.slice(0);
var i, len, val;
arr.length = 0;
for (i = 0, len = originalArr.length; i < len; ++i) {
val = originalArr[i];
if (!arr.some(function(item) { return equals(item, val); })) {
arr.push(val);
}
}
}
function thingsEqual(thing1, thing2) {
return thing1.place === thing2.place
&& thing1.name === thing2.name;
}
var things = [
{place:"here",name:"stuff"},
{place:"there",name:"morestuff"},
{place:"there",name:"morestuff"}
];
removeDuplicates(things, thingsEqual);
console.log(things);
Original ES3 answer
function arrayContains(arr, val, equals) {
var i = arr.length;
while (i--) {
if ( equals(arr[i], val) ) {
return true;
}
}
return false;
}
function removeDuplicates(arr, equals) {
var originalArr = arr.slice(0);
var i, len, j, val;
arr.length = 0;
for (i = 0, len = originalArr.length; i < len; ++i) {
val = originalArr[i];
if (!arrayContains(arr, val, equals)) {
arr.push(val);
}
}
}
function thingsEqual(thing1, thing2) {
return thing1.place === thing2.place
&& thing1.name === thing2.name;
}
removeDuplicates(things.thing, thingsEqual);
Solution 13 - Javascript
If you can wait to eliminate the duplicates until after all the additions, the typical approach is to first sort the array and then eliminate duplicates. The sorting avoids the N * N approach of scanning the array for each element as you walk through them.
The "eliminate duplicates" function is usually called unique or uniq. Some existing implementations may combine the two steps, e.g., prototype's uniq
This post has few ideas to try (and some to avoid :-) ) if your library doesn't already have one! Personally I find this one the most straight forward:
function unique(a){
a.sort();
for(var i = 1; i < a.length; ){
if(a[i-1] == a[i]){
a.splice(i, 1);
} else {
i++;
}
}
return a;
}
// Provide your own comparison
function unique(a, compareFunc){
a.sort( compareFunc );
for(var i = 1; i < a.length; ){
if( compareFunc(a[i-1], a[i]) === 0){
a.splice(i, 1);
} else {
i++;
}
}
return a;
}
Solution 14 - Javascript
I think the best approach is using reduce and Map object. This is a single line solution.
const data = [
{id: 1, name: 'David'},
{id: 2, name: 'Mark'},
{id: 2, name: 'Lora'},
{id: 4, name: 'Tyler'},
{id: 4, name: 'Donald'},
{id: 5, name: 'Adrian'},
{id: 6, name: 'Michael'}
]
const uniqueData = [...data.reduce((map, obj) => map.set(obj.id, obj), new Map()).values()];
console.log(uniqueData)
/*
in `map.set(obj.id, obj)`
'obj.id' is key. (don't worry. we'll get only values using the .values() method)
'obj' is whole object.
*/
Solution 15 - Javascript
To add one more to the list. Using ES6 and Array.reduce
with Array.find
.
In this example filtering objects based on a guid
property.
let filtered = array.reduce((accumulator, current) => {
if (! accumulator.find(({guid}) => guid === current.guid)) {
accumulator.push(current);
}
return accumulator;
}, []);
Extending this one to allow selection of a property and compress it into a one liner:
const uniqify = (array, key) => array.reduce((prev, curr) => prev.find(a => a[key] === curr[key]) ? prev : prev.push(curr) && prev, []);
To use it pass an array of objects and the name of the key you wish to de-dupe on as a string value:
const result = uniqify(myArrayOfObjects, 'guid')
Solution 16 - Javascript
Dang, kids, let's crush this thing down, why don't we?
let uniqIds = {}, source = [{id:'a'},{id:'b'},{id:'c'},{id:'b'},{id:'a'},{id:'d'}]; let filtered = source.filter(obj => !uniqIds[obj.id] && (uniqIds[obj.id] = true)); console.log(filtered); // EXPECTED: [{id:'a'},{id:'b'},{id:'c'},{id:'d'}];
Solution 17 - Javascript
You could also use a Map
:
const dedupThings = Array.from(things.thing.reduce((m, t) => m.set(t.place, t), new Map()).values());
Full sample:
const things = new Object();
things.thing = new Array();
things.thing.push({place:"here",name:"stuff"});
things.thing.push({place:"there",name:"morestuff"});
things.thing.push({place:"there",name:"morestuff"});
const dedupThings = Array.from(things.thing.reduce((m, t) => m.set(t.place, t), new Map()).values());
console.log(JSON.stringify(dedupThings, null, 4));
Result:
[ { "place": "here", "name": "stuff" }, { "place": "there", "name": "morestuff" }]
Solution 18 - Javascript
A TypeScript solution
This will remove duplicate objects and also preserve the types of the objects.
function removeDuplicateObjects(array: any[]) {
return [...new Set(array.map(s => JSON.stringify(s)))]
.map(s => JSON.parse(s));
}
Solution 19 - Javascript
One liners with Map ( High performance, Does not preserve order )
Find unique id
's in array arr
.
const arrUniq = [...new Map(arr.map(v => [v.id, v])).values()]
If the order is important check out the solution with filter: Solution with filter
Unique by multiple properties ( place
and name
) in array arr
const arrUniq = [...new Map(arr.map(v => [JSON.stringify([v.place,v.name]), v])).values()]
Unique by all properties in array arr
const arrUniq = [...new Map(arr.map(v => [JSON.stringify(v), v])).values()]
Keep the first occurrence in array arr
const arrUniq = [...new Map(arr.slice().reverse().map(v => [v.id, v])).values()].reverse()
Solution 20 - Javascript
Considering lodash.uniqWith
const objects = [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }, { 'x': 1, 'y': 2 }];
_.uniqWith(objects, _.isEqual);
// => [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }]
Solution 21 - Javascript
let myData = [{place:"here",name:"stuff"},
{place:"there",name:"morestuff"},
{place:"there",name:"morestuff"}];
let q = [...new Map(myData.map(obj => [JSON.stringify(obj), obj])).values()];
console.log(q)
One-liner using ES6 and new Map()
.
// assign things.thing to myData
let myData = things.thing;
[...new Map(myData.map(obj => [JSON.stringify(obj), obj])).values()];
Details:-
-
Doing
.map()
on the data list and converting each individual object into a[key, value]
pair array(length =2), the first element(key) would be thestringified
version of the object and second(value) would be anobject
itself. -
Adding above created array list to
new Map()
would have the key asstringified
object and any same key addition would result in overriding the already existing key. -
Using
.values()
would give MapIterator with all values in a Map (obj
in our case) -
Finally,
spread ...
operator to give new Array with values from the above step.
Solution 22 - Javascript
const things = [
{place:"here",name:"stuff"},
{place:"there",name:"morestuff"},
{place:"there",name:"morestuff"}
];
const filteredArr = things.reduce((thing, current) => { const x = thing.find(item => item.place === current.place); if (!x) { return thing.concat([current]); } else { return thing; } }, []); console.log(filteredArr)
Solution Via Set
Object | According to the data type
const seen = new Set();
const things = [
{place:"here",name:"stuff"},
{place:"there",name:"morestuff"},
{place:"there",name:"morestuff"}
];
const filteredArr = things.filter(el => {
const duplicate = seen.has(el.place);
seen.add(el.place);
return !duplicate;
});
console.log(filteredArr)
Set
Object Feature
Each value in the Set Object has to be unique, the value equality will be checked
The Purpose of Set object storing unique values according to the Data type , whether primitive values or object references.it has very useful four Instance methods add
, clear
, has
& delete
.
Unique & data Type feature:..
add
method
it's push unique data into collection by default also preserve data type .. that means it prevent to push duplicate item into collection also it will check data type by default...
has
method
sometime needs to check data item exist into the collection and . it's handy method for the collection to cheek unique id or item and data type..
delete
method
it will remove specific item from the collection by identifying data type..
clear
method
it will remove all collection items from one specific variable and set as empty object
Set
object has also Iteration methods & more feature..
Better Read from Here : Set - JavaScript | MDN
Solution 23 - Javascript
removeDuplicates()
takes in an array of objects and returns a new array without any duplicate objects (based on the id property).
const allTests = [
{name: 'Test1', id: '1'},
{name: 'Test3', id: '3'},
{name: 'Test2', id: '2'},
{name: 'Test2', id: '2'},
{name: 'Test3', id: '3'}
];
function removeDuplicates(array) {
let uniq = {};
return array.filter(obj => !uniq[obj.id] && (uniq[obj.id] = true))
}
removeDuplicates(allTests);
Expected outcome:
[
{name: 'Test1', id: '1'},
{name: 'Test3', id: '3'},
{name: 'Test2', id: '2'}
];
First, we set the value of variable uniq to an empty object.
Next, we filter through the array of objects. Filter creates a new array with all elements that pass the test implemented by the provided function.
return array.filter(obj => !uniq[obj.id] && (uniq[obj.id] = true));
Above, we use the short-circuiting functionality of &&. If the left side of the && evaluates to true, then it returns the value on the right of the &&. If the left side is false, it returns what is on the left side of the &&.
For each object(obj) we check uniq for a property named the value of obj.id (In this case, on the first iteration it would check for the property '1'.) We want the opposite of what it returns (either true or false) which is why we use the ! in !uniq[obj.id]
. If uniq has the id property already, it returns true which evaluates to false (!) telling the filter function NOT to add that obj. However, if it does not find the obj.id property, it returns false which then evaluates to true (!) and returns everything to the right of the &&, or (uniq[obj.id] = true). This is a truthy value, telling the filter method to add that obj to the returned array, and it also adds the property {1: true} to uniq. This ensures that any other obj instance with that same id will not be added again.
Solution 24 - Javascript
This way works well for me:
function arrayUnique(arr, uniqueKey) {
const flagList = new Set()
return arr.filter(function(item) {
if (!flagList.has(item[uniqueKey])) {
flagList.add(item[uniqueKey])
return true
}
})
}
const data = [
{
name: 'Kyle',
occupation: 'Fashion Designer'
},
{
name: 'Kyle',
occupation: 'Fashion Designer'
},
{
name: 'Emily',
occupation: 'Web Designer'
},
{
name: 'Melissa',
occupation: 'Fashion Designer'
},
{
name: 'Tom',
occupation: 'Web Developer'
},
{
name: 'Tom',
occupation: 'Web Developer'
}
]
console.table(arrayUnique(data, 'name'))// work well
printout
┌─────────┬───────────┬────────────────────┐
│ (index) │ name │ occupation │
├─────────┼───────────┼────────────────────┤
│ 0 │ 'Kyle' │ 'Fashion Designer' │
│ 1 │ 'Emily' │ 'Web Designer' │
│ 2 │ 'Melissa' │ 'Fashion Designer' │
│ 3 │ 'Tom' │ 'Web Developer' │
└─────────┴───────────┴────────────────────┘
ES5:
function arrayUnique(arr, uniqueKey) {
const flagList = []
return arr.filter(function(item) {
if (flagList.indexOf(item[uniqueKey]) === -1) {
flagList.push(item[uniqueKey])
return true
}
})
}
These two ways are simpler and more understandable.
Solution 25 - Javascript
Here is a solution for ES6 where you only want to keep the last item. This solution is functional and Airbnb style compliant.
const things = {
thing: [
{ place: 'here', name: 'stuff' },
{ place: 'there', name: 'morestuff1' },
{ place: 'there', name: 'morestuff2' },
],
};
const removeDuplicates = (array, key) => {
return array.reduce((arr, item) => {
const removed = arr.filter(i => i[key] !== item[key]);
return [...removed, item];
}, []);
};
console.log(removeDuplicates(things.thing, 'place'));
// > [{ place: 'here', name: 'stuff' }, { place: 'there', name: 'morestuff2' }]
Solution 26 - Javascript
Fast (less runtime) and type-safe answer for lazy Typescript developers:
export const uniqueBy = <T>( uniqueKey: keyof T, objects: T[]): T[] => {
const ids = objects.map(object => object[uniqueKey]);
return objects.filter((object, index) => !ids.includes(object[uniqueKey], index + 1));
}
Solution 27 - Javascript
I know there is a ton of answers in this question already, but bear with me...
Some of the objects in your array may have additional properties that you are not interested in, or you simply want to find the unique objects considering only a subset of the properties.
Consider the array below. Say you want to find the unique objects in this array considering only propOne
and propTwo
, and ignore any other properties that may be there.
The expected result should include only the first and last objects. So here goes the code:
const array = [{
propOne: 'a',
propTwo: 'b',
propThree: 'I have no part in this...'
},
{
propOne: 'a',
propTwo: 'b',
someOtherProperty: 'no one cares about this...'
},
{
propOne: 'x',
propTwo: 'y',
yetAnotherJunk: 'I am valueless really',
noOneHasThis: 'I have something no one has'
}];
const uniques = [...new Set(
array.map(x => JSON.stringify(((o) => ({
propOne: o.propOne,
propTwo: o.propTwo
}))(x))))
].map(JSON.parse);
console.log(uniques);
Solution 28 - Javascript
Another option would be to create a custom indexOf function, which compares the values of your chosen property for each object and wrap this in a reduce function.
var uniq = redundant_array.reduce(function(a,b){
function indexOfProperty (a, b){
for (var i=0;i<a.length;i++){
if(a[i].property == b.property){
return i;
}
}
return -1;
}
if (indexOfProperty(a,b) < 0 ) a.push(b);
return a;
},[]);
Solution 29 - Javascript
Here I found a simple solution for removing duplicates from an array of objects using reduce method. I am filtering elements based on the position key of an object
const med = [
{name: 'name1', position: 'left'},
{name: 'name2', position: 'right'},
{name: 'name3', position: 'left'},
{name: 'name4', position: 'right'},
{name: 'name5', position: 'left'},
{name: 'name6', position: 'left1'}
]
const arr = [];
med.reduce((acc, curr) => {
if(acc.indexOf(curr.position) === -1) {
acc.push(curr.position);
arr.push(curr);
}
return acc;
}, [])
console.log(arr)
Solution 30 - Javascript
Continuing exploring ES6 ways of removing duplicates from array of objects: setting thisArg
argument of Array.prototype.filter
to new Set
provides a decent alternative:
const things = [
{place:"here",name:"stuff"},
{place:"there",name:"morestuff"},
{place:"there",name:"morestuff"}
];
const filtered = things.filter(function({place, name}) {
const key =`${place}${name}`;
return !this.has(key) && this.add(key);
}, new Set);
console.log(filtered);
However, it will not work with arrow functions () =>
, as this
is bound to their lexical scope.
Solution 31 - Javascript
es6 magic in one line... readable at that!
// returns the union of two arrays where duplicate objects with the same 'prop' are removed
const removeDuplicatesWith = (a, b, prop) => {
a.filter(x => !b.find(y => x[prop] === y[prop]));
};
Solution 32 - Javascript
This will remove the duplicate object without passing any key.
uniqueArray = a => [...new Set(a.map(o => JSON.stringify(o)))].map(s => JSON.parse(s));
var objects = [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }, { 'x': 1, 'y': 2 }];
var unique = uniqueArray(objects);
console.log('Original Object',objects);
console.log('Unique',unique);
uniqueArray = a => [...new Set(a.map(o => JSON.stringify(o)))].map(s => JSON.parse(s));
var objects = [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }, { 'x': 1, 'y': 2 }];
var unique = uniqueArray(objects);
console.log(objects);
console.log(unique);
Solution 33 - Javascript
If you strictly want to remove duplicates based on one property, you can reduce
the array into and object based on the place
property, since the object can only have unique keys, you can then just get the values
to get back to an array:
const unique = Object.values(things.thing.reduce((o, t) => ({ ...o, [t.place]: t }), {}))
Solution 34 - Javascript
I believe a combination of reduce
with JSON.stringify
to perfectly compare Objects and selectively adding those who are not already in the accumulator is an elegant way.
Keep in mind that JSON.stringify
might become a performance issue in extreme cases where the array has many Objects and they are complex, BUT for majority of the time, this is the shortest way to go IMHO.
var collection= [{a:1},{a:2},{a:1},{a:3}]
var filtered = collection.reduce((filtered, item) => {
if( !filtered.some(filteredItem => JSON.stringify(filteredItem) == JSON.stringify(item)) )
filtered.push(item)
return filtered
}, [])
console.log(filtered)
Another way of writing the same (but less efficient):
collection.reduce((filtered, item) =>
filtered.some(filteredItem =>
JSON.stringify(filteredItem ) == JSON.stringify(item))
? filtered
: [...filtered, item]
, [])
Solution 35 - Javascript
npm i lodash
let non_duplicated_data = _.uniqBy(pendingDeposits, v => [v.stellarAccount, v.externalTransactionId].join());
Solution 36 - Javascript
The problem can be simplified to removing duplicates from the thing
array.
You can implement a faster O(n) solution (assuming native key lookup is negligible) by using an object to both maintain unique criteria as keys and storing associated values.
Basically, the idea is to store all objects by their unique key, so that duplicates overwrite themselves:
const thing = [{ place: "here", name:"stuff" }, { place: "there", name:"morestuff" }, { place: "there", name:"morestuff" } ]
const uniques = {}
for (const t of thing) {
const key = t.place + '$' + t.name // Or whatever string criteria you want, which can be generified as Object.keys(t).join("$")
uniques[key] = t // Last duplicate wins
}
const uniqueThing = Object.values(uniques)
console.log(uniqueThing)
Solution 37 - Javascript
const objectsMap = new Map();
const placesName = [
{ place: "here", name: "stuff" },
{ place: "there", name: "morestuff" },
{ place: "there", name: "morestuff" },
];
placesName.forEach((object) => {
objectsMap.set(object.place, object);
});
console.log(objectsMap);
Solution 38 - Javascript
This solution worked best for me , by utilising Array.from Method, And also its shorter and readable.
let person = [
{name: "john"},
{name: "jane"},
{name: "imelda"},
{name: "john"},
{name: "jane"}
];
const data = Array.from(new Set(person.map(JSON.stringify))).map(JSON.parse);
console.log(data);
Solution 39 - Javascript
let data = [
{
'name': 'Amir',
'surname': 'Rahnama'
},
{
'name': 'Amir',
'surname': 'Stevens'
}
];
let non_duplicated_data = _.uniqBy(data, 'name');
Solution 40 - Javascript
If you don't mind your unique array being sorted afterwards, this would be an efficient solution:
things.thing
.sort(((a, b) => a.place < b.place)
.filter((current, index, array) =>
index === 0 || current.place !== array[index - 1].place)
This way, you only have to compare the current element with the previous element in the array. Sorting once before filtering (O(n*log(n))
) is cheaper than searching for a duplicate in the entire array for every array element (O(n²)
).
Solution 41 - Javascript
str =[
{"item_id":1},
{"item_id":2},
{"item_id":2}
]
obj =[]
for (x in str){
if(check(str[x].item_id)){
obj.push(str[x])
}
}
function check(id){
flag=0
for (y in obj){
if(obj[y].item_id === id){
flag =1
}
}
if(flag ==0) return true
else return false
}
console.log(obj)
str is an array of objects. There exists objects having same value (here a small example, there are two objects having same item_id as 2). check(id) is a function that checks if any object having same item_id exists or not. if it exists return false otherwise return true. According to that result, push the object into a new array obj
The output of the above code is
[{"item_id":1},{"item_id":2}]
Solution 42 - Javascript
Have you heard of Lodash library? I recommend you this utility, when you don't really want to apply your logic to the code, and use already present code which is optimised and reliable.
Consider making an array like this
things.thing.push({place:"utopia",name:"unicorn"});
things.thing.push({place:"jade_palace",name:"po"});
things.thing.push({place:"jade_palace",name:"tigress"});
things.thing.push({place:"utopia",name:"flying_reindeer"});
things.thing.push({place:"panda_village",name:"po"});
Note that if you want to keep one attribute unique, you may very well do that by using lodash library. Here, you may use _.uniqBy
> .uniqBy(array, [iteratee=.identity])
This method is like _.uniq (which returns a duplicate-free version of an array, in which only the first occurrence of each element is kept) except that it accepts iteratee which is invoked for each element in array to generate the criterion by which uniqueness is computed.
So, for example, if you want to return an array having unique attribute of 'place'
> _.uniqBy(things.thing, 'place')
Similarly, if you want unique attribute as 'name' > _.uniqBy(things.thing, 'name')
Hope this helps.
Cheers!
Solution 43 - Javascript
- This solution is generic for any kind of object and checks for every
(key, value)
of theObject
in the array. - Using an temporary object as a hash table to see if the entire
Object
was ever present as a key. - If the string representation of the
Object
is found then that item is removed from the array.
var arrOfDup = [{'id':123, 'name':'name', 'desc':'some desc'},
{'id':125, 'name':'another name', 'desc':'another desc'},
{'id':123, 'name':'name', 'desc':'some desc'},
{'id':125, 'name':'another name', 'desc':'another desc'},
{'id':125, 'name':'another name', 'desc':'another desc'}];
function removeDupes(dupeArray){
let temp = {};
let tempArray = JSON.parse(JSON.stringify(dupeArray));
dupeArray.forEach((item, pos) => {
if(temp[JSON.stringify(item)]){
tempArray.pop();
}else{
temp[JSON.stringify(item)] = item;
}
});
return tempArray;
}
arrOfDup = removeDupes(arrOfDup);
arrOfDup.forEach((item, pos) => {
console.log(`item in array at position ${pos} is ${JSON.stringify(item)}`);
});
Solution 44 - Javascript
const uniqueElements = (arr, fn) => arr.reduce((acc, v) => {
if (!acc.some(x => fn(v, x))) { acc.push(v); }
return acc;
}, []);
const stuff = [
{place:"here",name:"stuff"},
{place:"there",name:"morestuff"},
{place:"there",name:"morestuff"},
];
const unique = uniqueElements(stuff, (a,b) => a.place === b.place && a.name === b.name );
//console.log( unique );
[{ "place": "here", "name": "stuff" }, { "place": "there", "name": "morestuff"}]
Solution 45 - Javascript
You can convert the array objects into strings so they can be compared, add the strings to a Set so the comparable duplicates will be automatically removed and then convert each of the strings back into objects.
It might not be as performant as other answers, but it's readable.
const things = {};
things.thing = [];
things.thing.push({place:"here",name:"stuff"});
things.thing.push({place:"there",name:"morestuff"});
things.thing.push({place:"there",name:"morestuff"});
const uniqueArray = (arr) => {
const stringifiedArray = arr.map((item) => JSON.stringify(item));
const set = new Set(stringifiedArray);
return Array.from(set).map((item) => JSON.parse(item));
}
const uniqueThings = uniqueArray(things.thing);
console.log(uniqueThings);
Solution 46 - Javascript
Simple solution with ES6 'reduce' and 'find' array helper methods
Works efficiently and perfectly fine!
"use strict";
var things = new Object();
things.thing = new Array();
things.thing.push({
place: "here",
name: "stuff"
});
things.thing.push({
place: "there",
name: "morestuff"
});
things.thing.push({
place: "there",
name: "morestuff"
});
// the logic is here
function removeDup(something) {
return something.thing.reduce(function (prev, ele) {
var found = prev.find(function (fele) {
return ele.place === fele.place && ele.name === fele.name;
});
if (!found) {
prev.push(ele);
}
return prev;
}, []);
}
console.log(removeDup(things));
Solution 47 - Javascript
For a readable and a simple solution searcher, her is my version:
function removeDupplicationsFromArrayByProp(originalArray, prop) {
let results = {};
for(let i=0; i<originalArray.length;i++){
results[originalArray[i][prop]] = originalArray[i];
}
return Object.values(results);
}
Solution 48 - Javascript
My two cents here. If you know the properties are in the same order, you can stringify the elements and remove dupes from the array and parse the array again. Something like this:
var things = new Object();
things.thing = new Array();
things.thing.push({place:"here",name:"stuff"});
things.thing.push({place:"there",name:"morestuff"});
things.thing.push({place:"there",name:"morestuff"});
let stringified = things.thing.map(i=>JSON.stringify(i));
let unique = stringified.filter((k, idx)=> stringified.indexOf(k) === idx)
.map(j=> JSON.parse(j))
console.log(unique);
Solution 49 - Javascript
-
Removing Duplicates From Array Of Objects in react js (Working perfectly)
let optionList = []; var dataArr = this.state.itemArray.map(item => { return [item.name, item] }); var maparr = new Map(dataArr); var results = [...maparr.values()]; if (results.length > 0) { results.map(data => { if (data.lead_owner !== null) { optionList.push({ label: data.name, value: data.name }); } return true; }); } console.log(optionList)
Solution 50 - Javascript
You could use Set
along with Filter
method to accomplish this,
var arrObj = [{
a: 1,
b: 2
}, {
a: 1,
b: 1
}, {
a: 1,
b: 2
}];
var duplicateRemover = new Set();
var distinctArrObj = arrObj.filter((obj) => {
if (duplicateRemover.has(JSON.stringify(obj))) return false;
duplicateRemover.add(JSON.stringify(obj));
return true;
});
console.log(distinctArrObj);
Set
is a unique collection of primitive types, thus, won't work directly on objects, however JSON.stringify
will convert it into a primitive type ie. String
thus, we can filter.
If you want to remove duplicates based on only some particular key, for eg. key
, you could replace JSON.stringify(obj)
with obj.key
Solution 51 - Javascript
This is a single loop approach with a Set
and some closures to prevent using declared variables outside function declarations and to get a short appearance.
const array = [{ place: "here", name: "stuff", n: 1 }, { place: "there", name: "morestuff", n: 2 }, { place: "there", name: "morestuff", n: 3 }], keys = ['place', 'name'], unique = array.filter( (s => o => (v => !s.has(v) && s.add(v))(keys.map(k => o[k]).join('|'))) (new Set) );
console.log(unique);
.as-console-wrapper { max-height: 100% !important; top: 0; }
Solution 52 - Javascript
Here is another technique to find number of duplicate and and remove it easily from you data object. "dupsCount" is number of duplicate files count. sort your data first then remove. it will gives you fastest duplication remove.
dataArray.sort(function (a, b) {
var textA = a.name.toUpperCase();
var textB = b.name.toUpperCase();
return (textA < textB) ? -1 : (textA > textB) ? 1 : 0;
});
for (var i = 0; i < dataArray.length - 1; ) {
if (dataArray[i].name == dataArray[i + 1].name) {
dupsCount++;
dataArray.splice(i, 1);
} else {
i++;
}
}
Solution 53 - Javascript
If you need an unique array based on multiple properties in the object you can do this with map and combining the properties of the object.
var hash = array.map(function(element){
var string = ''
for (var key in element){
string += element[key]
}
return string
})
array = array.filter(function(element, index){
var string = ''
for (var key in element){
string += element[key]
}
return hash.indexOf(string) == index
})
Solution 54 - Javascript
Generic for any array of objects:
/**
* Remove duplicated values without losing information
*/
const removeValues = (items, key) => {
let tmp = {};
items.forEach(item => {
tmp[item[key]] = (!tmp[item[key]]) ? item : Object.assign(tmp[item[key]], item);
});
items = [];
Object.keys(tmp).forEach(key => items.push(tmp[key]));
return items;
}
Hope it could help to anyone.
Solution 55 - Javascript
Another way would be to use reduce function and have a new array to be the accumulator. If there is already a thing
with the same name in the accumulator array then don't add it there.
let list = things.thing;
list = list.reduce((accumulator, thing) => {
if (!accumulator.filter((duplicate) => thing.name === duplicate.name)[0]) {
accumulator.push(thing);
}
return accumulator;
}, []);
thing.things = list;
I'm adding this answer, because I couldn't find nice, readable es6 solution (I use babel to handle arrow functions) that's compatible with Internet Explorer 11. The problem is IE11 doesn't have Map.values()
or Set.values()
without polyfill. For the same reason I used filter()[0]
to get first element instead of find()
.
Solution 56 - Javascript
var testArray= ['a','b','c','d','e','b','c','d'];
function removeDuplicatesFromArray(arr){
var obj={};
var uniqueArr=[];
for(var i=0;i<arr.length;i++){
if(!obj.hasOwnProperty(arr[i])){
obj[arr[i]] = arr[i];
uniqueArr.push(arr[i]);
}
}
return uniqueArr;
}
var newArr = removeDuplicatesFromArray(testArray);
console.log(newArr);
Output:- [ 'a', 'b', 'c', 'd', 'e' ]
Solution 57 - Javascript
If you don't want to specify a list of properties:
function removeDuplicates(myArr) {
var props = Object.keys(myArr[0])
return myArr.filter((item, index, self) =>
index === self.findIndex((t) => (
props.every(prop => {
return t[prop] === item[prop]
})
))
)
}
OBS! Not compatible with IE11.
Solution 58 - Javascript
here is my solution, it searches for duplicates based on object.prop
and when it finds a duplicate object it replaces its value
in array1 with array2 value
function mergeSecondArrayIntoFirstArrayByProperty(array1, array2) {
for (var i = 0; i < array2.length; i++) {
var found = false;
for (var j = 0; j < array1.length; j++) {
if (array2[i].prop === array1[j].prop) { // if item exist in array1
array1[j] = array2[i]; // replace it in array1 with array2 value
found = true;
}
}
if (!found) // if item in array2 not found in array1, add it to array1
array1.push(array2[i]);
}
return array1;
}
Solution 59 - Javascript
What about this:
function dedupe(arr, compFn){
let res = [];
if (!compFn) compFn = (a, b) => { return a === b };
arr.map(a => {if(!res.find(b => compFn(a, b))) res.push(a)});
return res;
}
Solution 60 - Javascript
If you find yourself needing to remove duplicate objects from arrays based on particular fields frequently, it might be worth creating a distinct(array, predicate)
function that you can import from anywhere in your project. This would look like
const things = [{place:"here",name:"stuff"}, ...];
const distinctThings = distinct(things, thing => thing.place);
The distinct function can use any of the implementations given in the many good answers above. The easiest one uses findIndex
:
const distinct = (items, predicate) => items.filter((uniqueItem, index) =>
items.findIndex(item =>
predicate(item) === predicate(uniqueItem)) === index);
Solution 61 - Javascript
You can use Object.values() combined with Array.prototype.reduce():
const things = new Object();
things.thing = new Array();
things.thing.push({place:"here",name:"stuff"});
things.thing.push({place:"there",name:"morestuff"});
things.thing.push({place:"there",name:"morestuff"});
const result = Object.values(things.thing.reduce((a, c) => (a[`${c.place}${c.name}`] = c, a), {}));
console.log(result);
.as-console-wrapper { max-height: 100% !important; top: 0; }
Solution 62 - Javascript
Make Something simple. Fancy is good but unreadable code is useless. Enjoy :-)
var a = [
{
executiveId: 6873702,
largePhotoCircle: null,
name: "John A. Cuomo",
photoURL: null,
primaryCompany: "VSE CORP",
primaryTitle: "Chief Executive Officer, President and Director"
},
{
executiveId: 6873702,
largePhotoCircle: null,
name: "John A. Cuomo",
photoURL: null,
primaryCompany: "VSE CORP",
primaryTitle: "Chief Executive Officer, President and Director"
},
{
executiveId: 6873703,
largePhotoCircle: null,
name: "John A. Cuomo",
photoURL: null,
primaryCompany: "VSE CORP",
primaryTitle: "Chief Executive Officer, President and Director",
}
];
function filterDuplicate(myArr, prop) {
// Format - (1)
// return myArr.filter((obj, pos, arr) => {
// return arr.map(mapObj => mapObj[prop]).indexOf(obj[prop]) === pos;
// });
// Format - (2)
var res = {};
var resArr = [];
for (var elem of myArr) {
res[elem.executiveId] = elem;
}
for (let [index, elem] of Object.entries(res)) {
resArr.push(elem);
}
return resArr;
}
let finalRes = filterDuplicate(a,"executiveId");
console.log("finalResults : ",finalRes);
Solution 63 - Javascript
You can also create a generic function which will filter the array based on the object key you pass to the function
function getUnique(arr, comp) {
return arr
.map(e => e[comp])
.map((e, i, final) => final.indexOf(e) === i && i) // store the keys of the unique objects
.filter(e => arr[e]).map(e => arr[e]); // eliminate the dead keys & store unique objects
}
and you can call the function like this,
getUnique(things.thing,'name') // to filter on basis of name
getUnique(things.thing,'place') // to filter on basis of place
Solution 64 - Javascript
If you want to de-duplicate your array based on all arguments and not just one. You can use the uniqBy
function of lodash that can take a function as a second argument.
You will have this one-liner:
_.uniqBy(array, e => { return e.place && e.name })
Solution 65 - Javascript
function dupData() {
var arr = [{ comment: ["a", "a", "bbb", "xyz", "bbb"] }];
let newData = [];
comment.forEach(function (val, index) {
if (comment.indexOf(val, index + 1) > -1) {
if (newData.indexOf(val) === -1) { newData.push(val) }
}
})
}
Solution 66 - Javascript
function genFilterData(arr, key, key1) {
let data = [];
data = [...new Map(arr.map((x) => [x[key] || x[key1], x])).values()];
const makeData = [];
for (let i = 0; i < data.length; i += 1) {
makeData.push({ [key]: data[i][key], [key1]: data[i][key1] });
}
return makeData;
}
const arr = [
{make: "here1", makeText:'hj',k:9,l:99},
{make: "here", makeText:'hj',k:9,l:9},
{make: "here", makeText:'hj',k:9,l:9}]
const finalData= genFilterData(data, 'Make', 'MakeText');
console.log(finalData);
Solution 67 - Javascript
If you are using Lodash library you can use the below function as well. It should remove duplicate objects.
var objects = [{ 'x': 1, 'y': 2 }, { 'x': 2, 'y': 1 }, { 'x': 1, 'y': 2 }];
_.uniqWith(objects, _.isEqual);
Solution 68 - Javascript
We can leverage Javascript's Set object and Array's Filter function: For example:
// Example Array
const arr = [{ id: '1' }, { id: '2' }, { id: '1' }];
// Gather Unique Element Id's based on which you want to filter the elements.
const uniqIds = arr.reduce((ids, el) => ids.add(el.id), new Set());
// Filter out uniq elements.
const uniqElements = arr.filter((el) => uniqIds.delete(el.id));
console.log(uniqElements);
Solution 69 - Javascript
You can use for loop and condition to make it unique
const data = [
{ id: 1 },
{ id: 2 },
{ id: 3 },
{ id: 4 },
{ id: 5 },
{ id: 6 },
{ id: 6 },
{ id: 6 },
{ id: 7 },
{ id: 8 },
{ id: 8 },
{ id: 8 },
{ id: 8 }
];
const filtered= []
for(let i=0; i<data.length; i++ ){
let isHasNotEqual = true
for(let j=0; j<filtered.length; j++ ){
if (filtered[j].id===data[i].id){
isHasNotEqual=false
}
}
if (isHasNotEqual){
filtered.push(data[i])
}
}
console.log(filtered);
/*
output
[ { id: 1 },
{ id: 2 },
{ id: 3 },
{ id: 4 },
{ id: 5 },
{ id: 6 },
{ id: 7 },
{ id: 8 } ]
*/
Solution 70 - Javascript
This is simple way how to remove duplicity from array of objects.
I work with data a lot and this is useful for me.
const data = [{name: 'AAA'}, {name: 'AAA'}, {name: 'BBB'}, {name: 'AAA'}];
function removeDuplicity(datas){
return datas.filter((item, index,arr)=>{
const c = arr.map(item=> item.name);
return index === c.indexOf(item.name)
})
}
console.log(removeDuplicity(data))
will print into console :
[[object Object] {
name: "AAA"
}, [object Object] {
name: "BBB"
}]
Solution 71 - Javascript
function filterDuplicateQueries(queries){
let uniqueQueries = [];
queries.forEach((l, i)=>{
let alreadyExist = false;
if(uniqueQueries.length>0){
uniqueQueries.forEach((k, j)=>{
if(k.query == l.query){
alreadyExist = true;
}
});
}
if(!alreadyExist){
uniqueQueries.push(l)
}
});
Solution 72 - Javascript
var things = new Object();
things.thing = new Array();
things.thing.push({place:"here",name:"stuff"});
things.thing.push({place:"there",name:"morestuff"});
things.thing.push({place:"there",name:"morestuff"});
console.log(things);
function removeDuplicate(result, id) {
let duplicate = {};
return result.filter(ele => !duplicate[ele[id]] && (duplicate[ele[id]] = true));
}
let resolverarray = removeDuplicate(things.thing,'place')
console.log(resolverarray);
Solution 73 - Javascript
Here is a solution using new filter function of JavaScript that is quite easy . Let's say you have an array like this.
var duplicatesArray = ['AKASH','AKASH','NAVIN','HARISH','NAVIN','HARISH','AKASH','MANJULIKA','AKASH','TAPASWENI','MANJULIKA','HARISH','TAPASWENI','AKASH','MANISH','HARISH','TAPASWENI','MANJULIKA','MANISH'];
The filter function will allow you to create a new array, using a callback function once for each element in the array. So you could set up the unique array like this.
var uniqueArray = duplicatesArray.filter(function(elem, pos) {return duplicatesArray.indexOf(elem) == pos;});
> In this scenario your unique array will run through all of the values in the duplicate array. The elem variable represents the value of the element in the array (mike,james,james,alex), the position is it's 0-indexed position in the array (0,1,2,3...), and the duplicatesArray.indexOf(elem) value is just the index of the first occurrence of that element in the original array. So, because the element 'james' is duplicated, when we loop through all of the elements in the duplicatesArray and push them to the uniqueArray, the first time we hit james, our "pos" value is 1, and our indexOf(elem) is 1 as well, so James gets pushed to the uniqueArray. The second time we hit James, our "pos" value is 2, and our indexOf(elem) is still 1 (because it only finds the first instance of an array element), so the duplicate is not pushed. Therefore, our uniqueArray contains only unique values.
Here is the Demo of above function.Click Here for the above function example