Populate nested array in mongoose

How can I populate "components" in the example document:

  {
    "__v": 1,
    "_id": "5252875356f64d6d28000001",
    "pages": [
      {
        "__v": 1,
        "_id": "5252875a56f64d6d28000002",
        "page": {
          "components": [
            "525287a01877a68528000001"
          ]
        }
      }
    ],
    "author": "Book Author",
    "title": "Book Title"
  }

This is my JS where I get document by Mongoose:

  Project.findById(id).populate('pages').exec(function(err, project) {
    res.json(project);
  });

Mongoose 4.5 support this

Project.find(query)
  .populate({ 
     path: 'pages',
     populate: {
       path: 'components',
       model: 'Component'
     } 
  })
  .exec(function(err, docs) {});

And you can join more than one deep level.

Edit 03/17/2021: This is the library's implementation, what it do behind the scene is make another query to fetch thing for you and then join in memory. Although this work but we really should not rely on. It will make your db design look like SQL tables. This is costly operation and does not scale well. Please try to design your document so that it reduce join.

That works for me:

 Project.find(query)
  .lean()
  .populate({ path: 'pages' })
  .exec(function(err, docs) {

    var options = {
      path: 'pages.components',
      model: 'Component'
    };

    if (err) return res.json(500);
    Project.populate(docs, options, function (err, projects) {
      res.json(projects);
    });
  });

Documentation: Model.populate

As others have noted, Mongoose 4 supports this. It is very important to note that you can recurse deeper than one level too, if needed—though it is not noted in the docs:

Project.findOne({name: req.query.name})
	.populate({
		path: 'threads',
		populate: {
			path: 'messages', 
            model: 'Message',
			populate: {
				path: 'user',
				model: 'User'
			}
		}
	})

You can populate multiple nested documents like this.

   Project.find(query)
    .populate({ 
      path: 'pages',
      populate: [{
       path: 'components',
       model: 'Component'
      },{
        path: 'AnotherRef',
        model: 'AnotherRef',
        select: 'firstname lastname'
      }] 
   })
   .exec(function(err, docs) {});

It's is the best solution:

Car
 .find()
 .populate({
   path: 'pages.page.components'
})

I found this very helpful creating a feathersjs before hook to populate a 2 ref level deep relation. The mongoose models simply have

tables = new Schema({
  ..
  tableTypesB: { type: Schema.Types.ObjectId, ref: 'tableTypesB' },
  ..
}
tableTypesB = new Schema({
  ..
  tableType: { type: Schema.Types.ObjectId, ref: 'tableTypes' },
  ..
}

then in feathersjs before hook:

module.exports = function(options = {}) {
  return function populateTables(hook) {
    hook.params.query.$populate = {
      path: 'tableTypesB',
      populate: { path: 'tableType' }
    }

    return Promise.resolve(hook)
  }
}

So simple compared to some other methods I was trying to achieve this.

If you would like to populate another level deeper, here's what you need to do:

Airlines.findById(id)
      .populate({
        path: 'flights',
        populate:[
          {
            path: 'planeType',
            model: 'Plane'
          },
          {
          path: 'destination',
          model: 'Location',
          populate: { // deeper
            path: 'state',
            model: 'State',
            populate: { // even deeper
              path: 'region',
              model: 'Region'
            }
          }
        }]
      })

Mongoose 5.4 supports this

Project.find(query)
.populate({
  path: 'pages.page.components',
  model: 'Component'
})

This is how you can make nested population

Car
  .find()
  .populate({
    path: 'partIds',
    model: 'Part',
    populate: {
      path: 'otherIds',
      model: 'Other'
    }
  })

You can do this using $lookup aggregation as well and probably the best way as now populate is becoming extinct from the mongo

Project.aggregate([
  { "$match": { "_id": mongoose.Types.ObjectId(id) } },
  { "$lookup": {
    "from": Pages.collection.name,
    "let": { "pages": "$pages" },
    "pipeline": [
      { "$match": { "$expr": { "$in": [ "$_id", "$$pages" ] } } },
      { "$lookup": {
        "from": Component.collection.name,
        "let": { "components": "$components" },
        "pipeline": [
          { "$match": { "$expr": { "$in": [ "$_id", "$$components" ] } } },
        ],
        "as": "components"
      }},
    ],
    "as": "pages"
  }}
])

I found this question through another question which was KeystoneJS specific but was marked as duplicate. If anyone here might be looking for a Keystone answer, this is how I did my deep populate query in Keystone.

Mongoose two level population using KeystoneJs [duplicate]

exports.getStoreWithId = function (req, res) {
	Store.model
		.find()
		.populate({
			path: 'productTags productCategories',
			populate: {
				path: 'tags',
			},
		})
		.where('updateId', req.params.id)
		.exec(function (err, item) {
			if (err) return res.apiError('database error', err);
			// possibly more than one
			res.apiResponse({
				store: item,
			});
		});
};

For someone who has the problem with populate and also wants to do this:

• chat with simple text & quick replies (bubbles)
• 4 database collections for chat: clients, users, rooms, messasges.
• same message DB structure for 3 types of senders: bot, users & clients
• refPath or dynamic reference
• populate with path and model options
• use findOneAndReplace/replaceOne with $exists
• create a new document if the fetched document doesn't exist

---

CONTEXT

Goal

1. Save a new simple text message to the database & populate it with the user or client data (2 different models).
2. Save a new quickReplies message to the database and populate it with the user or client data.
3. Save each message its sender type: clients, users & bot.
4. Populate only the messages who have the sender clients or users with its Mongoose Models. _sender type client models is clients, for user is users.

Message schema:

const messageSchema = new Schema({
    room: {
        type: Schema.Types.ObjectId,
        ref: 'rooms',
        required: [true, `Room's id`]
    },
    sender: {
         _id: { type: Schema.Types.Mixed },
        type: {
            type: String,
            enum: ['clients', 'users', 'bot'],
            required: [true, 'Only 3 options: clients, users or bot.']
        }
    },
    timetoken: {
        type: String,
        required: [true, 'It has to be a Nanosecond-precision UTC string']
    },
    data: {
        lang: String,
        // Format samples on https://docs.chatfuel.com/api/json-api/json-api
        type: {
            text: String,
            quickReplies: [
                {
                    text: String,
                    // Blocks' ids.
                    goToBlocks: [String]
                }
            ]
        }
    }

mongoose.model('messages', messageSchema);

---

SOLUTION

My server side API request

My code

Utility function (on chatUtils.js file) to get the type of message that you want to save:

/**
 * We filter what type of message is.
 *
 * @param {Object} message
 * @returns {string} The type of message.
 */
const getMessageType = message => {
    const { type } = message.data;
    const text = 'text',
        quickReplies = 'quickReplies';

    if (type.hasOwnProperty(text)) return text;
    else if (type.hasOwnProperty(quickReplies)) return quickReplies;
};

/**
 * Get the Mongoose's Model of the message's sender. We use
 * the sender type to find the Model.
 *
 * @param {Object} message - The message contains the sender type.
 */
const getSenderModel = message => {
    switch (message.sender.type) {
        case 'clients':
            return 'clients';
        case 'users':
            return 'users';
        default:
            return null;
    }
};

module.exports = {
    getMessageType,
    getSenderModel
};

My server side (using Nodejs) to get the request of saving the message:

app.post('/api/rooms/:roomId/messages/new', async (req, res) => {
        const { roomId } = req.params;
        const { sender, timetoken, data } = req.body;
        const { uuid, state } = sender;
        const { type } = state;
        const { lang } = data;

        // For more info about message structure, look up Message Schema.
        let message = {
            room: new ObjectId(roomId),
            sender: {
                _id: type === 'bot' ? null : new ObjectId(uuid),
                type
            },
            timetoken,
            data: {
                lang,
                type: {}
            }
        };

        // ==========================================
        //          CONVERT THE MESSAGE
        // ==========================================
        // Convert the request to be able to save on the database.
        switch (getMessageType(req.body)) {
            case 'text':
                message.data.type.text = data.type.text;
                break;
            case 'quickReplies':
                // Save every quick reply from quickReplies[].
                message.data.type.quickReplies = _.map(
                    data.type.quickReplies,
                    quickReply => {
                        const { text, goToBlocks } = quickReply;

                        return {
                            text,
                            goToBlocks
                        };
                    }
                );
                break;
            default:
                break;
        }

        // ==========================================
        //           SAVE THE MESSAGE
        // ==========================================
        /**
         * We save the message on 2 ways:
         * - we replace the message type `quickReplies` (if it already exists on database) with the new one.
         * - else, we save the new message.
         */
        try {
            const options = {
                // If the quickRepy message is found, we replace the whole document.
                overwrite: true,
                // If the quickRepy message isn't found, we create it.
                upsert: true,
                // Update validators validate the update operation against the model's schema.
                runValidators: true,
                // Return the document already updated.
                new: true
            };

            Message.findOneAndUpdate(
                { room: roomId, 'data.type.quickReplies': { $exists: true } },
                message,
                options,
                async (err, newMessage) => {
                    if (err) {
                        throw Error(err);
                    }

                    // Populate the new message already saved on the database.
                    Message.populate(
                        newMessage,
                        {
                            path: 'sender._id',
                            model: getSenderModel(newMessage)
                        },
                        (err, populatedMessage) => {
                            if (err) {
                                throw Error(err);
                            }

                            res.send(populatedMessage);
                        }
                    );
                }
            );
        } catch (err) {
            logger.error(
                `#API Error on saving a new message on the database of roomId=${roomId}. ${err}`,
                { message: req.body }
            );

            // Bad Request
            res.status(400).send(false);
        }
    });

TIPs:

For the database:

• Every message is a document itself.
• Instead of using refPath, we use the util getSenderModel that is used on populate(). This is because of the bot. The sender.type can be: users with his database, clients with his database and bot without a database. The refPath needs true Model reference, if not, Mongooose throw an error.
• sender._id can be type ObjectId for users and clients, or null for the bot.

For API request logic:

• We replace the quickReply message (Message DB has to have only one quickReply, but as many simple text messages as you want). We use the findOneAndUpdate instead of replaceOne or findOneAndReplace.
• We execute the query operation (the findOneAndUpdate) and the populate operation with the callback of each one. This is important if you don't know if use async/await, then(), exec() or callback(err, document). For more info look the Populate Doc.
• We replace the quick reply message with the overwrite option and without $set query operator.
• If we don't find the quick reply, we create a new one. You have to tell to Mongoose this with upsert option.
• We populate only one time, for the replaced message or the new saved message.
• We return to callbacks, whatever is the message we've saved with findOneAndUpdate and for the populate().
• In populate, we create a custom dynamic Model reference with the getSenderModel. We can use the Mongoose dynamic reference because the sender.type for bot hasn't any Mongoose Model. We use a Populating Across Database with model and path optins.

---

I've spend a lot of hours solving little problems here and there and I hope this will help someone! 

I use following syntax that is clean. This codeblock is from my project

const result = await Result.find(filter).populate('student exam.subject')

Explanation

Say you have two schemas

Exam Schema

const ExamSchema = new mongoose.Schema({
   ...
   type: String,
   ...
})

Result Schema

const resultSchema = new mongoose.Schema({
    ...
    exam: ExamSchema,
    student: {
        type: mongoose.Schema.Types.ObjectId,
        ref: 'User',
        required: true
    }
})

If I wanted to query and populate from results

1. Only by student id

   const result = await Result.find(filter).populate('student')
   

2. Only by exam type

   const result = await Result.find(filter).populate('exam.type')
   

3. By both student id and exam type

   const result = await Result.find(filter).populate('student exam.type')
   

If you need any more clarification plz ask in comments

I struggled with this for a whole bloody day. None of the solutions above worked. The only thing that worked in my case for an example like the following:

{
  outerProp1: {
    nestedProp1: [
      { prop1: x, prop2: y, prop3: ObjectId("....")},
      ...
    ],
    nestedProp2: [
      { prop1: x, prop2: y, prop3: ObjectId("....")},
      ...
    ]
  },
  ...
}
    

is to do the following: (Assuming populating after fetch - but also works when calling populate from the Model class (followed by exec))

await doc.populate({
  path: 'outerProp1.nestedProp1.prop3'
}).execPopulate()

// doc is now populated

In other words, the outermost path property has to contain the full path. No partially complete path coupled with populate properties seemed to work (and the model property doesn't seem to be necessary; makes sense since it is included in the schema). Took me a whole damn day to figure this out! Not sure why the other examples don't work.

(Using Mongoose 5.5.32)

Answer with one level nested populate and projection, you may find it interesting.

https://mongoplayground.net/p/2dpeZWsXR-V

query:

db.booking.aggregate([
  {
    "$match": {
      id: "61fdfeef678791001880da25"
    }
  },
  {
    $unwind: "$cart"
  },
  {
    "$lookup": {
      "from": "products",
      "localField": "cart.product",
      "foreignField": "id",
      "as": "prod"
    }
  },
  {
    "$unwind": "$prod"
  },
  {
    "$project": {
      id: 1,
      status: 1,
      cart: [
        {
          id: "$cart.id",
          date: "$cart.date",
          timeSlots: "$cart.timeSlots",
          product: {
            id: "$prod.id",
            name: "$prod.name",
            
          }
        }
      ],
      
    }
  }
])

db:

db={
  "booking": [
    {
      "status": "0",
      "cart": [
        {
          "id": "61fdffc7678791001880da5f",
          "date": "2022-02-05T00:00:00.000Z",
          "product": "61fd7bc5801207001b94d949",
          "timeSlots": [
            {
              "id": "61fd7bf2801207001b94d99c",
              "spots": 1
            }
          ],
          "createdAt": "2022-02-05T04:40:39.155Z",
          "updatedAt": "2022-02-05T04:40:39.155Z"
        }
      ],
      "version": 1,
      "id": "61fdfeef678791001880da25"
    }
  ],
  "products": [
    {
      "meta": {
        "timeZone": "America/New_York"
      },
      "photos": [],
      "name": "Guide To Toronto Canada",
      "timeSlots": [
        {
          "id": "61fd7bcf801207001b94d94d",
          "discount": null,
          "endTime": "2022-02-05T03:01:00.000Z",
          "spots": null,
          "startTime": "2022-02-04T14:00:00.000Z"
        },
        {
          "id": "61fd7bf2801207001b94d99c",
          "discount": null,
          "endTime": "2022-02-04T20:18:00.000Z",
          "spots": 15,
          "startTime": "2022-02-04T19:18:00.000Z"
        },
        
      ],
      "mrp": 20,
      "id": "61fd7bc5801207001b94d949"
    }
  ]
}

I tried with the latest version of mongoose 5.10

Please consider the below use case of Schema definition and then review how you can populate

const jobsSchema: Schema = new Schema({
    employerId:String
}, {strict : false})

jobsSchema.virtual('employer', {
    ref: 'Employer',
    localField: 'employerId',
    foreignField: '_id',
    justOne: true 
});

Another Model is the job view

const jobsViewSchema: Schema = new Schema({
    jobId:String
}, {strict : false})

jobsViewSchema.virtual('job', {
    ref: 'Jobs',
    localField: 'jobId',
    foreignField: '_id',
    justOne: true
});

Now to populate

this.JobViewModel.find(query).populate({ 
     path: 'job', 
     populate: ['employer', 'Virtual2', 'Virtual3'] 
})

This will populate the whole object perfectly.

Remove docs reference

if (err) {
    return res.json(500);
}
Project.populate(docs, options, function (err, projects) {
    res.json(projects);
});

This worked for me.

if (err) {
    return res.json(500);
}
Project.populate(options, function (err, projects) {
    res.json(projects);
});