Unit testing with MongoDB

My database of choice is MongoDB. I'm writing a data-layer API to abstract implementation details from client applications - that is, I'm essentially providing a single public interface (an object which acts as an IDL).

I'm testing my logic as I go in a TDD manner. Before each unit test, an @Before method is called to create a database singleton, after which, when the test completes, an @After method is called to drop the database. This helps to promote independence among unit tests.

Nearly all unit tests, i.e. performing a contextual query, require some kind of insertion logic to occur before hand. My public interface provides an insert method - yet, it seems incorrect to use this method as precursor logic to each unit test.

Really I need some kind of mocking mechanism, yet, I haven't had much experience with mocking frameworks, and it seems that Google returns nothing re a mocking framework one might use with MongoDB.

What do others do in these situations? That is, how do people unit test code that interacts with a database?

Also, my public interface connects to a database defined in a external configuration file - it seems incorrect to use this connection for my unit testing - again, a situation that would benefit from some kind of mocking?

Technically tests that talk to a database (nosql or otherwise) are not unit tests, as the tests are testing interactions with an external system, and not just testing an isolated unit of code. However tests that talk to a database are often extremely useful, and are often fast enough to run with the other unit tests.

Usually I have a Service interface (eg UserService) which encapsulates all the logic for dealing with the database. Code that relies on UserService can use a mocked version of UserService and is easily tested.

When testing the implementation of the Service that talks to Mongo, (eg MongoUserService) it is easiest to write some java code that will start/stop a mongo process on the local machine, and have your MongoUserService connect to that, see this question for some notes.

You could try to mock the functionality of the database while testing MongoUserService, but generally that is too error prone, and doesn't test what you really want to test, which is interaction with a real database. So when writing tests for MongoUserService, you set up a database state for each test. Look at DbUnit for an example of a framework for doing so with a database.

As sbridges wrote in this post it is a bad idea not to have a dedicated service (sometimes also known as repository or DAO) which abstracts the data access from the logic. Then you could test the logic by providing a mock of the DAO.

Another approach which I do is to create a Mock of the Mongo object (e.g. PowerMockito) and then return the appropriate results. This because you don't have to test if the database works in unit tests but more over you should test if the right query was sent to the databse.

Mongo mongo = PowerMockito.mock(Mongo.class);
DB db = PowerMockito.mock(DB.class);
DBCollection dbCollection = PowerMockito.mock(DBCollection.class);

PowerMockito.when(mongo.getDB("foo")).thenReturn(db);
PowerMockito.when(db.getCollection("bar")).thenReturn(dbCollection);

MyService svc = new MyService(mongo); // Use some kind of dependency injection
svc.getObjectById(1);

PowerMockito.verify(dbCollection).findOne(new BasicDBObject("_id", 1));

That would also be an option. Of course the creation of the mocks and returning of the appropriate objects is just coded as an example above.

I wrote a MongoDB fake implementation in Java: mongo-java-server

Default is a in-memory backend, that can be easily used in Unit and Integration tests.

Example

MongoServer server = new MongoServer(new MemoryBackend());
// bind on a random local port
InetSocketAddress serverAddress = server.bind();

MongoClient client = new MongoClient(new ServerAddress(serverAddress));

DBCollection coll = client.getDB("testdb").getCollection("testcoll");
// creates the database and collection in memory and inserts the object
coll.insert(new BasicDBObject("key", "value"));

assertEquals(1, collection.count());
assertEquals("value", collection.findOne().get("key"));

client.close();
server.shutdownNow();

Today I think the best practice is to use testcontainers library (Java) or testcontainers-python port on Python. It allows to use Docker images with unit tests. To run container in Java code just instantiate GenericContainer object (example):

GenericContainer mongo = new GenericContainer("mongo:latest")
    .withExposedPorts(27017);

MongoClient mongoClient = new MongoClient(mongo.getContainerIpAddress(), mongo.getMappedPort(27017));
MongoDatabase database = mongoClient.getDatabase("test");
MongoCollection<Document> collection = database.getCollection("testCollection");

Document doc = new Document("name", "foo")
        .append("value", 1);
collection.insertOne(doc);

Document doc2 = collection.find(new Document("name", "foo")).first();
assertEquals("A record can be inserted into and retrieved from MongoDB", 1, doc2.get("value"));

or on Python (example):

mongo = GenericContainer('mongo:latest')
mongo.with_bind_ports(27017, 27017)

with mongo_container:
    def connect():
        return MongoClient("mongodb://{}:{}".format(mongo.get_container_host_ip(),
                                                    mongo.get_exposed_port(27017)))

    db = wait_for(connect).primer
    result = db.restaurants.insert_one(
        # JSON as dict object
    )

    cursor = db.restaurants.find({"field": "value"})
    for document in cursor:
        print(document)

I'm surprised no one advised to use fakemongo so far. It emulates mongo client pretty well, and it all runs on same JVM with tests - so integration tests become robust, and technically much more close to true "unit tests", since no foreign system interaction takes place. It's like using embedded H2 to unit test your SQL code. I was very happy using fakemongo in unit tests that test database integration code in end-to-end manner. Consider this configuration in test spring context:

@Configuration
@Slf4j
public class FongoConfig extends AbstractMongoConfiguration {
    @Override
    public String getDatabaseName() {
        return "mongo-test";
    }

    @Override
    @Bean
    public Mongo mongo() throws Exception {
        log.info("Creating Fake Mongo instance");
        return new Fongo("mongo-test").getMongo();
    }

    @Bean
    @Override
    public MongoTemplate mongoTemplate() throws Exception {
        return new MongoTemplate(mongo(), getDatabaseName());
    }

}

With this you can test your code that uses MongoTemplate from spring context, and in combination with nosql-unit, jsonunit, etc. you get robust unit tests that cover mongo querying code.

@Test
@UsingDataSet(locations = {"/TSDR1326-data/TSDR1326-subject.json"}, loadStrategy = LoadStrategyEnum.CLEAN_INSERT)
@DatabaseSetup({"/TSDR1326-data/dbunit-TSDR1326.xml"})
public void shouldCleanUploadSubjectCollection() throws Exception {
    //given
    JobParameters jobParameters = new JobParametersBuilder()
            .addString("studyId", "TSDR1326")
            .addString("execId", UUID.randomUUID().toString())
            .toJobParameters();

    //when
    //next line runs a Spring Batch ETL process loading data from SQL DB(H2) into Mongo
    final JobExecution res = jobLauncherTestUtils.launchJob(jobParameters);

    //then
    assertThat(res.getExitStatus()).isEqualTo(ExitStatus.COMPLETED);
    final String resultJson = mongoTemplate.find(new Query().with(new Sort(Sort.Direction.ASC, "topLevel.subjectId.value")),
            DBObject.class, "subject").toString();

    assertThatJson(resultJson).isArray().ofLength(3);
    assertThatDateNode(resultJson, "[0].topLevel.timestamp.value").isEqualTo(res.getStartTime());

    assertThatNode(resultJson, "[0].topLevel.subjectECode.value").isStringEqualTo("E01");
    assertThatDateNode(resultJson, "[0].topLevel.subjectECode.timestamp").isEqualTo(res.getStartTime());

    ... etc
}

I used fakemongo without problems with mongo 3.4 driver, and community is really close to release a version that supports 3.6 driver (https://github.com/fakemongo/fongo/issues/316).