EasyMock vs Mockito: design vs maintainability?

One way of thinking about this is: if we care about the design of the code then EasyMock is the better choice as it gives feedback to you by its concept of expectations.

If we care about the maintainability of tests (easier to read, write and having less brittle tests which are not affected much by change), then Mockito seems a better choice.

My questions are:

• If you have used EasyMock in large scale projects, do you find that your tests are harder to maintain?
• What are the limitations of Mockito (other than endo testing)?

I won't argue about test readability, size or testing techniques of these frameworks, I believe they are equal, but on a simple example I'll show you the difference.

Given: We have a class which is responsible for storing something somewhere:

public class Service {

    public static final String PATH = "path";
    public static final String NAME = "name";
    public static final String CONTENT = "content";
    private FileDao dao;

    public void doSomething() {
        dao.store(PATH, NAME, IOUtils.toInputStream(CONTENT));
    }

    public void setDao(FileDao dao) {
        this.dao = dao;
    }
}

and we want to test it:

Mockito:

public class ServiceMockitoTest {

    private Service service;

    @Mock
    private FileDao dao;

    @Before
    public void setUp() {
        MockitoAnnotations.initMocks(this);
        service = new Service();
        service.setDao(dao);
    }

    @Test
    public void testDoSomething() throws Exception {
        // given
        // when
        service.doSomething();
        // then
        ArgumentCaptor<InputStream> captor = ArgumentCaptor.forClass(InputStream.class);
        Mockito.verify(dao, times(1)).store(eq(Service.PATH), eq(Service.NAME), captor.capture());
        assertThat(Service.CONTENT, is(IOUtils.toString(captor.getValue())));
    }
}

EasyMock:

public class ServiceEasyMockTest {
    private Service service;
    private FileDao dao;

    @Before
    public void setUp() {
        dao = EasyMock.createNiceMock(FileDao.class);
        service = new Service();
        service.setDao(dao);
    }

    @Test
    public void testDoSomething() throws Exception {
        // given
        Capture<InputStream> captured = new Capture<InputStream>();
        dao.store(eq(Service.PATH), eq(Service.NAME), capture(captured));
        replay(dao);
        // when
        service.doSomething();
        // then
        assertThat(Service.CONTENT, is(IOUtils.toString(captured.getValue())));
        verify(dao);
    }
}

As you can see both test are fairly the same and both of them are passing. Now, let’s imagine that somebody else changed Service implementation and trying to run tests.

New Service implementation:

dao.store(PATH + separator, NAME, IOUtils.toInputStream(CONTENT));

separator was added at the end of PATH constant

How the tests results will look like right now ? First of all both tests will fail, but with different error messages:

EasyMock:

java.lang.AssertionError: Nothing captured yet
	at org.easymock.Capture.getValue(Capture.java:78)
	at ServiceEasyMockTest.testDoSomething(ServiceEasyMockTest.java:36)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

Mockito:

Argument(s) are different! Wanted:
dao.store(
    "path",
    "name",
    <Capturing argument>
);
-> at ServiceMockitoTest.testDoSomething(ServiceMockitoTest.java:34)
Actual invocation has different arguments:
dao.store(
    "path\",
    "name",
    java.io.ByteArrayInputStream@1c99159
);
-> at Service.doSomething(Service.java:13)
 

What happened in EasyMock test, why result wasn't captured ? Is store method wasn't executed, but wait a minute, it was, why EasyMock lies to us?

It's because EasyMock mixing two responsibilities in a single line - stubbing and verification. That's why when something is wrong it's hard to understand which part is causing failure.

Of course you can tell me - just change the test and move verify before assertion. Wow, are you serious, developers should keep in mind some magic order inforced by mocking framework?

By the way, it won’t help:

java.lang.AssertionError: 
  Expectation failure on verify:
    store("path", "name", capture(Nothing captured yet)): expected: 1, actual: 0
	at org.easymock.internal.MocksControl.verify(MocksControl.java:111)
	at org.easymock.classextension.EasyMock.verify(EasyMock.java:211)

Still, it is saying to me that method was not executed, but it was, only with another parameters.

Why Mockito is better ? This framework doesn't mix two responsibilities in a single place and when your tests will fail, you will easily understand why.

>if we care about the Design of the code then Easymock is the better choice as it gives feedback to you by its concept of expectations

Interesting. I found that 'concept of expectations' makes many devs put more & more expectations in the tests only to satisfy UnexpectedMethodCall problem. How does it influence the design?

The test should not break when you change code. The test should break when the feature stops working. If one likes the tests to break when any code change happens I suggest to write a test that asserts the md5 checksum of the java file :)

I'm an EasyMock developer so a bit partial but of course I've used EasyMock on large scale projects.

My opinion is that EasyMock tests will indeed breaks once in a while. EasyMock forces you to do a complete recording of what you expect. This requires some discipline. You should really record what is expected not what the tested method currently needs. For instance, if it doesn't matter how many time a method is called on a mock, don't be afraid of using andStubReturn. Also, if you don't care about a parameter, use anyObject() and so on. Thinking in TDD can help on that.

My analyze is that EasyMock tests will break more often but Mockito ones won't when you would want them to. I prefer my tests to break. At least I'm aware of what was the impacts of my development. This is of course, my personal point of view.

I don't think you should be too concerned about this. Both Easymock and Mockito can be configured to be 'strict' or 'nice' the only difference is that by default Easymock is strict wheras Mockito is nice.

As with all testing there's no hard and fast rule, you need to balance test confidence against maintainability. I typically find there are certain functional or technical areas that demand a high level of confidence for which I would use 'strict' mocks. For example we probably wouldn't want the debitAccount() method to be called more than once! However there are other cases in which the mock is really little more than a stub so we can test the real 'meat' of the code.

In the early days of Mockito's life API compatibility was a problem but more tools now support the framework. Powermock (a personal favorite) now has a mockito extension

I prefer mockito to be honest. been using EasyMock with unitils and the combination of both oftenly results in exceptions like IllegalArgumentException: not an interface as well as MissingBehaviorExceptions. In both cases though the code and test code are perfectly fine. It appeared that the MissingBehaviorException was due to the fact that mocked objects created with createMock (using classextentions!!) did produce this error. When using @Mock it did work! I do not like that kind of misleading behavior and for me that is a clear indication the developers of it do not know what they are doing. A good framework should always be easy to use and not ambiguous. The IllegalArgumentException was also due to some mingle of EasyMock internals. Also, the recording is not what I want to do. I want to test if my code throws exceptions or not and that it returns the expected results. That in combination with code coverage is the right tool for me. I do not want my tests to break whenever I put 1 line of code above or below the previous one because that improves performance or so. With mockito it is no problem. With EasyMock, it will result tests to fail even though the code is not broken. That is bad. It costs time, thus money. You want to test for expected behavior. Do you really care about the order of things? I suppose in rare occasions you might. Use Easymock then. In other case, I think you'll spend considerably less time using mockito to write your tests.

Kind regards Lawrence