JUnit 4 vs TestNG - Update 2013 - 2014

JUnit 4 and TestNG used to be comparable. What are the pros and cons of the two testing frameworks?

I was comparing TestNG and JUnit4 today, and the main advantage that I can point out with my limited experience in testing-frameworks is that TestNG has a more elegant way of handling parametrized tests with the data-provider concept.

As far as I can tell with JUnit4 you have to create a separate test class for each set of parameters you want to test with (ran with @RunWith(Parameterized.class)). With TestNG you can have multiple data-providers in a single test class, so you can keep all your tests for a single class in a single test-class as well.

So far that is the only thing that I can point out as a benefit of TestNG over JUnit4.

Intellij IDEA includes support for TestNG and JUnit out of the box. However, Eclipse only supports JUnit out of the box and needs a TestNG plugin installed to make it work.

However, a more annoying problem I ran into with TestNG is that your test classes need to extend PowerMockTestCase if you are using PowerMock to mock dependencies in your tests. Apparently there are ways to configure the object-factory your test framework needs to know about when using PowerMock via a special method, or via the testng.xml suite definition, but those seem to be broken at the moment. I dislike having test-classes extend test-framework classes, it seems hackish.

If you don't use PowerMock this is not an issue of course, but all in all I get the impression that JUnit4 is better supported.

Based on my experience with both frameworks, testng has a few convenient features that the JUnit team has declined to implement for several years. I prefer it to JUnit for this reason. Converting to testng is easy since it basically supports pretty much everything in JUnit (there's a converter plugin for eclipse even), converting back to JUnit is not so great because of these missing features.

• In testng @BeforeClass methods are not static and execute before the tests in that class run rather than when the test class is loaded (the JUnit behavior). I once had a JUnit project where all of the database tests (a few dozen) initialized the database right at the start, which was quite idiotic behavior. There was a lot of debate for and against this in the JUnit community. The gist of it was that every test method should have its own test fixture and therefore you should not have a beforeAll style method that is non static because that would allow you to sneakily set an instance variable once and then use it in all your tests. Valid, but really annoying for integration tests. TestNG gives users the choice here. Junit does not, by design, which is annoying.

• Testng data providers are a bit more flexible than the JUnit equivalent. You can specify per test which data provider method should be providing the input instead of having a one size fits all approach for the entire class as in JUnit. So you can have positive and negative case data providers for your tests in one class. Very nice to have.

• You can mark up a class with @Test in testng, which means: every public method is a test. In Junit you need to copy/paste @Test on every method.

An annoyance with both is the way hamcrest is bundled with JUnit and the way JUnit is bundled with testng. There are alterate jars in maven that don't have this issue.

My big worry with both frameworks is that they both seem to have stopped evolving. Releases are increasingly less frequent and tend to have less and less noteworthy features. The whole BDD movement seems to have had little impact on either framework for example. Also, JUnit could simply have adopted most of what I listed above. There is no good technical reason why JUnit can't implement any of these things; the people behind JUnit just choose not to implement these things. Both projects seem to be lacking a vision for future directions as well and they seem to be happy doing just minor tweaks for the last few years.

I was looking for a good reasons to switch TestNG over JUnit and I found this slides by Tomek Kaczanowski addressing this question very well. Tomek is author of Practical Unit Testing book which seems to be respected very much by developers and testers.

If you work on Java/Scala project and Gradle is your build tool of choice, bear in mind that ScalaTest framework has only JUnitRunner to run your scala tests. In other words, your have a choice:

• Java JUnit tests + Scala Tests run with JUnitRunner => Better Gradle integration
• Java testNG tests + Scala Tests run with Scala Runner => Poor Gradle Integration, since scala test runner is a single bulk task from Gradle's point of view.

You can use Mockito as your mocking framework. It integrates nicely with TestNG. You don't have to extend any class to use Mockito with TestNG. You testing code is less coupled this way and if for any reason you need to use other mocking frameworks it's easy to do so.

In a nutshell..

If your scope is limited to fine detailed unit tests with no dependencies in between them then JUnit can be used.

If your scope requires functional testing that may/may not require dependencies and sharing of data (parameters) between tests then pick TestNG. Also, TestNG can do unit testing similar to JUnit. SO you can have a suite of unit tests and a suite of functional tests.