Running junit tests in parallel in a Maven build?
JavaMavenJunitJava Problem Overview
I'm using JUnit 4.4 and Maven and I have a large number of long-running integration tests.
When it comes to parallelizing test suites there are a few solutions that allow me to run each test method in a single test-class in parallel. But all of these require that I change the tests in one way or another.
I really think it would be a much cleaner solution to run X different test classes in X threads in parallel. I have hundreds of tests so I don't really care about threading individual test-classes.
Is there any way to do this?
Java Solutions
Solution 1 - Java
Use maven plugin:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.7.1</version>
<configuration>
<parallel>classes</parallel>
<threadCount>5</threadCount>
</configuration>
</plugin>
</plugins>
</build>
Solution 2 - Java
From junit 4.7 it's now possible to run tests in parallel without using TestNG. Actually it has been possible since 4.6, but there are a number of fixes being made in 4.7 that will make it a viable option. You may also run parallel tests with spring, which you can read about here
Solution 3 - Java
Inspired by JUnit's experimental ParallelComputer runner I've built my own ParallelSuite and ParallelParameterized runners. Using these runners one can easily parallelize test suites and parameterized tests.
ParallelSuite.java
public class ParallelSuite extends Suite {
public ParallelSuite(Class<?> klass, RunnerBuilder builder) throws InitializationError {
super(klass, builder);
setScheduler(new RunnerScheduler() {
private final ExecutorService service = Executors.newFixedThreadPool(4);
public void schedule(Runnable childStatement) {
service.submit(childStatement);
}
public void finished() {
try {
service.shutdown();
service.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e) {
e.printStackTrace(System.err);
}
}
});
}
}
ParallelParameterized.java
public class ParallelParameterized extends Parameterized {
public ParallelParameterized(Class<?> arg0) throws Throwable {
super(arg0);
setScheduler(new RunnerScheduler() {
private final ExecutorService service = Executors.newFixedThreadPool(8);
public void schedule(Runnable childStatement) {
service.submit(childStatement);
}
public void finished() {
try {
service.shutdown();
service.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
} catch (InterruptedException e) {
e.printStackTrace(System.err);
}
}
});
}
}
Usage is simple. Just change @RunWith annotations value to one of these Parallel* classes.
@RunWith(ParallelSuite.class)
@SuiteClasses({ATest.class, BTest.class, CTest.class})
public class ABCSuite {}
Solution 4 - Java
tempus-fugit offers something similar, check the docs for details. It relies on JUnit 4.7 and you just mark your test to @RunWith(ConcurrentTestRunner).
Cheers
Solution 5 - Java
You can check out the open source library - Test Load Balancer. It does exactly what you ask for - run different test classes in parallel. This integrates at the ant-junit level so that you do not have to change your tests in anyway. I am one of the authors of the library.
Also, think about not running them in threads as you may need a process level sandbox. For example, if you are hitting a DB in your integration tests, you do not want one test to fail because another test added some data in a different thread. Most of the times, tests are not written with this in mind.
Finally, how have solved this problem till now?
Solution 6 - Java
You can run the tests in parallel using ParallelComputer provided by Junit itself. Here's a small snippet to get you started.
Class[] cls = { TestCase1.class, TestCase2.class };
Result result = JUnitCore.runClasses(ParallelComputer.classes(), cls);
List<Failure> failures = result.getFailures();
This will help when you need to run tests from code as it has no dependencies on Maven or any other build management tools.
Please note that, this will run all test cases in parallel, if you have any dependencies between different test cases it might result in false positives. You SHOULD NOT have interdependent tests anyway.
Solution 7 - Java
TestNG can do that (this was my first reflex - then I saw you're already having a lot of testcases).
For JUnit, look at parallel-junit.
Solution 8 - Java
Another choice: Punner, a new parallel junit runner and maven plugin. You don't have to change your code, copy it to your pom.xml:
<!-- Disable default surefire based testing -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<groupId>com.github.marks-yag</groupId>
<artifactId>punner-maven-plugin</artifactId>
<version>${version}</version>
<configuration>
</configuration>
<executions>
<execution>
<id>test</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
</executions>
</plugin>
Punner can run test methods in parallel, can keep test outputs separately and clean.
Punner will reduce your mvn console outputs, like this:
[INFO] --- punner-maven-plugin:0.9.13:test (test) @ ipc ---
[INFO] Punner report directory: /Users/guile/workspace/ipc/target/punner-reports
[INFO]
[INFO] com.github.yag.ipc.IPCTest.testConnectionHandler.............. PASSED
[INFO] com.github.yag.ipc.IPCTest.testSequence....................... PASSED
[INFO] com.github.yag.ipc.IPCTest.testPartialContent................. PASSED
[INFO] com.github.yag.ipc.IPCTest.testResponseContent................ PASSED
[INFO] com.github.yag.ipc.IPCTest.testPingPong....................... PASSED
[INFO] com.github.yag.ipc.IPCTest.testServerClose.................... PASSED
[INFO] com.github.yag.ipc.IPCTest.testServerSideHeartbeatTimeout..... PASSED
[INFO] com.github.yag.ipc.IPCTest.testClientSideHeartbeatTimeout..... PASSED
[INFO] com.github.yag.ipc.IPCTest.testClientSideHeartbeat............ PASSED
[INFO] com.github.yag.ipc.IPCTest.testClientReconnect................ PASSED
[INFO]
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.952 sec, Time saved: 25.919 sec.
Punner produce surefire compatible outputs, you can also get raw log data and a markdown format report from reports directory:
ā ipc git:(develop) ll target/punner-reports
total 104
-rw-r--r-- 1 guile staff 11K Oct 15 23:07 TEST-com.github.yag.ipc.IPCTest.xml
-rw-r--r-- 1 guile staff 298B Oct 15 23:07 com.github.yag.ipc.IPCTest.txt
drwxr-xr-x 12 guile staff 384B Oct 8 00:50 logs
-rw-r--r-- 1 guile staff 33K Oct 15 23:07 report.md
Punner is my personal project, I written Punner to speed up unit test phase of some other projects such as IPC framework, fine-grained locking, journal service, distributed workflow engine, etc. It saved a lot of my waiting time.
Punner don't support some advanced feature yet. I'm very glad if you could try it and give me some feedback.
Solution 9 - Java
You can change your test to be TestNg test in a minute (you just need to change imports), TestNG is the best in parallel testing.
Solution 10 - Java
You could try Gridgain that lets you run distribute your tests across a compute grid.