Is there a way to "fail fast" for junit with the maven surefire plugin?

Maven 2 Problem Overview

I'm currently working on a java project using maven. We use the maven surefire plugin to run our junit suite as part of the build process.

Our test suite is rapidly growing, in both coverage and execution time. The execution time is very frustrating and time consuming when you end up waiting ten minutes to find out that a test failed in the first minute of testing.

I would like to find a way to make the build process fail upon the first error/failure in the test suite. I know that this is possible for other build tools, but I have been unable to find a way to do this with maven surefire.

I know that there is [an unresolved ticket for this functionality][1] in the surefire jira, but I'm hoping that there is an existing solution out there for this.

[1]: http://jira.codehaus.org/browse/SUREFIRE-580 "An unresolved ticket for this in the surefire jira"

Maven 2 Solutions

Solution 1 - Maven 2

As of September 6th 2015 it appears there is, -Dsurefire.skipAfterFailureCount=1.

As of October 19th 2015 version 2.19 has been released.

Solution 2 - Maven 2

As far as I know, no, and this really requires the resolution of SUREFIRE-580. If you want to make this happen faster, you should at least vote for the issue and, optionally, submit a patch ;)

Solution 3 - Maven 2

There may be a suitable workaround, but it depends on your requirements and you need to use a CI server which can handle jvm process return codes.

The basic idea is to stop Maven's JVM process altogether and let the OS know that the process has stopped unexpectedly. Then, a continuous integration server like Jenkins/Hudson should be able to check for a non-zero exit code and let you know that a test has failed.

The first step is to make surefire exit the JVM at the first test failure. You can do that with JUnit 4.7 or higher by using a custom RunListener (put it in src/test/java):

package org.example
import org.junit.runner.notification.Failure;
import org.junit.runner.notification.RunListener;
public class FailFastListener extends RunListener {
        public void testFailure(Failure failure) throws Exception {
                System.err.println("FAILURE: " + failure);
                System.exit(-1);
        }
}

Then, you need to configure that class so surefire will register it with the JUnit 4 Runner. Edit your pom.xml and add the listener configuration property to the maven-surefire-plugin. You will also need to configure surefire to not fork a new JVM process for executing the tests. Otherwise, it will just go on with the next test cases.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.10</version>
    <configuration>
        <forkMode>never</forkMode>
        <properties>
            <property>
                <name>listener</name>
                <value>org.example.FailFastListener</value>
            </property>
        </properties>
    </configuration>
</plugin>

If this does not help, I'd try to fork the maven surefire junit provider plugin.

Btw, unit tests, by definition, should run faster than 0.1 seconds. If your build really takes so long due to the unit tests, you will have to make them run faster in the future.

Solution 4 - Maven 2

This isn't a direct answer to the question, but it can also be useful to feed the output of maven through grep, to strip out most stuff and help you to see where the test failures are.

Like this:

mvn test | grep -w 'Running\|Tests'

Which produces output (for my code) like this:

Running scot.mygov.pp.test.dashboard.DashboardJsonSerDesCRUDTest
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.029 sec
Running scot.mygov.pp.test.dashboard.DashboardBaselineCRUDTest
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.026 sec
Running scot.mygov.pp.test.dashboard.DashboardDatabaseValidationTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.264 sec
Running scot.mygov.pp.test.dashboard.DashboardServiceWebServiceIsolationCRUDTest
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec

Much easier to see where the first error of failure is.

Solution 5 - Maven 2

You can run maven with --fail-fast option:

> The -ff option is very useful for developers running interactive > builds who want to have rapid feedback during the development cycle.

an example can be:

mvn clean test -ff

http://books.sonatype.com/mvnref-book/reference/running-sect-options.html

Solution 6 - Maven 2

This doesn't solve the question exactly, but the solution that my workplace ultimately came up with was to use Atlassian's Clover to run specialized builds of just tests pertaining to changed code.

We have a Clover build that runs the tests for the changed code and in turn kicks off the full test build.

This has proven to be a satisfactory solution.

Solution 7 - Maven 2

A few ways to speed it up if not exactly what you need:

If it's a multi module build, add --fail-fast to the command line to drop out after the first module.
Look into failsafe to move long running integration tests onto a different phase of the lifecycle.
Look into a profile based solution for fast and slow tests - https://stackoverflow.com/questions/4389744/is-there-a-way-to-tell-surefire-to-skip-tests-in-a-certain-package.

Content Type	Original Author	Original Content on Stackoverflow
Question	Eliza Brock Marcum	View Question on Stackoverflow
Solution 1 - Maven 2	Captain Man	View Answer on Stackoverflow
Solution 2 - Maven 2	Pascal Thivent	View Answer on Stackoverflow
Solution 3 - Maven 2	mhaller	View Answer on Stackoverflow
Solution 4 - Maven 2	user2800708	View Answer on Stackoverflow
Solution 5 - Maven 2	Mike D3ViD Tyson	View Answer on Stackoverflow
Solution 6 - Maven 2	Eliza Brock Marcum	View Answer on Stackoverflow
Solution 7 - Maven 2	Benjamin Wootton	View Answer on Stackoverflow