How to run all tests belonging to a certain Category in JUnit 4

JUnit 4.8 contains a nice new feature called "Categories" that allows you to group certain kinds of tests together. This is very useful, e.g. to have separate test runs for slow and fast tests. I know the stuff mentioned in JUnit 4.8 release notes, but would like to know how I can actually run all the tests annotated with certain category.

The JUnit 4.8 release notes show an example suite definition, where SuiteClasses annotation selects the tests from certain category to run, like this:

@RunWith(Categories.class)
@IncludeCategory(SlowTests.class)
@SuiteClasses( { A.class, B.class }) // Note that Categories is a kind of Suite
public class SlowTestSuite {
  // Will run A.b and B.c, but not A.a
}

Does anyone know how I could run all the tests in SlowTests category? It seems that you must have the SuiteClasses annotation...

I found out one possible way to achieve what I want, but I don't consider this to be the best possible solution as it relies on ClassPathSuite library that is not part of JUnit.

I define the test suite for slow tests like this:

@RunWith(Categories.class)
@Categories.IncludeCategory(SlowTests.class)
@Suite.SuiteClasses( { AllTests.class })
public class SlowTestSuite {
}

AllTests class is defined like this:

@RunWith(ClasspathSuite.class)
public class AllTests {
}

I had to use ClassPathSuite class from ClassPathSuite project here. It will find all the classes with tests.

Here are some of the main differences between TestNG and JUnit when it comes to groups (or categories, like JUnit calls them):

• JUnit's are typed (annotations) while TestNG's are strings. I made this choice because I wanted to be able to use regular expressions when running tests, for example "run all the tests that belong to the group "database*". Also, having to create a new annotation whenever you need to create a new category is annoying, although it has the benefit that an IDE will tell you right away where this category is used (TestNG shows you this in its reports).

• TestNG separates very clearly your static model (the code of your tests) from the runtime model (which tests get run). If you want to run the groups "front-end" first and then "servlets", you can do this without having to recompile anything. Because JUnit defines groups in annotations and you need to specify these categories as parameters to the runner, you usually have to recompile your code whenever you want to run a different set of categories, which defeats the purpose in my opinion.

One downside to Kaitsu's solution is that Eclipse will run your tests twice, and the SlowTests 3 times, when running all the tests in a project. This is because the Eclipse will run all the tests, then the AllTests suite, then the SlowTestSuite.

Here is a solution that involves creating subclasses of the Kaitsu solution test runners to skip the suites unless a certain system property is set. A shameful hack, but all I have come up with so far.

@RunWith(DevFilterClasspathSuite.class)
public class AllTests {}

.

@RunWith(DevFilterCategories.class)
@ExcludeCategory(SlowTest.class)
@SuiteClasses(AllTests.class)
public class FastTestSuite
{
}

.

public class DevFilterCategories extends Suite
{
	private static final Logger logger = Logger
		.getLogger(DevFilterCategories.class.getName());
	public DevFilterCategories(Class<?> suiteClass, RunnerBuilder builder) throws InitializationError {
		super(suiteClass, builder);
		try {
			filter(new CategoryFilter(getIncludedCategory(suiteClass),
					getExcludedCategory(suiteClass)));
			filter(new DevFilter());
		} catch (NoTestsRemainException e) {
			logger.info("skipped all tests");
		}
		assertNoCategorizedDescendentsOfUncategorizeableParents(getDescription());
	}

	private Class<?> getIncludedCategory(Class<?> klass) {
		IncludeCategory annotation= klass.getAnnotation(IncludeCategory.class);
		return annotation == null ? null : annotation.value();
	}

	private Class<?> getExcludedCategory(Class<?> klass) {
		ExcludeCategory annotation= klass.getAnnotation(ExcludeCategory.class);
		return annotation == null ? null : annotation.value();
	}

	private void assertNoCategorizedDescendentsOfUncategorizeableParents(Description description) throws InitializationError {
		if (!canHaveCategorizedChildren(description))
			assertNoDescendantsHaveCategoryAnnotations(description);
		for (Description each : description.getChildren())
			assertNoCategorizedDescendentsOfUncategorizeableParents(each);
	}
	
	private void assertNoDescendantsHaveCategoryAnnotations(Description description) throws InitializationError {			
		for (Description each : description.getChildren()) {
			if (each.getAnnotation(Category.class) != null)
				throw new InitializationError("Category annotations on Parameterized classes are not supported on individual methods.");
			assertNoDescendantsHaveCategoryAnnotations(each);
		}
	}

	// If children have names like [0], our current magical category code can't determine their
	// parentage.
	private static boolean canHaveCategorizedChildren(Description description) {
		for (Description each : description.getChildren())
			if (each.getTestClass() == null)
				return false;
		return true;
	}
}

.

public class DevFilterClasspathSuite extends ClasspathSuite
{
	private static final Logger logger = Logger
		.getLogger(DevFilterClasspathSuite.class.getName());
	public DevFilterClasspathSuite(Class<?> suiteClass, RunnerBuilder builder) 
		throws InitializationError {
		super(suiteClass, builder);
		try
		{
			filter(new DevFilter());
		} catch (NoTestsRemainException e)
		{
			logger.info("skipped all tests");
		}
	}
}

.

public class DevFilter extends Filter
{
	private static final String RUN_DEV_UNIT_TESTS = "run.dev.unit.tests";

	@Override
	public boolean shouldRun(Description description)
	{
		return Boolean.getBoolean(RUN_DEV_UNIT_TESTS);
	}

	@Override
	public String describe()
	{
		return "filter if "+RUN_DEV_UNIT_TESTS+" system property not present";
	}
}

So, in your FastTestSuite launcher, just add -Drun.dev.unit.tests=true to the VM arguments. (Note that this solution references a fast test suite instead of a slow one.)

To run categorized tests without specifying all of them explicily in @Suite.SuiteClasses annotation you can provide your own implementation of Suite. For example a org.junit.runners.ParentRunner can be extended. Instead of using an array of classes provided by @Suite.SuiteClasses, new implementation should perform search for categorized tests in classpath.

See this project as an example of such approach. Usage:

@Categories(categoryClasses = {IntegrationTest.class, SlowTest.class})
@BasePackage(name = "some.package")
@RunWith(CategorizedSuite.class)
public class CategorizedSuiteWithSpecifiedPackage {
   
}

I am not sure, what exactly your problem is.

Just add all the tests to a suite (or hirachy of suites). Then use the Categories Runner and Include/ExcludeCategory annotation, to specify the categories you want to run.

A good idea might be to have one suite containing all the tests, and a couple of seperate suites referring to the first one, specifying the different set of Categories you neeed.

Not a direct answer to your problem, but maybe the general approach could be improved...

Why are your tests slow? Maybe the set-up lasts long (database, I/O etc.), maybe the tests are testing too much? If this is the case I would seperate the real unit-tests from the "long-running" ones, which often indeed are integration tests.

In my setups I have staging env, where unit-tests are run often and integration-tests constantly but more rarely (e.g. after each commit in version control). I have never worked with grouping for unit tests, because they should be loosely coupled alltogether. I only work with grouping and relationship of test-cases in integration-test setups (but with TestNG).

But good to know that JUnit 4.8 introduced some grouping features.