JUnit test with dynamic number of tests

In our project I have several JUnit tests that e.g. take every file from a directory and run a test on it. If I implement a testEveryFileInDirectory method in the TestCase this shows up as only one test that may fail or succeed. But I am interested in the results on each individual file. How can I write a TestCase / TestSuite such that each file shows up as a separate test e.g. in the graphical TestRunner of Eclipse? (Coding an explicit test method for each file is not an option.)

Compare also the question ParameterizedTest with a name in Eclipse Testrunner.

Take a look at Parameterized Tests in JUnit 4.

Actually I did this a few days ago. I'll try to explain ...

First build your test class normally, as you where just testing with one input file. Decorate your class with:

@RunWith(Parameterized.class)

Build one constructor that takes the input that will change in every test call (in this case it may be the file itself)

Then, build a static method that will return a Collection of arrays. Each array in the collection will contain the input arguments for your class constructor e.g. the file. Decorate this method with:

@Parameters

Here's a sample class.

@RunWith(Parameterized.class)
public class ParameterizedTest {

	private File file;
	
	public ParameterizedTest(File file) {
		this.file = file;
	}
	
	@Test
	public void test1() throws Exception {	}
	
	@Test
	public void test2() throws Exception {	}
	
	@Parameters
	public static Collection<Object[]> data() {
		// load the files as you want
		Object[] fileArg1 = new Object[] { new File("path1") };
		Object[] fileArg2 = new Object[] { new File("path2") };
		
		Collection<Object[]> data = new ArrayList<Object[]>();
		data.add(fileArg1);
		data.add(fileArg2);
 		return data;
	}
}

Also check this example

JUnit 3

public class XTest extends TestCase {

	public File file;

	public XTest(File file) {
		super(file.toString());
		this.file = file;
	}

	public void testX() {
		fail("Failed: " + file);
	}

}

public class XTestSuite extends TestSuite {

	public static Test suite() {
		TestSuite suite = new TestSuite("XTestSuite");
		File[] files = new File(".").listFiles();
		for (File file : files) {
			suite.addTest(new XTest(file));
		}
		return suite;
	}

}

JUnit 4

import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class TestY {

	@Parameters
	public static Collection<Object[]> getFiles() {
		Collection<Object[]> params = new ArrayList<Object[]>();
		for (File f : new File(".").listFiles()) {
			Object[] arr = new Object[] { f };
			params.add(arr);
		}
		return params;
	}

	private File file;

	public TestY(File file) {
		this.file = file;
	}

	@Test
	public void testY() {
		fail(file.toString());
	}

}

Junit 5 Parameterized Tests

JUnit 5 parameterized tests support this by allowing the use of a method as data source:

@ParameterizedTest
@MethodSource("fileProvider")
void testFile(File f) {
    // Your test comes here
}

static Stream<File> fileProvider() {
    return Arrays.asList(new File(".").list()).stream();
}

JUnit 5 DynamicTests

JUnit 5 also supports this through the notion of a DynamicTest, which is to be generated in a @TestFactory, by means of the static method dynamicTest.

import org.junit.jupiter.api.DynamicTest;
import org.junit.jupiter.api.TestFactory;
import static org.junit.jupiter.api.DynamicTest.dynamicTest;

import java.util.stream.Stream;

@TestFactory
public Stream<DynamicTest> testFiles() {
    return Arrays.asList(new File(".").list())
            .stream()
            .map((file) -> dynamicTest(
                    "Test for file: " + file,
                    () -> { /* Your test comes here */ }));
}

The tests run in your IDE (IntelliJ here) will be displayed like this:

Should be possible in JUnit 3 by inheriting from TestSuite and overriding the tests() method to list the files and for each return an instance of a subclass of TestCase that takes the filename as constructor parameter and has a test method that tests the file given in the constructor.

In JUnit 4 it might be even easier.

You could consider using JUnitParams library, so you would have a few more (cleaner) options:

@org.junit.runner.RunWith(junitparams.JUnitParamsRunner.class)
public class ParameterizedTest {

    @org.junit.Test
    @junitparams.Parameters(method = "data")
    public void test1(File file) throws Exception {  }

    @org.junit.Test
    @junitparams.Parameters(method = "data")
    public void test2(File file) throws Exception {  }

    public static File[] data() {
        return new File[] { new File("path1"), new File("path2") };
    }
}

---

@org.junit.runner.RunWith(junitparams.JUnitParamsRunner.class)
public class ParameterizedTest {

    @org.junit.Test
    @junitparams.Parameters(value = { "path1", "path2" })
    public void test1(String path) throws Exception {
        File file = new File(path);
    }

    @org.junit.Test
    @junitparams.Parameters(value = { "path1", "path2" })
    public void test2(String path) throws Exception {
        File file = new File(path);
    }
}

---

You can see more samples of usage here.

In addition about JUnitParams, why writting parameterized tests with it is easier and more readable:

> JUnitParams project adds a new runner to JUnit and provides much > easier and readable parametrised tests for JUnit >=4.6. > > Main differences to standard JUnit Parametrised runner: > > - more explicit - params are in test method params, not class fields > - less code - you don't need a constructor to set up parameters > - you can mix parametrised with non-parametrised methods in one class > - params can be passed as a CSV string or from a parameters provider class > - parameters provider class can have as many parameters providing methods as you want, so that you can group different cases > - you can have a test method that provides parameters (no external classes or statics anymore) > - you can see actual parameter values in your IDE (in JUnit's Parametrised it's only consecutive numbers of parameters)

If TestNG is an option, you could use Parameters with DataProviders.

Each individual file's test will have its result shown in the text-based report or Eclipse's TestNG plugin UI. The number of total tests run will count each of your files individually.

This behavior differs from JUnit Theories, in which all results are lumped under one "theory" entry and only count as 1 test. If you want separate result reporting in JUnit, you can try Parameterized Tests.

Test and inputs

public class FileTest {

    @DataProvider(name="files")
    public File[][] getFiles(){
        return new File[][] {
            { new File("file1") },
            { new File("file2") }
        };
        // or scan a directory
    }

    @Test(dataProvider="files")
    public void testFile(File file){
        //run tests on file
    }
}

Example output##

PASSED: testFile(file1)
PASSED: testFile(file2)

===============================================
    Default test
    Tests run: 2, Failures: 0, Skips: 0
===============================================

I had a similar problem and ended up writing a simple JUnit 4 runner that allows med to dynamically generate tests.

https://github.com/kimble/junit-test-factory