Comparison of C++ unit test frameworks

I know there are already a few questions regarding recommendations for C++ unit test frameworks, but all the answers did not help as they just recommend one of the frameworks but do not provide any information about a (feature) comparison.

I think the most interesting frameworks are CppUnit, Boost and the new Google testing framework. Has anybody done any comparison yet?

A new player is Google Test (also known as Google C++ Testing Framework) which is pretty nice though.

#include <gtest/gtest.h>

TEST(MyTestSuitName, MyTestCaseName) {
    int actual = 1;
    EXPECT_GT(actual, 0);
    EXPECT_EQ(1, actual) << "Should be equal to one";
}

Main features:

• Portable
• Fatal and non-fatal assertions
• Easy assertions informative messages: ASSERT_EQ(5, Foo(i)) << " where i = " << i;
• Google Test automatically detects your tests and doesn't require you to enumerate them in order to run them
• Make it easy to extend your assertion vocabulary
• Death tests (see advanced guide)
• SCOPED_TRACE for subroutine loops
• You can decide which tests to run
• XML test report generation
• Fixtures / Mock / Templates...

I've just pushed my own framework, CATCH, out there. It's still under development but I believe it already surpasses most other frameworks. Different people have different criteria but I've tried to cover most ground without too many trade-offs. Take a look at my linked blog entry for a taster. My top five features are:

• Header only
• Auto registration of function and method based tests
• Decomposes standard C++ expressions into LHS and RHS (so you don't need a whole family of assert macros).
• Support for nested sections within a function based fixture
• Name tests using natural language - function/ method names are generated

It also has Objective-C bindings. The project is hosted on Github

See this question for some discussion.

They recommend the articles: Exploring the C++ Unit Testing Framework Jungle, By Noel Llopis. And the more recent: C++ Test Unit Frameworks

I have not found an article that compares googletest to the other frameworks yet.

Boost Test Library is a very good choice especially if you're already using Boost.

// TODO: Include your class to test here.
#define BOOST_TEST_MODULE MyTest
#include <boost/test/unit_test.hpp>

BOOST_AUTO_TEST_CASE(MyTestCase)
{
	// To simplify this example test, let's suppose we'll test 'float'.
	// Some test are stupid, but all should pass.
	float x = 9.5f;

	BOOST_CHECK(x != 0.0f);
	BOOST_CHECK_EQUAL((int)x, 9);
	BOOST_CHECK_CLOSE(x, 9.5f, 0.0001f); // Checks differ no more then 0.0001%
}

It supports:

• Automatic or manual tests registration
• Many assertions
• Automatic comparison of collections
• Various output formats (including XML)
• Fixtures / Templates...

PS: I wrote an article about it that may help you getting started: C++ Unit Testing Framework: A Boost Test Tutorial

Wikipedia has a comprehensive list of unit testing frameworks, with tables that identify features supported or not.

I've recently released xUnit++, specifically as an alternative to Google Test and the Boost Test Library (view the comparisons). If you're familiar with xUnit.Net, you're ready for xUnit++.

#include "xUnit++/xUnit++.h"

FACT("Foo and Blah should always return the same value")
{
    Check.Equal("0", Foo()) << "Calling Foo() with no parameters should always return \"0\".";
    Assert.Equal(Foo(), Blah());
}

THEORY("Foo should return the same value it was given, converted to string", (int input, std::string expected),
    std::make_tuple(0, "0"),
    std::make_tuple(1, "1"),
    std::make_tuple(2, "2"))
{
    Assert.Equal(expected, Foo(input));
}

Main features:

• Incredibly fast: tests run concurrently.

• Portable

• Automatic test registration

• Many assertion types (Boost has nothing on xUnit++)

• Compares collections natively.

• Assertions come in three levels:

  • fatal errors
  • non-fatal errors
  • warnings

• Easy assert logging: Assert.Equal(-1, foo(i)) << "Failed with i = " << i;

• Test logging: Log.Debug << "Starting test"; Log.Warn << "Here's a warning";

• Fixtures

• Data-driven tests (Theories)

• Select which tests to run based on:

  • Attribute matching
  • Name substring matchin
  • Test Suites

CppUTest - very nice, light weight framework with mock libraries. Worthwhile taking a closer look.

CPUnit (<http://cpunit.sourceforge.net>;) is a framework that is similar to Google Test, but which relies on less macos (asserts are functions), and where the macros are prefixed to avoid the usual macro pitfall. Tests look like:

#include <cpunit>

namespace MyAssetTest {
    using namespace cpunit;

    CPUNIT_FUNC(MyAssetTest, test_stuff) {
        int some_value = 42;
        assert_equals("Wrong value!", 666, some_value);
    }
    
    // Fixtures go as follows:
    CPUNIT_SET_UP(MyAssetTest) {
        // Setting up suite here...
        // And the same goes for tear-down.
    }

}

They auto-register, so you need not more than this. Then it is just compile and run. I find using this framework very much like using JUnit, for those who have had to spend some time programming Java. Very nice!

There are some relevant C++ unit testing resources at http://www.progweap.com/resources.html

API Sanity Checker — test framework for C/C++ libraries:

>An automatic generator of basic unit tests for a shared C/C++ library. It is able to generate reasonable (in most, but unfortunately not all, cases) input data for parameters and compose simple ("sanity" or "shallow"-quality) test cases for every function in the API through the analysis of declarations in header files. > >The quality of generated tests allows to check absence of critical errors in simple use cases. The tool is able to build and execute generated tests and detect crashes (segfaults), aborts, all kinds of emitted signals, non-zero program return code and program hanging.

Unique features in comparison with CppUnit, Boost and Google Test:

• Automatic generation of test data and input arguments (even for complex data types)
• Modern and highly reusable specialized types instead of fixtures and templates