Using CMake, how do I get verbose output from CTest?
CmakeCtestCmake Problem Overview
I'm using CMake to build my project. I have added a unit test binary which is using the Boost unit testing framework. This one binary contains all of the unit tests. I've added that binary to be run by CTest:
ADD_EXECUTABLE( tftest test-main.cpp )
ENABLE_TESTING()
ADD_TEST( UnitTests tftest)
But the build output in Visual Studio only shows the result of running CTest:
Start 1: UnitTests
1/1 Test #1: UnitTests ................***Failed 0.05 sec
0% tests passed, 1 tests failed out of 1
This is not very helpful, because I can't see which test failed. If I run ctest manually from the command line with --verbose
I get the output from a Boost unit test which tells what actually failed:
1: Test command: tftest.exe
1: Test timeout computed to be: 9.99988e+006
1: Running 4 test cases...
1: test-main.cpp(20): error in "sanity_check3": check 1 == 2 failed
1:
1: *** 1 failure detected in test suite "Master Test Suite"
1/1 Test #1: UnitTests ................***Failed 0.00 sec
So, what do I need to change in the CMakeLists.txt to have CTest run with --verbose
at all times? Is there a better way to use Boost unit tests with CMake/CTest?
Cmake Solutions
Solution 1 - Cmake
You can set the environment variable CTEST_OUTPUT_ON_FAILURE
, which will show you any output from the test program whenever the test fails. One way to do this when using Makefiles and the command line would be as follows:
env CTEST_OUTPUT_ON_FAILURE=1 make check
This Stack Overflow question and answer shows how to set environment variables in Visual Studio.
Solution 2 - Cmake
You could call ctest
directly, after cmaking and making your project.
ctest --verbose
Solution 3 - Cmake
-
You can check the
Testing/Temporary
subfolder. It is automatically created after running make test. This folder contains two files:LastTest.log
andLastTestsFailed.log
.LastTest.log
contains desired output for run tests.LastTestFailed.log
contains names of failed tests. So you can check them manually after executingmake test
. -
The second way is to get ctest to show you the content of log files after running tests:
-
place in build dir (from which you run
make test
) file CTestCustom.ctest (you can do it with configure file command, for example) with following contents
CTEST_CUSTOM_POST_TEST("cat Testing/Temporary/LastTest.log")
Instead of cat you may use whatever Windows cmd command that does similar things.
- run
make test
again and get profit!
additional info about customizing ctest you can find here. Just step to "Customizing cmake" section. Good luck!
Solution 4 - Cmake
There is a very simple solution (which for some reason is difficult to find via Google Search):
ctest --output-on-failure
If you use CMake with Visual Studio's open folder function you can add the
"ctestCommandArgs": "--output-on-failure"
setting to your build configuration.
Solution 5 - Cmake
I had to add "check" target by myself. "make tests" does nothing by some reason. So what I did (as was suggest somewhere on stackoverflow) - I added this target manually. To get verbose output I just wrote it like:
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND} --verbose)
Solution 6 - Cmake
Solution 7 - Cmake
This makes test output more verbose:
make test ARGS="-V"
Solution 8 - Cmake
My approach is a combination of the answers from ony, from zbyszek, and from tarc. I use the ${CMAKE_COMMAND}
variable (which is set to the absolute path to the invoked cmake executable) with the -E env CTEST_OUTPUT_ON_FAILURE=1
argument to invoke the actual ctest command using ${CMAKE_CTEST_COMMAND} -C $<CONFIG>
. To help clarify what is going on, I start with three cmake -E echo
commands to show the current working directory and the ctest command to be invoked. Here is how I call add_custom_target
.
add_custom_target(check
${CMAKE_COMMAND} -E echo CWD=${CMAKE_BINARY_DIR}
COMMAND ${CMAKE_COMMAND} -E echo CMD=${CMAKE_CTEST_COMMAND} -C $<CONFIG>
COMMAND ${CMAKE_COMMAND} -E echo ----------------------------------
COMMAND ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1
${CMAKE_CTEST_COMMAND} -C $<CONFIG>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
DEPENDS ALL_BUILD
)
This plays nice with the MSVC IDE where any test errors are shown as clickable compilation errors. See cmake -E env for documentation of the cmake -E
portable command line tool mode. I also add a dependency on ALL_BUILD
so that all projects will be built before invoking the check
target. (On Linux builds, one may need to replace ALL_BUILD
with ALL
; I have not tested this on Linux yet.)
Solution 9 - Cmake
For people using Visual Studio, here another variation (hack) on the theme:
cmake -E env CTEST_OUTPUT_ON_FAILURE=1 cmake --build . --target RUN_TESTS
Solution 10 - Cmake
ctest -VV
or ctest --extra-verbose
From documentation:
> Enable more verbose output from tests. > > Test output is normally suppressed and only summary information is > displayed. This option will show even more test output.
Solution 11 - Cmake
There's now a CMake variable that allows you to modify the behaviour of make test
. CMAKE_CTEST_ARGUMENTS lets you set a list of arguments to pass to ctest when run via make test
.
So adding this to your CMake file:
set(CMAKE_CTEST_ARGUMENTS "--verbose")
Means CTest will always run verbose. Or for just the output of the failed tests, use:
set(CMAKE_CTEST_ARGUMENTS "--output-on-failure")
Solution 12 - Cmake
to show the result with XML file you have to execute the test with the following command
~$ ctest -T Test
and we found the result in the Testing/1234123432/test.xml and other files are generated too in Testing Folder