Python unittests in Jenkins?
PythonUnit TestingJenkinsJunitXunitPython Problem Overview
How do you get Jenkins to execute python unittest cases?
Is it possible to JUnit style XML output from the builtin unittest
package?
Python Solutions
Solution 1 - Python
sample tests:
tests.py:
# tests.py
import random
try:
import unittest2 as unittest
except ImportError:
import unittest
class SimpleTest(unittest.TestCase):
@unittest.skip("demonstrating skipping")
def test_skipped(self):
self.fail("shouldn't happen")
def test_pass(self):
self.assertEqual(10, 7 + 3)
def test_fail(self):
self.assertEqual(11, 7 + 3)
JUnit with pytest
run the tests with:
py.test --junitxml results.xml tests.py
results.xml:
<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="1" name="pytest" skips="1" tests="2" time="0.097">
<testcase classname="tests.SimpleTest" name="test_fail" time="0.000301837921143">
<failure message="test failure">self = <tests.SimpleTest testMethod=test_fail>
def test_fail(self):
> self.assertEqual(11, 7 + 3)
E AssertionError: 11 != 10
tests.py:16: AssertionError</failure>
</testcase>
<testcase classname="tests.SimpleTest" name="test_pass" time="0.000109910964966"/>
<testcase classname="tests.SimpleTest" name="test_skipped" time="0.000164031982422">
<skipped message="demonstrating skipping" type="pytest.skip">/home/damien/test-env/lib/python2.6/site-packages/_pytest/unittest.py:119: Skipped: demonstrating skipping</skipped>
</testcase>
</testsuite>
JUnit with nose
run the tests with:
nosetests --with-xunit
nosetests.xml:
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="nosetests" tests="3" errors="0" failures="1" skip="1">
<testcase classname="tests.SimpleTest" name="test_fail" time="0.000">
<failure type="exceptions.AssertionError" message="11 != 10">
<![CDATA[Traceback (most recent call last):
File "/opt/python-2.6.1/lib/python2.6/site-packages/unittest2-0.5.1-py2.6.egg/unittest2/case.py", line 340, in run
testMethod()
File "/home/damien/tests.py", line 16, in test_fail
self.assertEqual(11, 7 + 3)
File "/opt/python-2.6.1/lib/python2.6/site-packages/unittest2-0.5.1-py2.6.egg/unittest2/case.py", line 521, in assertEqual
assertion_func(first, second, msg=msg)
File "/opt/python-2.6.1/lib/python2.6/site-packages/unittest2-0.5.1-py2.6.egg/unittest2/case.py", line 514, in _baseAssertEqual
raise self.failureException(msg)
AssertionError: 11 != 10
]]>
</failure>
</testcase>
<testcase classname="tests.SimpleTest" name="test_pass" time="0.000"></testcase>
<testcase classname="tests.SimpleTest" name="test_skipped" time="0.000">
<skipped type="nose.plugins.skip.SkipTest" message="demonstrating skipping">
<![CDATA[SkipTest: demonstrating skipping
]]>
</skipped>
</testcase>
</testsuite>
JUnit with nose2
You would need to use the nose2.plugins.junitxml
plugin. You can configure nose2
with a config file like you would normally do, or with the --plugin
command-line option.
run the tests with:
nose2 --plugin nose2.plugins.junitxml --junit-xml tests
nose2-junit.xml:
<testsuite errors="0" failures="1" name="nose2-junit" skips="1" tests="3" time="0.001">
<testcase classname="tests.SimpleTest" name="test_fail" time="0.000126">
<failure message="test failure">Traceback (most recent call last):
File "/Users/damien/Work/test2/tests.py", line 18, in test_fail
self.assertEqual(11, 7 + 3)
AssertionError: 11 != 10
</failure>
</testcase>
<testcase classname="tests.SimpleTest" name="test_pass" time="0.000095" />
<testcase classname="tests.SimpleTest" name="test_skipped" time="0.000058">
<skipped />
</testcase>
</testsuite>
JUnit with unittest-xml-reporting
Append the following to tests.py
if __name__ == '__main__':
import xmlrunner
unittest.main(testRunner=xmlrunner.XMLTestRunner(output='test-reports'))
run the tests with:
python tests.py
test-reports/TEST-SimpleTest-20131001140629.xml:
<?xml version="1.0" ?>
<testsuite errors="1" failures="0" name="SimpleTest-20131001140629" tests="3" time="0.000">
<testcase classname="SimpleTest" name="test_pass" time="0.000"/>
<testcase classname="SimpleTest" name="test_fail" time="0.000">
<error message="11 != 10" type="AssertionError">
<![CDATA[Traceback (most recent call last):
File "tests.py", line 16, in test_fail
self.assertEqual(11, 7 + 3)
AssertionError: 11 != 10
]]> </error>
</testcase>
<testcase classname="SimpleTest" name="test_skipped" time="0.000">
<skipped message="demonstrating skipping" type="skip"/>
</testcase>
<system-out>
<![CDATA[]]> </system-out>
<system-err>
<![CDATA[]]> </system-err>
</testsuite>
Solution 2 - Python
I would second using nose. Basic XML reporting is now built in. Just use the --with-xunit command line option and it will produce a nosetests.xml file. For example:
>nosetests --with-xunit
Then add a "Publish JUnit test result report" post build action, and fill in the "Test report XMLs" field with nosetests.xml (assuming that you ran nosetests in $WORKSPACE).
Solution 3 - Python
You can install the unittest-xml-reporting package to add a test runner that generates XML to the built-in unittest
.
We use pytest, which has XML output built in (it's a command line option).
Either way, executing the unit tests can be done by running a shell command.
Solution 4 - Python
I used nosetests. There are addons to output the XML for Jenkins
Solution 5 - Python
When using buildout we use collective.xmltestreport
to produce JUnit-style XML output, perhaps it's source code or the module itself could be of help.
Solution 6 - Python
python -m pytest --junit-xml=pytest_unit.xml source_directory/test/unit || true # tests may fail
Run this as shell from jenkins , you can get the report in pytest_unit.xml as artifact.