Running JavaScript unit tests headlessly in a Continuous Integration build

I have a webapp build plan running on a Continuous Integration system (Atlassian Bamboo 2.5). I need to incorporate QUnit-based JavaScript unit tests into the build plan so that on each build, the Javascript tests would be run and Bamboo would interpret the test results.

Preferably I would like to be able to make the build process "standalone" so that no connections to external servers would be required. Good ideas on how to accomplish this? The CI system running the build process is on an Ubuntu Linux server.

As I managed to come up with a solution myself, I thought it would be a good idea to share it. The approach might not be flawless, but it's the first one that seemed to work. Feel free to post improvements and suggestions.

What I did in a nutshell:

• Launch an instance of Xvfb, a virtual framebuffer
• Using JsTestDriver:
  • launch an instance of Firefox into the virtual framebuffer (headlessly)
  • capture the Firefox instance and run the test suite
  • generate JUnit-compliant test results .XML
• Use Bamboo to inspect the results file to pass or fail the build

I will next go through the more detailed phases. This is what my my directory structure ended up looking like:

lib/
JsTestDriver.jar
test/
qunit/
equiv.js
QUnitAdapter.js
jsTestDriver.conf
run_js_tests.sh
tests.js
test-reports/
build.xml

On the build server:

• Install Xvfb (apt-get install Xvfb)
• Install Firefox (apt-get install firefox)

Into your application to be built:

• Install JsTestDriver: http://code.google.com/p/js-test-driver/
  • add the QUnit adapters equiv.js and QUnitAdapter.js
  • configure JsTestDriver (jsTestDriver.conf):

server: http://localhost:4224

load:
Load QUnit adapters (may be omitted if QUnit is not used)

qunit/equiv.js
qunit/QUnitAdapter.js

Tests themselves (you'll want to add more files)

tests.js

Create a script file for running the unit tests and generating test results (example in Bash, run_js_tests.sh):

#!/bin/bash
# directory to write output XML (if this doesn't exist, the results will not be generated!)
OUTPUT_DIR="../test-reports"
mkdir $OUTPUT_DIR

XVFB=`which Xvfb`
if [ "$?" -eq 1 ];
then
	echo "Xvfb not found."
	exit 1
fi

FIREFOX=`which firefox`
if [ "$?" -eq 1 ];
then
	echo "Firefox not found."
	exit 1
fi

$XVFB :99 -ac &    # launch virtual framebuffer into the background
PID_XVFB="$!"      # take the process ID
export DISPLAY=:99 # set display to use that of the xvfb

# run the tests
java -jar ../lib/JsTestDriver.jar --config jsTestDriver.conf --port 4224 --browser $FIREFOX --tests all --testOutput $OUTPUT_DIR

kill $PID_XVFB     # shut down xvfb (firefox will shut down cleanly by JsTestDriver)
echo "Done."

Create an Ant target that calls the script:

<target name="test">		
	<exec executable="cmd" osfamily="windows">
		<!-- This might contain something different in a Windows environment -->
	</exec>
	
	<exec executable="/bin/bash" dir="test" osfamily="unix">
		<arg value="run_js_tests.sh" />
	</exec>
</target>	

Finally, tell the Bamboo build plan to both invoke the test target and look for JUnit test results. Here the default "**/test-reports/*.xml" will do fine.

For anyone interested in running their Jasmine BDD specs headlessly in maven, you might be interested in the jasmine-maven-plugin I maintain:

http://github.com/searls/jasmine-maven-plugin

As an alternative, you could also try TestSwarm. I've got it up and running using QUnit to run my JS tests.

• http://github.com/jquery/testswarm
• http://swarm.jquery.org/

I've played around with a number of solutions over the past year but I didn't find anything in the ballpark of Karma (formerly testacular). Give it a try

http://karma-runner.github.com/

I have used maven and junit to call rhino. It is not elegant, but I use it to test basic services and utility code.

It requires mocking unsupported classes, like XHR with Java libraries.

I found that it is best code everything in javascript (tests, etc) and only use junit for build organization and a hook into the CI.

I'd like to see if JsTestDriver can do it though. Or mocha w/ a junit reporter.

You may be able to use rhino, the headless browser, to run your unit tests on your CI machine. Of course, the disadvantage here is that it won't find bugs specific to browser X... but it does beat installing 2-3 OSes on your CI box, to cover all the main platforms...

But yes, this kind of sucks... but it might work just well enough in a CI scenario.

JS Test Runner is a pretty good solution. It uses PhantomJS and QUnit.