Haskell testing workflow

I just started a new Haskell project and wanted to set up a good testing workflow from the beginning. It seems like Haskell has a lot of excellent and unique testing tools and many different ways to integrate them.

I have looked into:

• HUnit
• QuickCheck
• benchpress
• HPC
• complexity

Which all seem to work very well in their domains, but I'm looking for a comprehensive approach to testing and was wondering what has worked well for other people.

Getting unit testing, code coverage, and benchmarks right is mostly about picking the right tools.

• test-framework provides a one-stop shop to run all your HUnit test-cases and QuickCheck properties all from one harness.
• Code coverage is built into GHC in the form of the HPC tool.
• Criterion provides some pretty great benchmarking machinery

I'll use as a running example a package that I just started enabling with unit testing, code coverage, and benchmarks:

http://github.com/ekmett/speculation

You can integrate your tests and benchmarks directly into your cabal file by adding sections for them, and masking them behind flags so that they don't make it so that every user of your library has to have access to (and want to use for themselves) the exact version of the testing tools you've chosen.

http://github.com/ekmett/speculation/blob/master/speculation.cabal

Then, you can tell cabal about how to run your test suite. As cabal test doesn't yet exist -- we have a student working on it for this year's summer of code! -- the best mechanism we have is Here is how to use cabal's user hook mechanism. This means switching to a 'Custom' build with cabal and setting up a testHook. An example of a testHook that runs a test program written with test-framework, and then applies hpc to profile it can be found here:

http://github.com/ekmett/speculation/blob/master/Setup.lhs

And then you can use test-framework to bundle up QuickCheck and HUnit tests into one program:

http://github.com/ekmett/speculation/blob/master/Test.hs

The cabal file there is careful to turn on -fhpc to enable code coverage testing, and then the testHook in Setup.lhs manually runs hpc and writes its output into your dist dir.

For benchmarking, the story is a little more manual, there is no 'cabal benchmark' option. You could wire your benchmarks into your test hook, but I like to run them by hand, since Criterion has so many graphical reporting options. You can add your benchmarks to the cabal file as shown above, give them separate compilation flags, hide them behind a cabal flag, and then use Criterion to do all the heavy lifting:

http://github.com/ekmett/speculation/blob/master/Benchmark.hs

You can then run your benchmarks from the command line and get pop-up KDE windows with benchmark results, etc.

Since in practice you're living in cabal anyways while developing Haskell code, it makes a lot of sense to integrate your toolchain with it.

Edit: Cabal test support now does exist. See http://www.haskell.org/cabal/release/cabal-latest/doc/users-guide/developing-packages.html#test-suites

The approach is advocate in RWH ch 11 and in XMonad is approximately:

• State all properties of the system in QuickCheck
• Show test coverage with HPC.
• Confirm space behavior with heap profiling.
• Confirm thread/parallel behavior with ThreadScope.
• Confirm microbenchmark behavior with Criterion.

Once your major invariants are established via QuickCheck, you can start refactoring, moving those tests into type invariants.

Practices to support your efforts:

• Run a simplified QuickCheck regression on every commit.
• Publish HPC coverage details.

The test-framework package is really awesome. You can easily integrate HUnit and QuickCheck tests, and get executables that run specified suites only, based on command-line flags, with multiple output targets.

Testing and profiling are different beasts though. For profiling I'd set up a separate executable that stresses just the section you want to profile, and just looking carefully at the results of profiling builds and runs (with -prof-auto-all for compilation and +RTS -p for a runtime flag).

For testing, I rely on HUnit and QuickCheck properties and use the Haskell Test Framework to collect all unit tests and all QuickCheck properties automatically.

Disclaimer: I'm the main developer of the Haskell Test Framework.