Unit Testing Embedded Software

What best practices have you used in unit testing embedded software that are peculiar to embedded systems?

Embedded software may have come a long way in the last 10 years but we generally did the following:

• for algorithms that didn't depend on the target hardware, we simply had unit tests that were built and tested on a non-embedded platform.
• for stuff that did require the hardware, unit tests were conditionally compiled into the code to use whatever hardware was available. In our case, it was a serial port on the target pushing the results to another, more capable, machine where the tests were checked for correctness.
• Depending on the hardware, you could sometimes dummy up a "virtual" device on a non-embedded platform. This usually consisted of having another thread of execution (or signal function) changing memory used by the program. Useful for memory mapped I/O but not IRQs and such.
• typically, you could only unit test a small subset of the complete code at a time (due to memory constraints).
• for testing of time-sensitive things, we didn't. Plain and simple. The hardware we used (8051 and 68302) was not always functional if it ran too slow. That sort of debugging had to be done initially with a CRO (oscilloscope) and (when we had some more money) an ICE (in-circuit emulator).

Hopefully the situation has improved since I last did it. I wouldn't wish that pain on my worst enemy.

There can be a lot to be gained by unit testing in a PC environment (compiling your code with a PC C compiler and running your code in a PC unit testing framework), with several provisos:

1. This doesn't apply to testing your low-level code, including start-up code, RAM tests, hardware drivers. You'll have to use more direct unit testing of those.
2. Your embedded system's compiler has to be trustworthy, so you're not hunting for bugs created by the compiler.
3. Your code has to be layered architecture, with hardware abstraction. You may need to write hardware driver simulators for your PC unit testing framework.
4. You should always use the stdint.h types such as uint16_t rather than plain unsigned int etc.

We've followed these rules, and found that after unit testing the application-layer code in a PC unit test framework, we can have a good amount of confidence that it works well.

Advantages of unit testing on the PC platform:

1. You don't face the problem of running out of ROM space on your embedded platform due to adding a unit testing framework.
2. The compile-link-run cycle is typically faster and simpler on the PC platform (and avoids the 'write/download' step which can potentially be several minutes).
3. You have more options for visualising progress (some embedded applications have limited I/O peripherals), storing input/output data for analysis, running more time-consuming tests.
4. You can use readily available PC-based unit test frameworks that aren't available/suitable for an embedded platform.

Embedded systems is a wide topic but in general, let's think of it as a specific-purpose product that combines both hardware and software. My embedded background is from mobile phones which is just a small subset of all embedded systems. I'll try to keep the following points a bit on the abstract side:

• Abstract out hardware dependencies whenever possible. This way you can run your unit tests on mocked "hardware" and also test various rare/exceptional cases that would be harder to test on target. To prevent abstraction costs, you can use e.g. conditional compilation.

• Have as little as possible depend on the hardware.

• Unit tests running on an emulator or cross-compiler environment still does not guarantee the code works on target hardware. You must test on target as well. Test on target as early as possible.

You might want to check out Test Driven Development for Embedded C by James W. Grenning. The book is scheduled to be published in August 2010, but the beta book is available now on The Pragmatic Bookshelf.

Voice of inexperience here, but this is something I've been thinking about as well lately. It seems to me that the best approach would be either

A) Write as much of your hardware-independent application code as you can in a PC environment, before you write it on the target, and write your unit tests at the same time (doing it this on the PC first should help force you to separate the hardware-independent stuff). This way you can use your choice of unit testers, then test the hardware-dependent stuff the old fashioned way - with RS-232 and/or oscilloscopes and I/O pins signalling time-dependent data, depending on how fast it has to run.

B) Write it all on the target hardware, but have a make target to conditionally compile a unit test build that will run unit tests and output the results (or data that can be analyzed for results) via RS-232 or some other means. If you don't have a lot of memory, this can be tricky.

Edit 7/3/2009 I just had another thought about how to unit test hardware dependent stuff. If your hardware events are happening too fast to record with RS-232, but you don't want to manually sift through tons of oscilloscope data checking to see if your I/O pin flags rise and fall as expected, you can use a PC card with integrated DIO (such as National Instruments' line of Data Acquisition cards) to automatically evaluate the timing of those signals. You would then just need to write the software on your PC to control the data acquisition card to synchronize with the currently running unit test.

We manage to get quite a bit of hardware dependent code tested using a simulator, we use Keil's simulator and IDE (not affiliated just use their tools). We write the simulator scripts to drive the 'hardware' in a way we expect it to react and we are able to pretty reliably test our working code. Granted it can take some effort to model the hardware for some tests, but for most things this works very well and allows us to get a lot done without any hardware available. We have been able to get near complete system working in the simulator before having access to hardware and have had very few issues to deal with once putting the code on the real thing. This can also significantly speed up production of code since everything can be done on the PC with the more in-depth debugger available while simulating the chip vs trying to do everything on the hardware.

Have gotten this to work reliably for complex control systems, memory interfaces, custom SPI driven ICs and even a mono-display.

There's lots of good answers here, some things that haven't been mentioned is to have diagnostic code running in order to:

Log HAL events (interrupts, bus messages, etc)

Have code to keep track of your resources, (all active semaphores, thread activity)

Have a capture ram mechanism to copy the heap and memory content to persistent storage (hard disk or equivalent) to detect and debug deadlocks, livelocks, memory leaks, buffer overflows, etc.

When I was facing this last year I really wanted to test on the embedded platform itself. I was developing a library and I was using the RTOS calls and other features of the embedded platform. There wasn't anything specific available so I adapted the UnitTest++ code to my purposes. I program on the NetBurner family and since it has an embedded web server, it was pretty straight forward to write a web based GUI test runner that give the classic RED/GREEN feedback. It turned out pretty well, and now unit testing is much easier and I feel much more confident knowing the code works on the actual hardware. I even use the unit testing framework to do integration tests. At first I mocks/stub the hardware and inject that interface to test. But eventually I write some man-in-the-loop tests that exercise the actual hardware. It turns out to be a much simpler way to learn about the hardware and have an easy way to recover from embedded traps. Since the tests all run from AJAX callbacks to the web server a trap only happens as the result of manually invoking a test and the system always restarts cleanly a few seconds after the trap.

The NetBurner is fast enough that the write/compile/download/run test cycle is about 30 seconds.

Lots of embedded processors are available on eval boards, so although you may not have your real i/o devices, often you can execute a good deal of your algorithms and logic on one of these kinds of things, often w/hardware debugging available via jtag. And 'unit' tests usually are more about your logic than your i/o anyway. Problem is usually getting your test artifacts back out of one of these environments.

Split the code between device-dependent & device-independent. The independent code can be unit-tested without too much pain. The dependent code will simply need to be hand-tested until you have a smooth communications interface.

If you're writing the communications interface, I'm sorry.