Unit testing Anti-patterns catalogue
Unit TestingTddAnti PatternsUnit Testing Problem Overview
anti-pattern : there must be at least two key elements present to formally distinguish an actual anti-pattern from a simple bad habit, bad practice, or bad idea:
- Some repeated pattern of action, process or structure that initially appears to be beneficial, but ultimately produces more bad consequences than beneficial results, and
- A refactored solution that is clearly documented, proven in actual practice and repeatable.
Vote for the TDD anti-pattern that you have seen "in the wild" one time too many.
The blog post by James Carr and Related discussion on testdrivendevelopment yahoogroup
If you've found an 'unnamed' one.. post 'em too. One post per anti-pattern please to make the votes count for something.
My vested interest is to find the top-n subset so that I can discuss 'em in a lunchbox meet in the near future.
Unit Testing Solutions
Solution 1 - Unit Testing
Second Class Citizens - test code isn't as well refactored as production code, containing a lot of duplicated code, making it hard to maintain tests.
Solution 2 - Unit Testing
The Free Ride / Piggyback -- James Carr, Tim Ottinger
Rather than write a new test case method to test another/distinct feature/functionality, a new assertion (and its corresponding actions i.e. Act steps from AAA) rides along in an existing test case.
Solution 3 - Unit Testing
Happy Path
The test stays on happy paths (i.e. expected results) without testing for boundaries and exceptions.
Solution 4 - Unit Testing
The Local Hero
A test case that is dependent on something specific to the development environment it was written on in order to run. The result is the test passes on development boxes, but fails when someone attempts to run it elsewhere.
The Hidden Dependency
Closely related to the local hero, a unit test that requires some existing data to have been populated somewhere before the test runs. If that data wasn’t populated, the test will fail and leave little indication to the developer what it wanted, or why… forcing them to dig through acres of code to find out where the data it was using was supposed to come from.
Sadly seen this far too many times with ancient .dlls which depend on nebulous and varied .ini files which are constantly out of sync on any given production system, let alone extant on your machine without extensive consultation with the three developers responsible for those dlls. Sigh.
Solution 5 - Unit Testing
Chain Gang
A couple of tests that must run in a certain order, i.e. one test changes the global state of the system (global variables, data in the database) and the next test(s) depends on it.
You often see this in database tests. Instead of doing a rollback in teardown(), tests commit their changes to the database. Another common cause is that changes to the global state aren't wrapped in try/finally blocks which clean up should the test fail.
Solution 6 - Unit Testing
The Mockery
Sometimes mocking can be good, and handy. But sometimes developers can lose themselves and in their effort to mock out what isn’t being tested. In this case, a unit test contains so many mocks, stubs, and/or fakes that the system under test isn’t even being tested at all, instead data returned from mocks is what is being tested.
Source: James Carr's post.
Solution 7 - Unit Testing
The Silent Catcher -- Kelly?
A test that passes if an exception is thrown.. even if the exception that actually occurs is one that is different than the one the developer intended.
See Also: Secret Catcher
[Test]
[ExpectedException(typeof(Exception))]
public void ItShouldThrowDivideByZeroException()
{
// some code that throws another exception yet passes the test
}
Solution 8 - Unit Testing
The Inspector
A unit test that violates encapsulation in an effort to achieve 100% code coverage, but knows so much about what is going on in the object that any attempt to refactor will break the existing test and require any change to be reflected in the unit test.
'how do I test my member variables without making them public... just for unit-testing?'
Solution 9 - Unit Testing
Excessive Setup -- James Carr
A test that requires a huge setup in order to even begin testing. Sometimes several hundred lines of code are used to prepare the environment for one test, with several objects involved, which can make it difficult to really ascertain what is tested due to the “noise” of all of the setup going on. (Src: James Carr's post)
Solution 10 - Unit Testing
Anal Probe
A test which has to use insane, illegal or otherwise unhealthy ways to perform its task like: Reading private fields using Java's setAccessible(true) or extending a class to access protected fields/methods or having to put the test in a certain package to access package global fields/methods.
If you see this pattern, the classes under test use too much data hiding.
The difference between this and The Inspector is that the class under test tries to hide even the things you need to test. So your goal is not to achieve 100% test coverage but to be able to test anything at all. Think of a class that has only private fields, a run() method without arguments and no getters at all. There is no way to test this without breaking the rules.
Comment by Michael Borgwardt: This is not really a test antipattern, it's pragmatism to deal with deficiencies in the code being tested. Of course it's better to fix those deficiencies, but that may not be possible in the case of 3rd party libraries.
Aaron Digulla: I kind of agree. Maybe this entry is really better suited for a "JUnit HOWTO" wiki and not an antipattern. Comments?
Solution 11 - Unit Testing
The Test With No Name -- Nick Pellow
The test that gets added to reproduce a specific bug in the bug tracker and whose author thinks does not warrant a name of its own. Instead of enhancing an existing, lacking test, a new test is created called testForBUG123.
Two years later, when that test fails, you may need to first try and find BUG-123 in your bug tracker to figure out the test's intent.
Solution 12 - Unit Testing
The Slow Poke
A unit test that runs incredibly slow. When developers kick it off, they have time to go to the bathroom, grab a smoke, or worse, kick the test off before they go home at the end of the day. (Src: James Carr's post)
a.k.a. the tests that won't get run as frequently as they should
Solution 13 - Unit Testing
The Butterfly
You have to test something which contains data that changes all the time, like a structure which contains the current date, and there is no way to nail the result down to a fixed value. The ugly part is that you don't care about this value at all. It just makes your test more complicated without adding any value.
The bat of its wing can cause a hurricane on the other side of the world. -- Edward Lorenz, The Butterfly Effect
Solution 14 - Unit Testing
Wait and See
A test that runs some set up code and then needs to 'wait' a specific amount of time before it can 'see' if the code under test functioned as expected. A testMethod that uses Thread.sleep() or equivalent is most certainly a "Wait and See" test.
Typically, you may see this if the test is testing code which generates an event external to the system such as an email, an http request or writes a file to disk.
Such a test may also be a Local Hero since it will FAIL when run on a slower box or an overloaded CI server.
The Wait and See anti-pattern is not to be confused with The Sleeper.
Solution 15 - Unit Testing
The Flickering Test (Source : Romilly Cocking)
A test which just occasionally fails, not at specific times, and is generally due to race conditions within the test. Typically occurs when testing something that is asynchronous, such as JMS.
Possibly a super set to the 'Wait and See' anti-pattern and 'The Sleeper' anti-pattern.
The build failed, oh well, just run the build again. -- Anonymous Developer
Solution 16 - Unit Testing
Inappropriately Shared Fixture -- Tim Ottinger
Several test cases in the test fixture do not even use or need the setup / teardown. Partly due to developer inertia to create a new test fixture... easier to just add one more test case to the pile
Solution 17 - Unit Testing
The Giant
A unit test that, although it is validly testing the object under test, can span thousands of lines and contain many many test cases. This can be an indicator that the system under tests is a God Object (James Carr's post).
A sure sign for this one is a test that spans more than a a few lines of code. Often, the test is so complicated that it starts to contain bugs of its own or flaky behavior.
Solution 18 - Unit Testing
I'll believe it when I see some flashing GUIs
An unhealthy fixation/obsession with testing the app via its GUI 'just like a real user'
> Testing business rules through the GUI
> is a terrible form of coupling. If
> you write thousands of tests through
> the GUI, and then change your GUI,
> thousands of tests break.
> Rather, test only GUI things through the GUI, and couple the
> GUI to a dummy system instead of the
> real system, when you run those tests.
> Test business rules through an API
> that doesn't involve the GUI. -- Bob Martin
“You must understand that seeing is believing, but also know that believing is seeing.” -- Denis Waitley
Solution 19 - Unit Testing
The Sleeper, aka Mount Vesuvius -- Nick Pellow
A test that is destined to FAIL at some specific time and date in the future. This often is caused by incorrect bounds checking when testing code which uses a Date or Calendar object. Sometimes, the test may fail if run at a very specific time of day, such as midnight.
'The Sleeper' is not to be confused with the 'Wait And See' anti-pattern.
That code will have been replaced long before the year 2000 -- Many developers in 1960
Solution 20 - Unit Testing
The Cuckoo -- Frank Carver
A unit test which sits in a test case with several others, and enjoys the same (potentially lengthy) setup process as the other tests in the test case, but then discards some or all of the artifacts from the setup and creates its own.
Advanced Symptom of : Inappropriately Shared Fixture
Solution 21 - Unit Testing
The Dead Tree
A test which where a stub was created, but the test wasn't actually written.
I have actually seen this in our production code:
class TD_SomeClass {
public void testAdd() {
assertEquals(1+1, 2);
}
}
I don't even know what to think about that.
Solution 22 - Unit Testing
got bit by this today:
Wet Floor:
The test creates data that is persisted somewhere, but the test does not clean up when finished. This causes tests (the same test, or possibly other tests) to fail on subsequent test runs.
In our case, the test left a file lying around in the "temp" dir, with permissions from the user that ran the test the first time. When a different user tried to test on the same machine: boom. In the comments on James Carr's site, Joakim Ohlrogge referred to this as the "Sloppy Worker", and it was part of the inspiration for "Generous Leftovers". I like my name for it better (less insulting, more familiar).
Solution 23 - Unit Testing
The Secret Catcher -- Frank Carver
A test that at first glance appears to be doing no testing, due to absence of assertions. But "The devil is in the details".. the test is really relying on an exception to be thrown and expecting the testing framework to capture the exception and report it to the user as a failure.
[Test]
public void ShouldNotThrow()
{
DoSomethingThatShouldNotThrowAnException();
}
Solution 24 - Unit Testing
The Forty Foot Pole Test
Afraid of getting too close to the class they are trying to test, these tests act at a distance, separated by countless layers of abstraction and thousands of lines of code from the logic they are checking. As such they are extremely brittle, and susceptible to all sorts of side-effects that happen on the epic journey to and from the class of interest.
Solution 25 - Unit Testing
The Turing Test
A testcase automagically generated by some expensive tool that has many, many asserts gleaned from the class under test using some too-clever-by-half data flow analysis. Lulls developers into a false sense of confidence that their code is well tested, absolving them from the responsibility of designing and maintaining high quality tests. If the machine can write the tests for you, why can't it pull its finger out and write the app itself!
Hello stupid. -- World's smartest computer to new apprentice (from an old Amiga comic).
Solution 26 - Unit Testing
The Environmental Vandal
A 'unit' test which for various 'requirements' starts spilling out into its environment, using and setting environment variables / ports. Running two of these tests simultaneously will cause 'unavailable port' exceptions etc.
These tests will be intermittent, and leave developers saying things like 'just run it again'.
One solution Ive seen is to randomly select a port number to use. This reduces the possibility of a conflict, but clearly doesnt solve the problem. So if you can, always mock the code so that it doesn't actually allocate the unsharable resource.
Solution 27 - Unit Testing
Doppelgänger
In order to test something, you have to copy parts of the code under test into a new class with the same name and package and you have to use classpath magic or a custom classloader to make sure it is visible first (so your copy is picked up).
This pattern indicates an unhealthy amount of hidden dependencies which you can't control from a test.
I looked at his face ... my face! It was like a mirror but made my blood freeze.
Solution 28 - Unit Testing
The Mother Hen -- Frank Carver
A common setup which does far more than the actual test cases need. For example creating all sorts of complex data structures populated with apparently important and unique values when the tests only assert for presence or absence of something.
Advanced Symptom of: Inappropriately Shared Fixture
I don't know what it does ... I'm adding it anyway, just in case. -- Anonymous Developer
Solution 29 - Unit Testing
The Test It All
I can't believe this hasn't been mentioned till now, but tests should not break the Single Responsibility Principle.
I have come across this so many times, tests that break this rule are by definition a nightmare to maintain.
Solution 30 - Unit Testing
Line hitter
On the first look tests covers everything and code coverage tools confirms it with 100%, but in reality tests only hit code without any output analyses.
Solution 31 - Unit Testing
The Conjoined Twins
Tests that people are calling "Unit Tests" but are really integration tests since they are not isolated from dependencies (file configuration, databases, services, other in other words the parts not being tested in your tests that people got lazy and did not isolate) and fail due to dependencies that should have been stubbed or mocked.