Best practice for debug Asserts during Unit testing

Does heavy use of unit tests discourage the use of debug asserts? It seems like a debug assert firing in the code under test implies the unit test shouldn't exist or the debug assert shouldn't exist. "There can be only one" seems like a reasonable principle. Is this the common practice? Or do you disable your debug asserts when unit testing, so they can be around for integration testing?

Edit: I updated 'Assert' to debug assert to distinguish an assert in the code under test from the lines in the unit test that check state after the test has run.

Also here is an example that I believe shows the dilema: A unit test passes invalid inputs for a protected function that asserts it's inputs are valid. Should the unit test not exist? It's not a public function. Perhaps checking the inputs would kill perf? Or should the assert not exist? The function is protected not private so it should be checking it's inputs for safety.

This is a perfectly valid question.

First of all, many people are suggesting that you are using assertions wrongly. I think many debugging experts would disagree. Although it is good practice to check invariants with assertions, assertions shouldn't be limited to state invariants. In fact, many expert debuggers will tell you to assert any conditions that may cause an exception in addition to checking invariants.

For example, consider the following code:

if (param1 == null)
    throw new ArgumentNullException("param1");

That's fine. But when the exception is thrown, the stack gets unwound until something handles the exception (probably some top level default handler). If execution pauses at that point (you may have a modal exception dialog in a Windows app), you have the chance to attach a debugger, but you have probably lost a lot of the information that could have helped you to fix the issue, because most of the stack has been unwound.

Now consider the following:

if (param1 == null)
{
    Debug.Fail("param1 == null");
    throw new ArgumentNullException("param1");
}

Now if the problem occurs, the modal assert dialog pops up. Execution is paused instantaneously. You are free to attach your chosen debugger and investigate exactly what's on the stack and all the state of the system at the exact point of failure. In a release build, you still get an exception.

Now how do we handle your unit tests?

Consider a unit test that tests the code above that includes the assertion. You want to check that the exception is thrown when param1 is null. You expect that particular assertion to fail, but any other assertion failures would indicate that something is wrong. You want to allow particular assertion failures for particular tests.

The way you solve this will depend upon what languages etc. you're using. However, I have some suggestions if you are using .NET (I haven't actually tried this, but I will in the future and update the post):

1. Check Trace.Listeners. Find any instance of DefaultTraceListener and set AssertUiEnabled to false. This stops the modal dialog from popping up. You could also clear the listeners collection, but you'll get no tracing whatsoever.
2. Write your own TraceListener which records assertions. How you record assertions is up to you. Recording the failure message may not be good enough, so you may want to walk the stack to find the method the assertion came from and record that too.
3. Once a test ends, check that the only assertion failures that occured were the ones you were expecting. If any others occurred, fail the test.

For an example of a TraceListener that contains the code to do a stack walk like that, I'd search for SUPERASSERT.NET's SuperAssertListener and check its code. (It's also worth integrating SUPERASSERT.NET if you're really serious about debugging using assertions).

Most unit test frameworks support test setup/teardown methods. You may want to add code to reset the trace listener and to assert that there aren't any unexpected assertion failures in those areas to mimimize duplication and prevent mistakes.

UPDATE:

Here is an example TraceListener that can be used to unit test assertions. You should add an instance to the Trace.Listeners collection. You'll probably also want to provide some easy way that your tests can get hold of the listener.

NOTE: This owes an awful lot to John Robbins' SUPERASSERT.NET.

/// <summary>
/// TraceListener used for trapping assertion failures during unit tests.
/// </summary>
public class DebugAssertUnitTestTraceListener : DefaultTraceListener
{
	/// <summary>
	/// Defines an assertion by the method it failed in and the messages it
	/// provided.
	/// </summary>
	public class Assertion
	{
		/// <summary>
		/// Gets the message provided by the assertion.
		/// </summary>
		public String Message { get; private set; }

		/// <summary>
		/// Gets the detailed message provided by the assertion.
		/// </summary>
		public String DetailedMessage { get; private set; }

		/// <summary>
		/// Gets the name of the method the assertion failed in.
		/// </summary>
		public String MethodName { get; private set; }

		/// <summary>
		/// Creates a new Assertion definition.
		/// </summary>
		/// <param name="message"></param>
		/// <param name="detailedMessage"></param>
		/// <param name="methodName"></param>
		public Assertion(String message, String detailedMessage, String methodName)
		{
			if (methodName == null)
			{
				throw new ArgumentNullException("methodName");
			}

			Message = message;
			DetailedMessage = detailedMessage;
			MethodName = methodName;
		}

		/// <summary>
		/// Gets a string representation of this instance.
		/// </summary>
		/// <returns></returns>
		public override string ToString()
		{
			return String.Format("Message: {0}{1}Detail: {2}{1}Method: {3}{1}",
				Message ?? "<No Message>",
				Environment.NewLine,
				DetailedMessage ?? "<No Detail>",
				MethodName);
		}

		/// <summary>
		/// Tests this object and another object for equality.
		/// </summary>
		/// <param name="obj"></param>
		/// <returns></returns>
		public override bool Equals(object obj)
		{
			var other = obj as Assertion;
			
			if (other == null)
			{
				return false;
			}

			return
				this.Message == other.Message &&
				this.DetailedMessage == other.DetailedMessage &&
				this.MethodName == other.MethodName;
		}

		/// <summary>
		/// Gets a hash code for this instance.
		/// Calculated as recommended at http://msdn.microsoft.com/en-us/library/system.object.gethashcode.aspx
		/// </summary>
		/// <returns></returns>
		public override int GetHashCode()
		{
			return
				MethodName.GetHashCode() ^
				(DetailedMessage == null ? 0 : DetailedMessage.GetHashCode()) ^
				(Message == null ? 0 : Message.GetHashCode());
		}
	}

	/// <summary>
	/// Records the assertions that failed.
	/// </summary>
	private readonly List<Assertion> assertionFailures;

	/// <summary>
	/// Gets the assertions that failed since the last call to Clear().
	/// </summary>
	public ReadOnlyCollection<Assertion> AssertionFailures { get { return new ReadOnlyCollection<Assertion>(assertionFailures); } }

	/// <summary>
	/// Gets the assertions that are allowed to fail.
	/// </summary>
	public List<Assertion> AllowedFailures { get; private set; }

	/// <summary>
	/// Creates a new instance of this trace listener with the default name
	/// DebugAssertUnitTestTraceListener.
	/// </summary>
	public DebugAssertUnitTestTraceListener() : this("DebugAssertUnitTestListener") { }

	/// <summary>
	/// Creates a new instance of this trace listener with the specified name.
	/// </summary>
	/// <param name="name"></param>
	public DebugAssertUnitTestTraceListener(String name) : base()
	{
		AssertUiEnabled = false;
		Name = name;
		AllowedFailures = new List<Assertion>();
		assertionFailures = new List<Assertion>();
	}

	/// <summary>
	/// Records assertion failures.
	/// </summary>
	/// <param name="message"></param>
	/// <param name="detailMessage"></param>
	public override void Fail(string message, string detailMessage)
	{
		var failure = new Assertion(message, detailMessage, GetAssertionMethodName());

		if (!AllowedFailures.Contains(failure))
		{
			assertionFailures.Add(failure);
		}
	}

	/// <summary>
	/// Records assertion failures.
	/// </summary>
	/// <param name="message"></param>
	public override void Fail(string message)
	{
		Fail(message, null);
	}

	/// <summary>
	/// Gets rid of any assertions that have been recorded.
	/// </summary>
	public void ClearAssertions()
	{
		assertionFailures.Clear();
	}

	/// <summary>
	/// Gets the full name of the method that causes the assertion failure.
	/// 
	/// Credit goes to John Robbins of Wintellect for the code in this method,
	/// which was taken from his excellent SuperAssertTraceListener.
	/// </summary>
	/// <returns></returns>
	private String GetAssertionMethodName()
	{
		
		StackTrace stk = new StackTrace();
		int i = 0;
		for (; i < stk.FrameCount; i++)
		{
			StackFrame frame = stk.GetFrame(i);
			MethodBase method = frame.GetMethod();
			if (null != method)
			{
				if(method.ReflectedType.ToString().Equals("System.Diagnostics.Debug"))
				{
					if (method.Name.Equals("Assert") || method.Name.Equals("Fail"))
					{
						i++;
						break;
					}
				}
			}
		}

		// Now walk the stack but only get the real parts.
		stk = new StackTrace(i, true);

		// Get the fully qualified name of the method that made the assertion.
		StackFrame hitFrame = stk.GetFrame(0);
		StringBuilder sbKey = new StringBuilder();
		sbKey.AppendFormat("{0}.{1}",
							 hitFrame.GetMethod().ReflectedType.FullName,
							 hitFrame.GetMethod().Name);
		return sbKey.ToString();
	}
}

You can add Assertions to the AllowedFailures collection at the start of each test for the assertions you expect.

At the end of every test (hopefully your unit testing framework supports a test teardown method) do:

if (DebugAssertListener.AssertionFailures.Count > 0)
{
    // TODO: Create a message for the failure.
    DebugAssertListener.ClearAssertions();
    DebugAssertListener.AllowedFailures.Clear();
    // TODO: Fail the test using the message created above.
}

IMHO debug.asserts rock. This great article shows how to stop them from interrupting your unit test by adding an app.config to your unit testing project, and disabling the dialog box:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.diagnostics>
    <assert assertuienabled="false"/>
</system.diagnostics>

As others have mentioned, Debug asserts are meant for things that should always be true. (The fancy term for this is invariants).

If your unit test is passing in bogus data that is tripping the assert, then you have to ask yourself the question - why is that happening?

• If the function under test is supposed to deal with the bogus data, then clearly that assert shouldn't be there.
• If the function is not equipped to deal with that kind of data (as indicated by the assert), then why are you unit testing for it?

The second point is one that quite a few developers seem to fall into. Unit test the heck out of all the things your code is built to deal with, and Assert or throw exceptions for everything else - After all, if your code is NOT build to deal with those situations, and you cause them to happen, what do you expect to happen?
You know those parts of the C/C++ documentation that talk about "undefined behaviour"? This is it. Bail and bail hard.

---

Update to clarify: The flipside of this is that you end up realising you should only use Debug.Assert for internal things calling other internal things. If your code is exposed to 3rd parties (i.e. it's a library or something) then there is no limit to what input you can expect, and thus you should validate properly and throw exceptions or whatever, and you should unit test for that too

Assertions in your code are (ought to be) statements to the reader that say "this condition should always be true at this point." Done with some discipline, they can be part of ensuring that the code is correct; most people use them as debug print statements. Unit Tests are code that demonstrates that your code correctly performs a particular test case; don'e well, they can both document the reuirements, and raise your confidence that the code is indeed correct.

Get the difference? The program assertions help you make it correct, the unit tests help you develop someone else's confidence that the code is correct.

A good unit test setup will have the ability to catch asserts. If an assert is triggered the current test should fail and the next is run.

In our libraries low-level debug functionality such as TTY/ASSERTS have handlers that are called. The default handler will printf/break, but client code can install custom handlers for different behavior.

Our UnitTest framework installs its own handlers that log messages and throw exceptions on asserts. The UnitTest code will then catch these exceptions if they occur and log them as an fail, along with the asserted statement.

You can also include assert testing in your unit test - e.g.

CHECK_ASSERT(someList.getAt(someList.size() + 1); // test passes if an assert occurs

I took the approach of disabling the assert only where needed as opposed to doing it project wide. Here is an approach where the assert can be suspend so it doesn't interfere with the test flow.

public static class TraceListenerCollectionEx
{
    /// <summary>
    /// This is a helper class that allows us to suspend asserts / all trace listeners
    /// </summary>
    public class SuspendTrackerDisposable : IDisposable
    {
        private readonly TraceListenerCollection _traceListenerCollection;
        private readonly TraceListener[] _suspendedListeners;

        public SuspendTrackerDisposable(TraceListenerCollection traceListenerCollection)
        {
            _traceListenerCollection = traceListenerCollection;

            var numListeners = traceListenerCollection.Count;
            _suspendedListeners = new TraceListener[numListeners];
            for( int index = 0; index < numListeners; index += 1 )
                _suspendedListeners[index] = traceListenerCollection[index];

            traceListenerCollection.Clear();
        }

        public void Dispose()
        {
            _traceListenerCollection.AddRange(_suspendedListeners);
        }
    }

    public static SuspendTrackerDisposable AssertSuspend(this TraceListenerCollection traceListenerCollection) => new SuspendTrackerDisposable(traceListenerCollection);
}

Here is an example usage within a test:

    [TestMethod]
    public void EnumDefaultTest()
    {
        using(Trace.Listeners.AssertSuspend()) {
            Enum<CarClass>.DefaultValue.ShouldBe(CarClass.Unknown);  
        }
    }

The code executed within the using block, only one line in this case, will have their asserts disabled.

Do you mean C++/Java asserts for "programming by contract" assertions, or CppUnit/JUnit asserts? That last question leads me to believe that it's the former.

Interesting question, because it's my understanding that those asserts are often turned off at runtime when you deploy to production. (Kinda defeats the purpose, but that's another question.)

I'd say that they should be left in your code when you test it. You write tests to ensure that the pre-conditions are being enforced properly. The test should be a "black box"; you should be acting as a client to the class when you test. If you happen to turn them off in production, it doesn't invalidate the tests.

First to have both Design by Contract assertions and unit tests, your unit testing framework shall be able to catch the assertions. If your unit tests abort because of a DbC abort, then you simply cannot run them. The alternative here is to disable those assertions while running (read compiling) your unit tests.

Since you're testing non-public functions, what is the risk of having a function invoked with invalid argument ? Don't your unit tests cover that risk ? If you write your code following the TDD (Test-Driven Development) technique, they should.

If you really want/need those Dbc-type asserts in your code, then you can remove the unit tests that pass the invalid arguments to the methods having those asserts.

However, Dbc-type asserts can be useful in lower level functions (that is not directly invoked by the unit tests) when you have coarse-grained unit tests.

You should keep your debug asserts, even with unit tests in place.

The problem here is not differentiating between Errors and Problems.

If a function checks its arguments which are erroneous, it should not result in a debug assertion. Instead it should return an error value back. It was an Error to call the function with wrong parameters.

If a function is passed correct data, but cannot operate properly due to the runtime has run out of memory, then the code should issue a debug assert due to this Problem. That an example of fundamental assumptions which if they don't hold, "all bets are off", so you must terminate.

In your case, do write the unit test that supplies erroneous values as arguments. It should expect an error return value (or similar). Getting an assert? -- refactor the code to produce an error instead.

Note a bug-free problem can still trigger asserts; e.g. the hardware could break. In your question, you mentioned integration testing; indeed, asserting against incorrectly composed integrated systems is assert territory; e.g. incompatible library version loaded.

Note, the reason for "debug"-asserts is a trade-off between being diligent/safe and being fast/small.

Like the others have mentioned, the Debug.Assert statements should always be true, even if arguements are incorrect, the assertion should be true to stop the app getting into an invalid state etc.

Debug.Assert(_counter == somethingElse, "Erk! Out of wack!");

You should not be able to test this (and probably don't want to because there is nothing you can do really!)

I could be way off but I get the impression that perhaps the asserts you may be talking about are better suited as "argument exceptions", e.g.

if (param1 == null)
  throw new ArgumentNullException("param1", "message to user")

That kind of "assertion" in your code is still very testable.

PK :-)

It has been a while since this question has been asked, but I think I have a different way of verifying Debug.Assert() calls from within a unit test using C# code. Note the #if DEBUG ... #endif block, which is needed for skipping the test when not running in debug configuration (in which case, Debug.Assert() won't be fired anyway).

[TestClass]
[ExcludeFromCodeCoverage]
public class Test
{
    #region Variables              |

    private UnitTestTraceListener _traceListener;
    private TraceListenerCollection _originalTraceListeners;

    #endregion

    #region TestInitialize         |

    [TestInitialize]
    public void TestInitialize() {
        // Save and clear original trace listeners, add custom unit test trace listener.
        _traceListener = new UnitTestTraceListener();
        _originalTraceListeners = Trace.Listeners;
        Trace.Listeners.Clear();
        Trace.Listeners.Add(_traceListener);

        // ... Further test setup
    }

    #endregion
    #region TestCleanup            |

    [TestCleanup]
    public void TestCleanup() {
        Trace.Listeners.Clear();
        Trace.Listeners.AddRange(_originalTraceListeners);
    }

    #endregion

    [TestMethod]
    public void TheTestItself() {
        // Arrange
        // ...

        // Act
        // ...
        Debug.Assert(false, "Assert failed");



    // Assert

#if DEBUG        
    // NOTE This syntax comes with using the FluentAssertions NuGet package.
    _traceListener.GetWriteLines().Should().HaveCount(1).And.Contain("Fail: Assert failed");
#endif

    }
}

The UnitTestTraceListener class looks like follows:

[ExcludeFromCodeCoverage]
public class UnitTestTraceListener : TraceListener
{
    private readonly List<string> _writes = new List<string>();
    private readonly List<string> _writeLines = new List<string>();

    // Override methods
    public override void Write(string message)
    {
        _writes.Add(message);
    }

    public override void WriteLine(string message)
    {
        _writeLines.Add(message);
    }

    // Public methods
    public IEnumerable<string> GetWrites()
    {
        return _writes.AsReadOnly();
    }

    public IEnumerable<string> GetWriteLines()
    {
        return _writeLines.AsReadOnly();
    }

    public void Clear()
    {
        _writes.Clear();
        _writeLines.Clear();
    }
}

> Does heavy use of unit tests discourage the use of debug asserts?

No. The opposite. Unit testing makes Debug asserts much more valuable, by double-checking the internal state while running the white-box tests you've written. Enabling Debug.Assert during unit test is essential, because you rarely ship DEBUG-enabled code (unless performance is not important at all). The only two times DEBUG code is run is when you are either 1) doing that tiny bit of integration testing you really do, all good intentions aside, and 2) running unit tests.

It is easy to instrument code with Debug.Assert tests to check invariants as you write it. These checks serve as sanity checks when unit tests run.

The other things Assert does is point to exactly the first point in the code where things went wrong. This can greatly reduce debugging time when your unit test does find the problem.

This increases the value of unit tests.

> It seems like a debug assert firing in the code under test implies the unit test shouldn't exist or the debug assert shouldn't exist.

Case in point. This question is about a real thing that happens. Right? Therefore you need debug asserts in your code, and you need them to trigger during unit tests. The possibility that a debug assert might fire during a unit test clearly demonstrates that debug asserts should be enabled during unit tests.

An assert firing means that either your tests are using your internal code incorrectly (and should be fixed), or some of the code under test is calling other internal code incorrectly, or that somewhere a fundamental assumption is wrong. You do not write tests because you think your assumptions are wrong, you... actually, you do. You write tests because at least some of your assumptions are probably wrong. Redundancy is OK in this situation.

> "There can be only one" seems like a reasonable principle. Is this the common practice? Or do you disable your debug asserts when unit testing, so they can be around for integration testing?

Redundancy hurts nothing but the runtime of your unit tests. If you really have 100% coverage, runtime might be an issue. Otherwise, no I strongly disagree. There is nothing wrong with checking your assumption automatically in the middle of a test. That's practically the definition of "testing".

> Also here is an example that I believe shows the dilema: A unit test passes invalid inputs for a protected function that asserts it's inputs are valid. Should the unit test not exist? It's not a public function. Perhaps checking the inputs would kill perf? Or should the assert not exist? The function is protected not private so it should be checking it's inputs for safety.

Usually, it is not the purpose of a unit test framework is to test the behavior of your code when the invariant assumptions have been violated. In other words, if the documentation you wrote says "if you pass null as a parameter, results are undefined", you do not need to verify that results are indeed unpredictable. If the failure results are clearly defined, they are not undefined, and 1) it should not be a Debug.Assert, 2) you should define exactly what the results are, and 3) test for that result. If you need to unit test the quality of your internal debug assertions, then 1) Andrew Grant's approach of making Assertion frameworks a testable asset should probably be checked as the answer, and 2) wow you have awesome test coverage! And I think this is largely a personal decision based on the project requirements. But I still think debug asserts are essential and valuable.

In other words: Debug.Assert() greatly increases the value of unit tests, and the redundancy is a feature.