Improving NUnit integration with testing harnesses

Published by Marco on

 These days nobody who’s anybody in the software-development world is writing software without tests. Just writing them doesn’t help make the software better, though. You also need to be able to execute tests—reliably and quickly and repeatably.

That said, you’ll have to get yourself a test runner, which is a different tool from the compiler or the runtime. That is, just because your tests compile (satisfy all of the language rules) and could be executed doesn’t mean that you’re done writing them yet.

Testing framework requirements

Every testing framework has its own rules for how the test runner selects methods for execution as tests. The standard configuration options are:

  • Which classes should be considered as test fixtures?
  • Which methods are considered tests?
  • Where do parameters for these methods come from?
  • Is there startup/teardown code to execute for the test or fixture?

Each testing framework will offer different ways of configuring your code so that the test runner can find and execute setup/test/teardown code. To write NUnit tests, you decorate classes, methods and parameters with C# attributes.

The standard scenario is relatively easy to execute—run all methods with a Test attribute in a class with a TestFixture attribute on it.

Test-runner Requirements

There are legitimate questions for which even the best specification does not provide answers.

When you consider multiple base classes and generic type arguments, each of which may also have NUnit attributes, things get a bit less clear. In that case, not only do you have to know what NUnit offers as possibilities but also whether the test runner that you’re using also understands and implements the NUnit specification in the same way. Not only that, but there are legitimate questions for which even the best specification does not provide answers.

At Encodo, we use Visual Studio 2015 with ReSharper 9.2 and we use the ReSharper test runner. We’re still looking into using the built-in VS test runner—the continuous-testing integration in the editor is intriguing[1]—but it’s quite weak when compared to the ReSharper one.

So, not only do we have to consider what the NUnit documentation says is possible, but we must also know what how the R# test runner interprets the NUnit attributes and what is supported.

Getting More Complicated

Where is there room for misunderstanding? A few examples,

  • What if there’s a TestFixture attribute on an abstract class?
  • How about a TestFixture attribute on a class with generic parameters?
  • Ok, how about a non-abstract class with Tests but no TestFixture attribute?
  • And, finally, a non-abstract class with Tests but no TestFixture attribute, but there are non-abstract descendants that do have a TestFixture attribute?

In our case, the answer to these questions depends on which version of R# you’re using. Even though it feels like you configured everything correctly and it logically should work, the test runner sometimes disagrees.

  • Sometimes it shows your tests as expected, but refuses to run them (Inconclusive FTW!)
  • Or other times, it obstinately includes generic base classes that cannot be instantiated into the session, then complains that you didn’t execute them. When you try to delete them, it brings them right back on the next build. When you try to run them—perhaps not noticing that it’s those damned base classes—then it complains that it can’t instantiate them. Look of disapproval.

Throw the TeamCity test runner into the mix—which is ostensibly the same as that from R# but still subtly different—and you’ll have even more fun.

Improving Integration with the R# Test Runner

At any rate, now that you know the general issue, I’d like to share how the ground rules we’ve come up with that avoid all of the issues described above. The text below comes from the issue I created for the impending release of Quino 2.

Environment

  • Windows 8.1 Enterprise
  • Visual Studio 2015
  • ReSharper 9.2

Expected behavior

Non-leaf-node base classes should never appear as nodes in test runners. A user should be able to run tests in descendants directly from a fixture or test in the base class.

Observed behavior

Non-leaf-node base classes are shown in the R# test runner in both versions 9 and 10. A user must navigate to the descendant to run a test. The user can no longer run all descendants or a single descendant directly from the test.

Analysis

Relatively recently, in order to better test a misbehaving test runner and accurately report issues to JetBrains, I standardized all tests to the same pattern:

  • Do not use abstract anywhere (the base classes don’t technically need it)
  • Use the TestFixture attribute only on leaf nodes

This worked just fine with ReSharper 8.x but causes strange behavior in both R# 9.x and 10.x. We discovered recently that not only did the test runner act strangely (something that they might fix), but also that the unit-testing integration in the files themselves behaved differently when the base class is abstract (something JetBrains is unlikely to fix).

You can see that R# treats a non-abstract class with tests as a testable entity, even when it doesn’t actually have a TestFixture attribute and even expects a generic type parameter in order to instantiate.

Here it’s not working well in either the source file or the test runner. In the source file, you can see that it offers to run tests in a category, but not the tests from actual descendants. If you try to run or debug anything from this menu, it shows the fixture with a question-mark icon and marks any tests it manages to display as inconclusive. This is not surprising, since the test fixture may not be abstract, but does require a type parameter in order to be instantiated.

 Non-abstract base class

Here it looks and acts correctly:

 A test in an abstract test fixture

I’ve reported this issue to JetBrains, but our testing structure either isn’t very common or it hasn’t made it to their core test cases, because neither 9 nor 10 handles them as well as the 8.x runner did.

Now that we’re also using TeamCity a lot more to not only execute tests but also to collect coverage results, we’ll capitulate and just change our patterns to whatever makes R#/TeamCity the happiest.

Solution

  • Make all testing base classes that include at least one {{Test}} or {{Category}} attribute {{abstract}}. Base classes that do not have any testing attributes do not need to be made abstract.

Once more to recap our ground rules for making tests:

  • Include TestFixture only on leafs (classes with no descendants)
  • You can put Category or Test attributes anywhere in the hierarchy, but need to declare the class as abstract.
  • Base classes that have no testing attributes do not need to be abstract
  • If you feel you need to execute tests in both a base class and one of its descendants, then you’re probably doing something wrong. Make two descendants of the base class instead.

When you make the change, you can see the improvement immediately.

 After making the base class abstract


[1] ReSharper 10.0 also offers continuous integration, but our experiments with the EAP builds and the first RTM build left us underwhelmed and we downgraded to 9.2 until JetBrains manages to release a stable 10.x.