Published by Marco on 4. Dec 2015 13:26:21
------------------------------------------------------------------------

[image]These days nobody who's anybody in the software-development world is
writing software without tests. Just writing them doesn't help make the software
better, though. You also need to be able to execute tests -- reliably and
quickly and repeatably.

That said, you'll have to get yourself a test runner, which is a different tool
from the compiler or the runtime. That is, just because your tests compile
(satisfy all of the language rules) and could be executed doesn't mean that
you're done writing them yet.

[Testing framework requirements]

Every testing framework has its own rules for how the test runner selects
methods for execution as tests. The standard configuration options are:

  * Which classes should be considered as test fixtures?
  * Which methods are considered tests?
  * Where do parameters for these methods come from?
  * Is there startup/teardown code to execute for the test or fixture?

Each testing framework will offer different ways of configuring your code so
that the test runner can find and execute setup/test/teardown code. To write
NUnit tests, you decorate classes, methods and parameters with C# attributes.

The standard scenario is relatively easy to execute -- run all methods with a
Test attribute in a class with a TestFixture attribute on it.

[Test-runner Requirements]

"There are legitimate questions for which even the best specification does not
provide answers."

When you consider multiple base classes and generic type arguments, each of
which may also have NUnit attributes, things get a bit less clear. In that case,
not only do you have to know what NUnit offers as possibilities but also whether
the test runner that you're using also understands and implements the NUnit
specification in the same way. Not only that, but there are legitimate questions
for which even the best specification does not provide answers.

At Encodo, we use Visual Studio 2015 with ReSharper 9.2 and we use the ReSharper
test runner. We're still looking into using the built-in VS test runner -- the
continuous-testing integration in the editor is intriguing [1] -- but it's quite
weak when compared to the ReSharper one.

So, not only do we have to consider what the NUnit documentation says is
possible, but we must also know what how the R# test runner interprets the NUnit
attributes and what is supported.

[Getting More Complicated]

Where is there room for misunderstanding? A few examples,

  * What if there's a TestFixture attribute on an abstract class?
  * How about a TestFixture attribute on a class with generic parameters?
  * Ok, how about a non-abstract class with Tests but no TestFixture attribute?
  * And, finally, a non-abstract class with Tests but no TestFixture attribute,
    but there are non-abstract descendants that do have a TestFixture attribute?

In our case, the answer to these questions depends on which version of R# you're
using. Even though it feels like you configured everything correctly and it
logically should work, the test runner sometimes disagrees. 

  * Sometimes it shows your tests as expected, but refuses to run them
    (Inconclusive FTW!)
  * Or other times, it obstinately includes generic base classes that cannot be
    instantiated into the session, then complains that you didn't execute them.
    When you try to delete them, it brings them right back on the next build.
    When you try to run them -- perhaps not noticing that it's those damned base
    classes -- then it complains that it can't instantiate them. Look of
    disapproval.

Throw the TeamCity test runner into the mix -- which is ostensibly the same as
that from R# but still subtly different -- and you'll have even more fun.

[Improving Integration with the R# Test Runner]

At any rate, now that you know the general issue, I'd like to share how the
ground rules we've come up with that avoid all of the issues described above.
The text below comes from the "issue"
 I created for the impending
release of Quino 2.

[Environment]

  * Windows 8.1 Enterprise
  * Visual Studio 2015
  * ReSharper 9.2

[Expected behavior]

Non-leaf-node base classes should never appear as nodes in test runners. A user
should be able to run tests in descendants directly from a fixture or test in
the base class.

[Observed behavior]

Non-leaf-node base classes are shown in the R# test runner in both versions 9
and 10. A user must navigate to the descendant to run a test. The user can no
longer run all descendants or a single descendant directly from the test.

[Analysis]

Relatively recently, in order to better test a misbehaving test runner and
accurately report issues to JetBrains, I standardized all tests to the same
pattern:

  * Do not use abstract anywhere (the base classes don't technically need it)
  * Use the TestFixture attribute only on leaf nodes

This worked just fine with ReSharper 8.x but causes strange behavior in both R#
9.x and 10.x. We discovered recently that not only did the test runner act
strangely (something that they might fix), but also that the unit-testing
integration in the files themselves behaved differently when the base class is
abstract (something JetBrains is unlikely to fix).

You can see that R# treats a non-abstract class with tests as a testable entity,
even when it doesn't actually have a TestFixture attribute and even expects a
generic type parameter in order to instantiate.

Here it's not working well in either the source file or the test runner. In the
source file, you can see that it offers to run tests in a category, but not the
tests from actual descendants. If you try to run or debug anything from this
menu, it shows the fixture with a question-mark icon and marks any tests it
manages to display as inconclusive. This is not surprising, since the test
fixture may not be abstract, but does require a type parameter in order to be
instantiated.

[image]

Here it looks and acts correctly:

[image]

I've reported this issue to JetBrains, but our testing structure either isn't
very common or it hasn't made it to their core test cases, because neither 9 nor
10 handles them as well as the 8.x runner did.

Now that we're also using TeamCity a lot more to not only execute tests but also
to collect coverage results, we'll capitulate and just change our patterns to
whatever makes R#/TeamCity the happiest.

[Solution]

  * Make all testing base classes that include at least one {{Test}} or
    {{Category}} attribute {{abstract}}. Base classes that do not have any
    testing attributes do not need to be made abstract.

Once more to recap our ground rules for making tests:

  * Include TestFixture only on leafs (classes with no descendants)
  * You can put Category or Test attributes anywhere in the hierarchy, but need
    to declare the class as abstract.
  * Base classes that have no testing attributes do not need to be abstract
  * If you feel you need to execute tests in both a base class and one of its
    descendants, then you're probably doing something wrong. Make two
    descendants of the base class instead.

When you make the change, you can see the improvement immediately.

[image]

--------------------------------------------------------------------------------


[1] ReSharper 10.0 also offers continuous integration, but our experiments with
    the EAP builds and the first RTM build left us underwhelmed and we
    downgraded to 9.2 until JetBrains manages to release a stable 10.x.