ReSharper Unit Test Runner 9.x update

Published by Marco on

Way back in February, I wrote about my experiences with ReSharper 9 when it first came out. The following article provides an update, this time with version 9.2, released just last week.

tl;dr: I’m back to ReSharper 8.2.3 and am a bit worried about the state of the 9.x series of ReSharper. Ordinarily, JetBrains has eliminated performance, stability and functional issues by the first minor version-update (9.1), to say nothing of the second (9.2).

Test Runner

In the previous article, my main gripe was with the unit-test runner, which was unusable due to flakiness in the UI, execution and change-detection. With the release of 9.2, the UI and change-detection problems have been fixed, but the runner is still quite flaky at executing tests.

What follows is the text of the report that I sent to JetBrains when they asked me why I uninstalled R# 9.2.

As with 9.0 and 9.1, I am unable to productively use the 9.2 Test Runner with many of my NUnit tests. These tests are not straight-up, standard tests, but R# 8.2.3 handled them without any issues whatsoever.

What’s special about my tests?

There are quite a few base classes providing base functionality. The top layers provide scenario-specific input via a generic type parameter.

- TestsBaseOtherBase<TMixin>
     (7 of these, one with an NUnit CategoryAttribute)
    − ConcreteTests<TMixin>
       (defines tests with NUnit TestAttributes)
      − ProviderAConcreteTests<TMixin>
         (CategoryAttribute)
        − ProtocolAProviderAConcreteTests
          (TMixin = ProtocolAProviderA; TestFixtureAttribute, CategoryAttributes)
        − ProtocolBProviderAConcreteTests
          (TMixin = ProtocolBProviderA; TestFixtureAttribute, CategoryAttributes)
      − ProviderBConcreteTests<TMixin>
         (CategoryAttribute)
        − ProtocolAProviderBConcreteTests
          (TMixin = ProtocolAProviderB; TestFixtureAttribute, CategoryAttributes)
        − ProtocolBProviderBConcreteTests
          (TMixin = ProtocolBProviderB; TestFixtureAttribute, CategoryAttributes)

The test runner in 9.2 is not happy with this at all. The test explorer shows all of the tests correctly, with the test counts correct. If I select a node for all tests for ProviderB and ProtocolA (696 tests in 36 fixtures), R# loads 36 non-expandable nodes into the runner and, after a bit of a wait, marks them all as inconclusive. Running an individual test-fixture node does not magically cause the tests to load or appear and also shows inconclusive (after a while; it seems the fixture setup executes as expected but the results are not displayed).

If I select a specific, concrete fixture and add or run those tests, R# loads and executes the runner correctly. If I select multiple test fixtures in the explorer and add them, they also show up as expandable nodes, with the correct test counts, and can be executed individually (per fixture). However, if I elect to run them all by running the parent node, R# once again marks everything as inconclusive.

As I mentioned, 8.2.3 handles this correctly and I feel R# 9.2 isn’t far off—the unit-test explorer does, after all, show the correct tests and counts. In 9.2, it’s not only inconvenient, but I’m worried that my tests are not being executed with the expected configuration.

Also, I really missed the StyleCop plugin for 9.2. There’s a beta version for 9.1 that caused noticeable lag, so I’m still waiting for a more unobtrusive version for 9.2 (or any version at all).

While it’s possible that there’s something I’m doing wrong, or there’s something in my installation that’s strange, I don’t think that’s the problem. As I mentioned, test-running for the exact same solution with 8.2.3 is error-free and a pleasure to use. In 9.2, the test explorer shows all of the tests correctly, so R# is clearly able to interpret the hierarchy and attributes (noted above) as I’ve intended them to be interpreted. This feels very much like a bug or a regression for which JetBrains doesn’t have test coverage. I will try to work with them to help them get coverage for this case.

Real-Time StyleCop rules

Additionally, the StyleCop plugin is absolutely essential for my workflow and there still isn’t an official release for any of the 9.x versions. ReSharper 9.2 isn’t supported at all yet, even in prerelease form. The official Codeplex page shows the latest official version as 4.7, released in January of 2012 for ReSharper 8.2 and Visual Studio 2013. One would imagine that VS2015 support is in the works, but it’s hard to say. There is a page for StyleCop in the ReSharper extensions gallery but that shows a beta4, released in April of 2015, that only works with ReSharper 9.1.x, not 9.2. I tested it with 9.1.x, but it noticeably slowed down the UI. While typing was mostly unaffected, scrolling and switching file-tabs was very laggy. Since StyleCop is essential for so many developers, it’s hard to see why the plugin gets so little love from either JetBrains or Microsoft.

GoTo Word

The “Go To Word” plugin is not essential but it is an extremely welcome addition, especially with so much more client-side work depending on text-based bindings that aren’t always detected by ReSharper. In those cases, you can find—for example—all the references of a Knockout template by searching just as you would for a type or member. Additionally, you benefit from the speed of the ReSharper indexing engine and search UI instead of using the comparatively slow and ugly “Find in Files” support in Visual Studio. Alternatives suggested in the comments to the linked issue above all depend on building yet another index of data (e.g. Sando Code Search Tool). JetBrains has pushed off integrating go-to-word until version 10. Again, not a deal-breaker, but a shame nonetheless, as I’ll have to do without it in 9.x until version 10 is released.

With so much more client-side development going on in Visual Studio and with dynamic languages and data-binding languages that use name-matching for data-binding, GoToWord is more and more essential. Sure, ReSharper can continue to integrate native support for finding such references, but until that happens, we’re stuck with the inferior Find-in-Files dialog or other extensions that increase the memory pressure for larger solutions.