Published by Marco on 10. Feb 2015 13:25:53
Updated by Marco on 10. Feb 2015 14:25:49
------------------------------------------------------------------------

We've been using ReSharper at Encodo since version 4. And we regularly use a ton
of other software from JetBrains [1] -- so we're big fans.

[How to Upgrade R#]

As long-time users of ReSharper, we've become accustomed to the following
pattern of adoption for new major versions:

[EAP]

   1. Read about cool new features and improvements on the JetBrains blog
   2. Check out the EAP builds page
   3. Wait for star ratings to get higher than 2 out of 5
   4. Install EAP of next major version
   5. Run into issues/problems that make testing EAP more trouble than it's
      worth
   6. Re-install previous major version

[RTM]

   1. Major version goes RTM
   2. Install immediately; new features! Yay!
   3. Experience teething problems in x.0 version
   4. Go through hope/disappointment cycle for a couple of bug-fix versions
      (e.g. x.0.1, x.0.2)
   5. Install first minor-version release immediately; stability! Yay!

This process can take anywhere from several weeks to a couple of months. The
reason we do it almost every time is that the newest version of ReSharper almost
always has a few killer features. For example, version 8 had initial TypeScript
support. Version 9 carries with it a slew of support improvements for Gulp,
TypeScript and other web technologies.

Unfortunately, if you need to continue to use the test-runner with C#, you're in
for a bumpy ride.

[History of the Test Runner]

Any new major version of ReSharper can be judged by its test runner. The test
runner seems to be rewritten from the ground-up in every major version. Until
the test runner has settled down, we can't really use that version of ReSharper
for C# development.

The 6.x and 7.x versions were terrible at the NUnit TestCase and Values
attributes. They were so bad that we actually converted tests back from using
those attributes. While 6.x had trouble reliably compiling and executing those
tests, 7.x was better at noticing that something had changed without forcing the
user to manually rebuild everything.

Unfortunately, this new awareness in 7.x came at a cost: it slowed editing in
larger NUnit fixtures down to a crawl, using a tremendous amount of memory and
sending VS into a 1.6GB+ memory-churn that made you want to tear your hair out.

8.x fixed all of this and, by 8.2.x was a model of stability and usefulness,
getting the hell out of the way and reliably compiling, displaying and running
tests.

[The 9.x Test Runner]

And then along came 9.x, with a whole slew of sexy new features that just had to
be installed. I tried the new features and they were good. They were fast. I was
looking forward to using the snazzy new editor to create our own formatting
template. ReSharper seemed to be using less memory, felt snappier, it was
lovely. 

And then I launched the test runner.

And then I uninstalled 9.x and reinstalled 8.x.

And then I needed the latest version of DotMemory and was forced to reinstall
9.x. So I tried the test runner again, which inspired this post. [2]

So what's not to love about the test runner? It's faster and seems much more
asynchronous. However, it gets quite confused about which tests to run, how to
handle test cases and how to handle abstract unit-test base classes.

Just like 6.x, ReSharper 9.x can't seem to keep track of which assemblies need
to be built based on changes made to the code and which test(s) the user would
like to run.

[image][image]

To be fair, we have some abstract base classes in our unit fixtures. For
example, we define all ORM query tests in multiple abstract test-fixtures and
then create concrete descendants that run those tests for each of our supported
databases. If I make a change to a common assembly and run the tests for
PostgreSql, then I expect -- at the very least -- that the base assembly and the
PostgreSql test assemblies will be rebuilt. 9.x isn't so good at that yet,
forcing you to "Rebuild All" -- something that I'd no longer had to do with
8.2.x.

[TestCases and the Unit Test Explorer]

It's the same with TestCases: whereas 8.x was able to reliably show changes and
to make sure that the latest version was run, 9.x suffers from the same issue
that 6.x and 7.x had: sometimes the test is shown as a single node without
children and sometimes it's shown with the wrong children. Running these tests
results in a spinning cursor that never ends. You have to manually abort the
test-run, rebuild all, reload the runner with the newly generated tests from the
explorer and try again. This is a gigantic pain in the ass compared to 8.x,
which just showed the right tests -- if not in the runner, then at-least very
reliably in the explorer.

[image][image]

And the explorer in 9.x! It's a hyperactive, overly sensitive, eager-to-please
puppy that reloads, refreshes, expands nodes and scrolls around -- all seemingly
with a mind of its own! Tests wink in and out of existence, groups expand
seemingly at random, the scrollbar extends and extends and extends to
accommodate all of the wonderful things that the unit-test explorer wants you to
see -- needs for you to see. Again, it's possible that this is due to our
abstract test fixtures, but this is new to 9.x. 8.2.x is perfectly capable of
displaying our tests in a far less effusive and frankly hyperactive manner.

[One last thing: output-formatting]

Even the output formatting has changed in 9.x, expanding all CR/LF pairs from
single-spacing to double-spacing. It's not a deal-breaker, but it's annoying:
copying text is harder, reading stack traces is harder. How could no one have
noticed this in testing?

[image]

[Conclusion]

The install/uninstall process is painless and supports jumping back and forth
between versions quite well, so I'll keep trying new versions of 9.x until the
test runner is as good as the one in 8.2.x is. For now, I'm back on 8.2.3. Stay
tuned.

--------------------------------------------------------------------------------


[1] In no particular order, we have used or are using:
  
     * DotMemory
     * DotTrace
     * DotPeek
     * DotCover
     * TeamCity
     * PHPStorm
     * WebStorm
     * PyCharm


[1] Although I was unable to install DotMemory without upgrading to ReSharper
    9.x, I was able to uninstall ReSharper 9.x afterwards and re-install
    ReSharper 8.x.