CodeRush for Roslyn: The Fastest .Net Test Runner

Once again it's time to benchmark the CodeRush Test Runner with the competition.

So my last post on “CodeRush for Roslyn: Running Tests” sparked a few questions

Like ‘Is the CodeRush test runner really that fast?’ and ‘How do other test runners stack up against CodeRush for Roslyn?’

Test Conditions

To reproduce this benchmark scenario, you will need:

Software

• 1 copy of Visual Studio 2015 – Update 3
• 1 copy of the Nancy FX Solution, which includes 2,410 unit tests.
• Latest versions of each test runner (I used the latest test runners available in August 2016).

Hardware

• I used my laptop –> Intel® Core™ i5 M 520 with 8Gb RAM.

Test Procedure

• Close any open instances of Visual Studio.
• Start a fresh instance of Visual Studio with the test runner you want to benchmark already installed. Allow time to settle – see notes below.
• Load the NancyFX solution. Allow time to settle – see notes below.
• With the test runner you want to benchmark:
  • Run all tests. Note raw time as ‘Run 1’
  • Run all tests. Note raw time as ‘Run 2’
  • Run all tests. Note raw time as ‘Run 3’
  • Average the times from the three test runs. Enter the data in the table.
  • Make note of the number of tests detected by the test runner.
• Close Visual Studio.

Results

We've changed competing product names so we don't unduly upset anybody.

Here are the total times for each Test Runner on the Nancy FX Solution (lower is better):

Here's our test run data, in table format:

Notes:

1. The "Test/Sec" column is calculated by taking the total number of tests (e.g., 2,410) divided by the test run time for each test runner. 
2. The "CodeRush Is Faster By" shows how much faster CodeRush is than the competing tool. Values in this column are calculated by taking the CodeRush Test/Sec rate (e.g., 20.1) and dividing that by the Test/Sec rate for each competing product.
3. The "JC" product tested failed to find 263 out of all 2,410 tests, so it's not doing as much work as the other test runners. Once this issue is fixed, the benchmark numbers for the "JC" product are likely to change.
4. We realized the "RS" product we tested was incorrectly reporting total time to run all tests (erroneously reporting that it took much more time to run the tests than it actually did). So to be fair we used a stopwatch operated by a human to time the results for this product. Results manually timed like this might be off by as much as 1-2 seconds.
5. Both the MS product and the JC product seem to have a "discovery" phase that runs every time before a test run actually gets started. During the discovery phase, test runner UI is blocked but the IDE remains responsive. We found this discovery phase for both products took two additional minutes to discover all the tests in the Nancy FX solution. We did not include those extra two minutes in calculating the total test run time for the MS and JC products, even though you could easily argue they are part of the test run. Note that the top three test runners (CodeRush, RS & TD) do not have a discovery phase, which means these three products are able to instantly start your test run.

Summary

Not all runners are created equally. Some are better than others.

Features to look for in a good test runner:

• Finds & runs 100% of your tests. At least one of the test runners we evaluated failed to find all the tests.
• Finds your tests quickly. Two of the test runners we evaluated locked up test runner for an apparent "discovery phase" that lasted almost two minutes before the tests were actually run.
• Collect test results as quickly as possible.

No matter how you slice those numbers, CodeRush for Roslyn is still the Fastest Test runner for .NET development.

Not convinced? Try it yourself. Download CodeRush for Roslyn and see just how much faster CodeRush is than your current test runner.

And one last thing…. Check out the price. CodeRush for Roslyn can be yours for only $49, and CodeRush is so much more than just a test runner.

Free DevExpress Products - Get Your Copy Today