Rory Becker - DevExpress CodeRush Blog

August 2016 - Posts

  • CodeRush for Roslyn: The Fastest .Net Test Runner

    Once again it's time to benchmark the CodeRush Test Runner with the competition.

    So my last post on “CodeRush for Roslyn: Running Tests” sparked a few questions

    Like ‘Is the CodeRush test runner really that fast?’ and ‘How do other test runners stack up against CodeRush for Roslyn?’

    Test Conditions

    To reproduce this benchmark scenario, you will need:

    Software

    • 1 copy of Visual Studio 2015 – Update 3
    • 1 copy of the Nancy FX Solution, which includes 2,410 unit tests.
    • Latest versions of each test runner (I used the latest test runners available in August 2016).

    Hardware

    • I used my laptop –> Intel® Core™ i5 M 520 with 8Gb RAM.

    Test Procedure

    • Close any open instances of Visual Studio.
    • Start a fresh instance of Visual Studio with the test runner you want to benchmark already installed. Allow time to settle – see notes below.
    • Load the NancyFX solution. Allow time to settle – see notes below.
    • With the test runner you want to benchmark:
      • Run all tests. Note raw time as ‘Run 1’
      • Run all tests. Note raw time as ‘Run 2’
      • Run all tests. Note raw time as ‘Run 3’
      • Average the times from the three test runs. Enter the data in the table.
      • Make note of the number of tests detected by the test runner.
    • Close Visual Studio.
    Repeat these steps for each test runner you want to evaluate.

    Results

    We've changed competing product names so we don't unduly upset anybody.

    Here are the total times for each Test Runner on the Nancy FX Solution (lower is better):



    Here's our test run data, in table format:

    Notes:

    1. The "Test/Sec" column is calculated by taking the total number of tests (e.g., 2,410) divided by the test run time for each test runner. 
    2. The "CodeRush Is Faster By" shows how much faster CodeRush is than the competing tool. Values in this column are calculated by taking the CodeRush Test/Sec rate (e.g., 20.1) and dividing that by the Test/Sec rate for each competing product.
    3. The "JC" product tested failed to find 263 out of all 2,410 tests, so it's not doing as much work as the other test runners. Once this issue is fixed, the benchmark numbers for the "JC" product are likely to change.
    4. We realized the "RS" product we tested was incorrectly reporting total time to run all tests (erroneously reporting that it took much more time to run the tests than it actually did). So to be fair we used a stopwatch operated by a human to time the results for this product. Results manually timed like this might be off by as much as 1-2 seconds.
    5. Both the MS product and the JC product seem to have a "discovery" phase that runs every time before a test run actually gets started. During the discovery phase, test runner UI is blocked but the IDE remains responsive. We found this discovery phase for both products took two additional minutes to discover all the tests in the Nancy FX solution. We did not include those extra two minutes in calculating the total test run time for the MS and JC products, even though you could easily argue they are part of the test run. Note that the top three test runners (CodeRush, RS & TD) do not have a discovery phase, which means these three products are able to instantly start your test run.
    In the category of number of Tests Run Per Second, CodeRush is the clear winner with an average of over 20 tests (in the Nancy FX solution) run per second:
    The next closest competitor, RS, is only able to run 13.3 tests per second in the Nancy FX solution.

    No competitor comes close to the speed and efficiency of the CodeRush test runner.

    Summary

    Not all runners are created equally. Some are better than others.

    Features to look for in a good test runner:

    • Finds & runs 100% of your tests. At least one of the test runners we evaluated failed to find all the tests.
    • Finds your tests quickly. Two of the test runners we evaluated locked up test runner for an apparent "discovery phase" that lasted almost two minutes before the tests were actually run.
    • Collect test results as quickly as possible.

    No matter how you slice those numbers, CodeRush for Roslyn is still the Fastest Test runner for .NET development.

    Not convinced? Try it yourself. Download CodeRush for Roslyn and see just how much faster CodeRush is than your current test runner.

    And one last thing…. Check out the price. CodeRush for Roslyn can be yours for only $49, and CodeRush is so much more than just a test runner.

  • CodeRush for Roslyn: Running Tests

    CodeRush for Roslyn is RTM!

    That’s right CodeRush for Roslyn is out of preview and available. With it we are shipping the best ever version of our test runner.

    It’s the fastest, it’s the simplest, and it supports all the major testing frameworks (NUnit, XUnit, MSpec and VSUnit) out of the box.

    Inline Tests

    This test runner really is the simplest. With CodeRush you can run your tests direct from the editor.

    • Locate any Test

    CRRTestingExampleTest

    • Place your caret within it.
    • Run your test with Ctrl+T, Ctrl+R. (Test –> Run)

    When Run, this test passes and changes its icon to reflect its new state.

    CRRTestingExampleTest

    Naturally enough some tests do not pass. When a test fails, its Test Icon changes to indicate this, and the failing assertion is accentuated with a red background.

    CRRTestRunnerFailingTest

    More information can be found about the nature of the error if you hover the mouse over the Test Icon

    CRRTestRunnerFailingTestWithHover

    You can of course run multiple tests at once.

    The keystrokes you use depend on the breadth of tests you wish to run.

    For all tests in the Current File, use Ctrl+T, Ctrl+F. (Test –> File)
    For all tests in the Current Class, use Ctrl+T, Ctrl+C. (Test –> Class)
    For all tests in the Current Project, use Ctrl+T, Ctrl+P. (Test –> Project)
    For all tests in the Solution, use Ctrl+T, Ctrl+S. (Test –> Solution)

    Debugging a Test

    Sometimes running a test is not enough. In some cases it can be very useful to debug directly into a test.

    In this case you should:

    • Locate the Test
    • Place your caret within it
    • Press  Ctrl+T, Ctrl+D. (Test –> Debug)

    When invoked this way, the Test will be run with the debugger attached. You’ll be able to stop on break points and use watch windows in all the usual ways. This allows you to setup a test for a particular situation and then observe exactly how your code behaves in that scenario. Literally your code under lab conditions.

    Alternatively you can click the Test Icon to reveal the test context menu.

    CRRTestingInlineMenu

    Naturally you can Run test and Debug test from this menu. You can also locate the Go to runner to locate the current test within the Test Runner (see the next section for details on the Test Runner) or Run Test with Coverage.

    A full description of Code Coverage is a topic for another post, but the short version is that CodeRush will track which lines of your code are executed when a test is run. This can give you a quick estimate of which parts of your code might require additional testing.

    The last item is Add to Session. I’ll leave that one for a minute or two whilst we discuss the Test Runner.

    Test Runner

    This is the CodeRush TestRunner. Shown here with some of the Unit Tests within NancyFX

    CRRTestRunnerNancyTests

    You can launch the Test Runner in 3 ways.

    • From the Context menu of a test or test fixture. (Go to runner)
    • From the Main CodeRush menu (CodeRush | Test | Test Runner)
    • Using the universal Test Runner Shortcut (Ctrl + T, Ctrl+T)

    The default view of the Test Runner will show you all the projects containing tests and the counts of tests within those projects.

    All the standard options are available from within the Test Runner. From left to right we have ‘Run Selected Tests’, ‘Run All Tests’, ‘Debug Selected Tests’. Naturally enough we follow that with the ability to ‘Run Selected Tests with Coverage’ and ‘Run All Tests with Coverage

    When you run your tests, the number in the brackets of each node expands to show not just the total, but the number of passing and failing tests within that node.

    CRRTestRunnerNancyTestsRun

    If you turn on the ‘Show Duration’ option CRRTestRunnerToolbarShowDuration you can also see how long any given node (single test or group) took to execute.

    CRRTestRunnerNancyTestsRunDuration

    At all times the status bar will track the number of the total number of tests, how many of them are currently Pass or Fail and how many are currently designated Ignored or Explicit.

    CRRTestRunnerStatusBarNancyFX

    Grouping and Filtering

    Your tests can be grouped by Project, Namespace, Category, File Path or just as a Flat List.

    CRRTestRunnerGroupingDropdown

    You can show any combination of Not Run, Passed, Failed, Ignored or Explicit tests using the toolbar toggles.

    CRRTestRunnerFilterToggles

    Having determined the selection of tests you’re interested in, you can run any of those remaining visible through the use of the right click menu or the previously mentioned icons on the toolbar. This context menu is as for an individual test in the editor, but will now apply to all nodes within the one you right-click. This allows you run all tests in a namespace, project or category at the click of a button.

    If you’re looking for a particular test, but aren’t sure where it is, than the test search box is your friend. As you might expect, enter some search text, hit enter and CodeRush will find any test that matches all your criteria (Filter Toggles, and Search)

    CRRTestRunnerToolbarSearch

     

    How Fast is the CodeRush Test Runner?

    So the single simplest test is to instruct CodeRush to run all the tests in the Solution. Since I had the NancyFX solution loaded, I simply hit Ctrl+T, Ctrl+S and CodeRush located and ran the 2410 tests present.

    Obviously I needed something to compare this with, so I also took a well known Brand ‘X’ test runner for a spin and these are the results I got.

    Test Runner Test Execution Time
    Brand ‘X’ 182s
    CodeRush (Default settings) 143s
    CodeRush (All Cores) 128s

    So just using the default settings, CodeRush’s test runner is 39 seconds (22.5%) faster than the Brand ‘X’ test runner.

    But you’ll notice that there is another entry for CodeRush. When CodeRush is allowed to use all my processor’s cores, it gets even quicker, eventually toping out (on my machine) at a little over 54 seconds (30%) faster than the Brand ‘X’ test runner.

    Sessions

    So you’ll recall a few sections back I hinted at a new CodeRush test runner feature.

    The CodeRush test runner, like all other test runners has long supported the idea of running either a single test or a set of tests defined by a class, file, project or indeed the entire solution.

    However you don’t always, especially in legacy projects, have the convenience of having all appropriate tests in a single namespace or project where you need them. This wouldn’t necessarily be an issue if there weren’t those strange group of tests written in a dark corner which take an excessively long time to run. Then there’s that other set that you’ve been told not to run because no-one is quite sure that they don’t have side effects.

    Then there are the tests that you really do want to run. They should all exist in the same place, but for some unearthly reason they have been defined in a number of strange disparate locations in several different projects. You will of course bring these tests together in some carefully crafted namespaces in just one or 2 projects in good time, but right now is not the time. They work well. they are essential to the stability of the project. But they are disparate and a pain to use.

    Enter ‘CodeRush Testing Sessions’

    A CodeRush Testing Session is a very simple thing. (Well all the best things are aren’t they.) A Session is a named ad-hoc selection of tests. Nothing more. No need to alter code to add categories. Just pick the tests you care about, and start to work with them. Of course the real key with sessions is in how you use them.

    To create a new session…

    Just visit the context menu of any test (or group of tests) and choose Add to Session | Create New Session. Your chosen tests are added to a new session with the name “Session 1” or similar. The Test runner has a new Tab named for the Session you’ve created, containing only those tests that you added. Feel free to add more tests to the Session. As many (or as few) as you want.

    To name a session

    Simply double-click the name of the session, and you’ll be able to name it whatever you like. I like to use ‘WIP’ (Work in Progress) to house those tests that are useful to me in my current situation.

    To add more tests to a session

    Pick Add to Session | Add to WIP Session from the Test (or Test Fixture) context menu, either from an appropriate node in the Test Runner, or from within the editor itself.

    To use a session…

    Simply use the same standard buttons as you do when there are no active session. ‘Run All Tests’ and ‘Run All Tests with Coverage’ now run within the confines of your active Session. Naturally because the Test Runner hides tests (and fixtures) which aren’t in your current session, it’s far simpler to find the results you’re after.

    To switch between sessions…

    It shouldn’t really come as much of a surprise, that you can have multiple sessions on the go at the same time. Each time you create one, a new tab is added to represent the tests you added to it.

    You can have Sessions for:

    • Those tests you need to run every few minutes, to check you’re not breaking some part of the subsystem you’re working on.
    • All the tests in your solution except those few slow tests you try to avoid running unless absolutely necessary.
    • Those slow tests you’ve been avoiding, but which need to run every now and again, because slow or not, you do need them.

    There are no rules. You could pick 3 tests from one project, all the tests from a 2nd project and 6 tests from 6 different test fixtures in a 3rd project.

    This is a great feature for use when you know you have a need to run 2 or 3 tests (or sets of tests) which are related by something other than the standard mechanisms. I personally like to create 3 sessions ‘WIP’ (Work in Progress), ‘All but Slow’ and ‘Slow Tests’, but feel free to use sessions for anything you like.

    So how do you switch between session? Simple: Just select the tab which represents the session when you want to work with.

    Oh and one last thing: CodeRush Test Sessions will persist between VS sessions. Which means they will survive the closing and re-opening of both your solution and studio, allowing you to setup exactly the sessions you need, safe in the knowledge that you won’t have to rebuild them in your next coding session.

    Testing is fast, fun and easy again. So go out there and build some rock solid and fully tested code.

    CodeRush for Roslyn – Free to try. Only $49 to buy! (Includes a year of free updates)

LIVE CHAT

Chat is one of the many ways you can contact members of the DevExpress Team.
We are available Monday-Friday between 7:30am and 4:30pm Pacific Time.

If you need additional product information, write to us at info@devexpress.com or call us at +1 (818) 844-3383

FOLLOW US

DevExpress engineers feature-complete Presentation Controls, IDE Productivity Tools, Business Application Frameworks, and Reporting Systems for Visual Studio, along with high-performance HTML JS Mobile Frameworks for developers targeting iOS, Android and Windows Phone. Whether using WPF, ASP.NET, WinForms, HTML5 or Windows 10, DevExpress tools help you build and deliver your best in the shortest time possible.

Copyright © 1998-2017 Developer Express Inc.
All trademarks or registered trademarks are property of their respective owners