Visual Studio’s Greatest Test Runner? Take a Look at CodeRush 10.2

15 December 2010

CodeRush 10.2 was released last week and the Test Runner that ships inside appears to be the greatest Test Runner available for .NET development.

So before I show you why the CodeRush Test Runner is the best, let’s first consider the criteria by which we might judge greatness. I suggest greatness can be measured in one or more of the following:

  • Native support for your test frameworks
  • Ability to run unit tests quickly
  • A user interface prioritizing clarity over noise
  • Ability to run and debug a single test
  • If you can think of important criteria that isn’t listed above please let me know in a comment below. However, if your criteria for greatness is listed above, we have really good news for you: The CodeRush Test Runner is going to impress you.

     

    Native Test Framework Support

    The CodeRush Test Runner natively supports NUnit, MSTest, xUnit, MbUnit and Silverlight. Native support means CodeRush fully supports each framework, so new extensions and new features added to the framework can be exploited immediately without needing an update from DevExpress. No other test runner out there has built-in native support for all these testing frameworks.

     

    Performance

    The CodeRush Test Runner is the fastest test runner available for .NET. With our new multi-threaded run option, you can run tests in multiple assemblies in separate threads, resulting in a test run that is up to three times as fast as the competition.

    TestRunCompareTime3

    Like waiting? You don’t have to. CodeRush finishes your multi-assembly test run up to three times faster.

     

    Run & Debug a Single Test

    Seems like a simple request, but being able to run and debug a single test is important, especially to Silverlight developers. CodeRush is currently the only unit test framework that lets you run or debug a single test in all frameworks supported, including Silverlight. Not even Visual Studio’s built-in test runner can do this.

     

    Clarity in the User Interface

    The CodeRush Test Runner user interface delivers more clarity and less noise than any of the competing test runners, helping you see essential information quickly.

    Here’s the output of a failed test in CodeRush:

    CodeRushTestRunnerResultsLineView

    Differences between expected and actual values are highlighted automatically.

    Redundant details (e.g., “String lengths are both 28. Strings differ at index 4.”) are removed to reduce noise and enhance clarity. CodeRush is the only test runner that visually highlights differences between expected and actual results like this.

    If you right-click the debug output…

    TestRunnerMenu

    you can change the call stack display from line view to tree view, which groups methods by parenting class, like this:

    CodeRushTestRunnerResultsTreeView

    You may find this call stack view more interesting as methods are grouped by owning class.

    You can change the syntax highlight colors in the DevExpress Options dialog on the Unit Testing\Test Runner Window options page:

    OptionsUnitTesting

    Note: If you don’t see this options page, change the Level combo box from "New User” to “Advanced”.

    AdvancedLevel

     

    Compare Results with the Competition

    Compare the readability of the CodeRush Test Runner results with similar results from competing test runners.

    Competitor #1:

    TestDrivenDotNetResults2

    This competing test runner (shown above) does not ship a test result tool window for Visual Studio, so test results can only be viewed through the Output window (the CodeRush Test Runner can send output here as well if needed). Note that only the first change is shown (the letter “f” in “fox”). Subsequent differences (e.g., “dog” vs. “log”) are much harder to see and must be determined by visually scanning and comparing the results.

     

    Competitor #2:

    resharperResult2 

    This competitor makes the most important information harder to read by including redundant details (e.g., “String lengths are both 28”) in the results. Paradoxically, the essential information is rendered in black text on a gray background. Contrast for important information needs to be higher so it’s easy to see. And just like competitor #1, only the first difference is shown.

     

    Competitor #3:
    JustCodeResults2

    This competitor shows the results in a variable-width font, so the difference indicator actually points to the wrong character (e.g., the “h” in “The” – but there’s no difference there). Also, this competitor wraps the results in the window, which can make reading the call stack more challenging.

     

    Here is the result from CodeRush Test Runner again:

    CodeRushTestRunnerResultsLineView

    Which presentation do you prefer to read?

     

    In-source Test Result Icons

    CodeRush also makes it easier to work with tests in the source code, with test result icons appearing next to each test case. In the screenshot below, the first test has not run yet (TestTube), however the next two tests have run, one passing (TestPassed) and the other failing (TestFailed):

    TestMethodIcons

     

    If your hand is on the mouse, you can click the icon to bring up a test case menu, which allows you to run or debug that test.

    TestCaseMenu

     

    Test fixtures & namespaces containing unit tests have icons summarizing the run results of their contents. For example, the namespace below and the test fixture both have a red “X” (TestFailed), indicating they contain at least one test which has failed:

    TestCaseParents

     

    If you hover the mouse over one of these test fixture icons, you can see a summary of results for unit tests in this class.

    TestRunSummary

     

    If you click one of these test fixture or test namespace icons, a popup menu appears that allows you to run or debug all tests contained within.

    RunAllContainedTests

     

    If your hands are on the keyboard, don’t worry – CodeRush gives you powerful commands to run test cases without reaching for the mouse.

     

    Full Keyboard Control

    CodeRush offers full keyboard control over your test runs from inside Visual Studio. The keyboard shortcuts for running tests that ship with 10.2:

    Shortcut   Behavior
    Ctrl+T, S   Run all tests in the solution
    Ctrl+T, P   Run all tests in the active project
    Ctrl+T, F   Run all tests in the active file
    Ctrl+T, R   Run active test
    Ctrl+T, D   Debug active test
    Ctrl+T, T   Show the CodeRush Test Runner

    If you prefer the Visual Studio shortcuts, CodeRush ships an alternate collection of bindings reflecting those commands. You can change the bindings for any of these commands if you like.

     

    What’s Coming in 2011

    OK, so we just shipped 10.2, so it seems a bit crazy to tell you what’s coming in 11.1. But, you know, things get a little crazy over here at DevExpress from time to time. Our intention is to continue to maintain test runner dominance among all .NET alternatives, including Visual Studio. With that in mind, we will add support for even more testing frameworks, add more commands for controlling which test cases to run, and we’re working on improving visualization of code coverage, as well as a bigger feature we are going to save for later discussions.

    no comments
    No Comments

    Please login or register to post comments.