CodeRush for Roslyn: Running Tests

CodeRush for Roslyn is RTM!

That’s right CodeRush for Roslyn is out of preview and available. With it we are shipping the best ever version of our test runner.

It’s the fastest, it’s the simplest, and it supports all the major testing frameworks (NUnit, XUnit, MSpec and VSUnit) out of the box.

Inline Tests

This test runner really is the simplest. With CodeRush you can run your tests direct from the editor.

• Locate any Test

• Place your caret within it.
• Run your test with Ctrl+T, Ctrl+R. (Test –> Run)

When Run, this test passes and changes its icon to reflect its new state.

Naturally enough some tests do not pass. When a test fails, its Test Icon changes to indicate this, and the failing assertion is accentuated with a red background.

More information can be found about the nature of the error if you hover the mouse over the Test Icon

You can of course run multiple tests at once.

The keystrokes you use depend on the breadth of tests you wish to run.

For all tests in the Current File, use Ctrl+T, Ctrl+F. (Test –> File)
For all tests in the Current Class, use Ctrl+T, Ctrl+C. (Test –> Class)
For all tests in the Current Project, use Ctrl+T, Ctrl+P. (Test –> Project)
For all tests in the Solution, use Ctrl+T, Ctrl+S. (Test –> Solution)

Debugging a Test

Sometimes running a test is not enough. In some cases it can be very useful to debug directly into a test.

In this case you should:

• Locate the Test
• Place your caret within it
• Press  Ctrl+T, Ctrl+D. (Test –> Debug)

When invoked this way, the Test will be run with the debugger attached. You’ll be able to stop on break points and use watch windows in all the usual ways. This allows you to setup a test for a particular situation and then observe exactly how your code behaves in that scenario. Literally your code under lab conditions.

Alternatively you can click the Test Icon to reveal the test context menu.

Naturally you can Run test and Debug test from this menu. You can also locate the Go to runner to locate the current test within the Test Runner (see the next section for details on the Test Runner) or Run Test with Coverage.

A full description of Code Coverage is a topic for another post, but the short version is that CodeRush will track which lines of your code are executed when a test is run. This can give you a quick estimate of which parts of your code might require additional testing.

The last item is Add to Session. I’ll leave that one for a minute or two whilst we discuss the Test Runner.

Test Runner

This is the CodeRush TestRunner. Shown here with some of the Unit Tests within NancyFX

You can launch the Test Runner in 3 ways.

• From the Context menu of a test or test fixture. (Go to runner)
• From the Main CodeRush menu (CodeRush | Test | Test Runner)
• Using the universal Test Runner Shortcut (Ctrl + T, Ctrl+T)

The default view of the Test Runner will show you all the projects containing tests and the counts of tests within those projects.

All the standard options are available from within the Test Runner. From left to right we have ‘Run Selected Tests’, ‘Run All Tests’, ‘Debug Selected Tests’. Naturally enough we follow that with the ability to ‘Run Selected Tests with Coverage’ and ‘Run All Tests with Coverage’

When you run your tests, the number in the brackets of each node expands to show not just the total, but the number of passing and failing tests within that node.

If you turn on the ‘Show Duration’ option you can also see how long any given node (single test or group) took to execute.

At all times the status bar will track the number of the total number of tests, how many of them are currently Pass or Fail and how many are currently designated Ignored or Explicit.

Grouping and Filtering

Your tests can be grouped by Project, Namespace, Category, File Path or just as a Flat List.

You can show any combination of Not Run, Passed, Failed, Ignored or Explicit tests using the toolbar toggles.

Having determined the selection of tests you’re interested in, you can run any of those remaining visible through the use of the right click menu or the previously mentioned icons on the toolbar. This context menu is as for an individual test in the editor, but will now apply to all nodes within the one you right-click. This allows you run all tests in a namespace, project or category at the click of a button.

If you’re looking for a particular test, but aren’t sure where it is, than the test search box is your friend. As you might expect, enter some search text, hit enter and CodeRush will find any test that matches all your criteria (Filter Toggles, and Search)

How Fast is the CodeRush Test Runner?

So the single simplest test is to instruct CodeRush to run all the tests in the Solution. Since I had the NancyFX solution loaded, I simply hit Ctrl+T, Ctrl+S and CodeRush located and ran the 2410 tests present.

Obviously I needed something to compare this with, so I also took a well known Brand ‘X’ test runner for a spin and these are the results I got.

So just using the default settings, CodeRush’s test runner is 39 seconds (22.5%) faster than the Brand ‘X’ test runner.

But you’ll notice that there is another entry for CodeRush. When CodeRush is allowed to use all my processor’s cores, it gets even quicker, eventually toping out (on my machine) at a little over 54 seconds (30%) faster than the Brand ‘X’ test runner.

Sessions

So you’ll recall a few sections back I hinted at a new CodeRush test runner feature.

The CodeRush test runner, like all other test runners has long supported the idea of running either a single test or a set of tests defined by a class, file, project or indeed the entire solution.

However you don’t always, especially in legacy projects, have the convenience of having all appropriate tests in a single namespace or project where you need them. This wouldn’t necessarily be an issue if there weren’t those strange group of tests written in a dark corner which take an excessively long time to run. Then there’s that other set that you’ve been told not to run because no-one is quite sure that they don’t have side effects.

Then there are the tests that you really do want to run. They should all exist in the same place, but for some unearthly reason they have been defined in a number of strange disparate locations in several different projects. You will of course bring these tests together in some carefully crafted namespaces in just one or 2 projects in good time, but right now is not the time. They work well. they are essential to the stability of the project. But they are disparate and a pain to use.

Enter ‘CodeRush Testing Sessions’

A CodeRush Testing Session is a very simple thing. (Well all the best things are aren’t they.) A Session is a named ad-hoc selection of tests. Nothing more. No need to alter code to add categories. Just pick the tests you care about, and start to work with them. Of course the real key with sessions is in how you use them.

To create a new session…

Just visit the context menu of any test (or group of tests) and choose Add to Session | Create New Session. Your chosen tests are added to a new session with the name “Session 1” or similar. The Test runner has a new Tab named for the Session you’ve created, containing only those tests that you added. Feel free to add more tests to the Session. As many (or as few) as you want.

To name a session

Simply double-click the name of the session, and you’ll be able to name it whatever you like. I like to use ‘WIP’ (Work in Progress) to house those tests that are useful to me in my current situation.

To add more tests to a session

Pick Add to Session | Add to WIP Session from the Test (or Test Fixture) context menu, either from an appropriate node in the Test Runner, or from within the editor itself.

To use a session…

Simply use the same standard buttons as you do when there are no active session. ‘Run All Tests’ and ‘Run All Tests with Coverage’ now run within the confines of your active Session. Naturally because the Test Runner hides tests (and fixtures) which aren’t in your current session, it’s far simpler to find the results you’re after.

To switch between sessions…

It shouldn’t really come as much of a surprise, that you can have multiple sessions on the go at the same time. Each time you create one, a new tab is added to represent the tests you added to it.

You can have Sessions for:

• Those tests you need to run every few minutes, to check you’re not breaking some part of the subsystem you’re working on.
• All the tests in your solution except those few slow tests you try to avoid running unless absolutely necessary.
• Those slow tests you’ve been avoiding, but which need to run every now and again, because slow or not, you do need them.

There are no rules. You could pick 3 tests from one project, all the tests from a 2nd project and 6 tests from 6 different test fixtures in a 3rd project.

This is a great feature for use when you know you have a need to run 2 or 3 tests (or sets of tests) which are related by something other than the standard mechanisms. I personally like to create 3 sessions ‘WIP’ (Work in Progress), ‘All but Slow’ and ‘Slow Tests’, but feel free to use sessions for anything you like.

So how do you switch between session? Simple: Just select the tab which represents the session when you want to work with.

Oh and one last thing: CodeRush Test Sessions will persist between VS sessions. Which means they will survive the closing and re-opening of both your solution and studio, allowing you to setup exactly the sessions you need, safe in the knowledge that you won’t have to rebuild them in your next coding session.

Testing is fast, fun and easy again. So go out there and build some rock solid and fully tested code.

CodeRush for Roslyn – Free to try. Only $49 to buy! (Includes a year of free updates)

Free DevExpress Products - Get Your Copy Today