Download

StudioTools

TestRunner

CodeSpell

Forums

TCL JACL .NET

Cart

Getting Started with StudioTools

Installation and Setup

StudioTools is installed as an extension to Visual Studio. No external applications or libraries need to be installed for it to function properly. If you can run Visual Studio, you can run StudioTools.

Entering your license key

In order to run StudioTools you must have a valid license key. If you have not yet purchased a license, you can obtain a fully functional, time limited license key from this website for evaluation purposes. You will be prompted to enter your license key when starting Visual Studio after installing the product.

Download the sample application

Throughout this guide we'll be using the sample "Money" application included with the NUnit distribution. While not a particular complex project, it is one familiar to anyone who is already familiar with unit testing, regardless of their language of choice. While not absolutely necessary, if you want to follow along with the examples in this guide you can download the source code from the NUnit website.

Configuring your solution

If your application already includes unit tests written with NUnit, there's nothing more you'll need to do. StudioTools does not require any special configuration or enforce any constraints about where your tests are. You can put tests into their own project, their own namespace, or even along side your application code. StudioTools will find them.

NUnit compatibility

StudioTools is built on, and is 100% compatible with, NUnit, the de-facto standard framework for unit testing in .NET. NUnit itself is a descendent of JUnit and other xUnit testing frameworks already familiar to most developers. If you are already unit testing, there's nothing more to learn. StudioTools does not add anything to the NUnit API, and your tests remain compatible with other NUnit test runners such as NAnt and CruiseControl. If you are new to unit testing, then rest assure that the NUnit API is simple to learn and that you'll be picking up a skill that is increasingly expected of any .NET developer.

Adding NUnit references to your projects

It is not necessary to install NUnit in order for StudioTools to function properly. Since your unit tests are written with NUnit however, you will need to include the proper assembly references for your tests to compile, as you would with any other library:

You can include references to your own copies of NUnit, or use the ones bundled with StudioTools. To add references to the bundled DLLs, simply select the Fix NUnit References option from the Test Explorer | Options menu. StudioTools always ships with the latest version of NUnit. If you are following along with the sample application, you'll need to do this now.

Building your solution

No extra steps are required when building your solution for use with StudioTools. Each time you build, StudioTools scans your assemblies for unit tests and updates its internal list. The first time you load your solution therefore, and any time you make changes, they will not be recognized until after you have built. The editor integration is always active however, and will recognize unit tests as soon as you've written them.

Running unit tests

The goal of StudioTools is to make unit testing easy and convenient, so that you can make it an integral part of your development process. Tight integration with Visual Studio, and a variety of mechanisms for executing tests help make testing a natural, simple process.

Using the Test Explorer

The Test Explorer panel (TestMatrix | Test Explorer) automatically discovers all of the tests in your solution, and organizes them by assembly, namespace, and fixture into a familiar explorer style interface. You can use the Test Explorer to browse, navigate through and execute any of the unit tests in your solution.

From the drop down at the top of the Test Explorer windows you can choose your entire solution, or a single project to load. This allows you to narrow the scope of your testing to particular assembly if you choose. If you are following along with our example, select the Money project from the drop down. The Test Explorer will load all of the tests and display them in a tree organized by namespace. You can open any test in the editor by left clicking on its entry in the tree.

Running tests from the Test Explorer

The Test Explorer allows you to selectively run any combination of unit tests you would like. You can select individual test cases, specific test fixtures, all tests in a given namespace or project, and many any other combinations. Test Explorer lets you select or exclude entire projects or namespaces with one click. Simply check off the tests you want to run and click on the Run Checked Tests button in the toolbar (the one that looks like a pair arrows) to start the testing run.

Running tests from the Solution Explorer

You can run tests from the Solution Explorer. Right click on any solution, project, folder, or class to access a unit testing context menu.

Running tests from within the editor

The Test Explorer is great for running large numbers of tests but your day to day testing will most likely take place in the editor itself. StudioTools automatically marks all unit tests in the project with a special gutter icon (three colored balls) and uses special highlighting to indicate which tests have passed or failed.

To run a unit test from the editor, right click on the unit test icon in the gutter at the top of the test declaration. From the context menu select Run or Debug to execute the test. If you have not built since the last change, your code will automatically be compiled before running the test. This menu is also available via the normal editor context menu, accessed by right clicking anywhere inside of the test case.

Running tests from the keyboard

By far the most efficient way to run your tests is via a hot key. StudioTools defines a number of important hot keys that make it possible to run your tests without taking your hands off the keyboard. If you don't like the defaults, you can always change them through the options panel.

Interpreting the test results

When you run your tests the Test Results window automatically appears to give you feedback about the testing process. In addition the progress bars in the Test Explore track the progress of running tests and provide a summary of testing results. The Test Results window can be filtered or sorted as needed. If you would prefer to view the raw test output, it is visible through the regular Visual Studio output window.

The color of the icon beside each test in the Test Explorer and Test Results window indicates the result of the test. You can show or hide different types of results by clicking on the appropriate filter icons in the Test Results window.

Dealing with failing tests

When a test fails, its icon will turn red. Click on the failing test in either the Test Results window or the Test Explorer to navigate to its source it in the editor. As shown in the figure below, the line that caused the failure will be marked with a red stack trace icon. In this case we can see that the assertion on line 242 has failed. Hover your mouse pointer over the stack trace icon to see an explanation of the failure.

Failing tests are added to their own tab in the Test Explorer. This makes it easy to work down the list, fixing each test one at a time. If you want to step through the tests in the Visual Studio debugger, you can right click on the test in the Test Explorer and select the Debug Test option from the context menu. Be sure to set a breakpoint first!

Tracking your code coverage

Code coverage is metric which tells you which lines of your source code have been executed when you run your unit tests, and which have not. The ultimate goal is to provide unit tests that cover 100% of your source code in order to ensure that every line is backed by automated tests. Most importantly, code coverage shows you which lines of code were missed by your unit tests, and answers the all important question: How many tests are enough?

Why measuring code coverage is a critical part of the development process

Unit tests improve the quality and reliability of your code, but just having a few tests for each class is not enough. It's easy to write a unit test, but that doesn't mean that it's actually testing all of your code that it should. Code coverage is the only way to know what you're testing and what you aren't. Miss testing a few switch statement values? Forget to test that failing conditional? Coverage profiling will alert you to these holes in your automated testing so that you can get them plugged. As code changes over time, measuring your code coverage helps ensure that each new feature or bug fix is backed by automated tests. Unlike other coverage tools that are run (usually infrequently) through an external application, StudioTools runs inside of Visual Studio and provides you with automatic, immediate visual feedback in the editor so you are constantly aware of your coverage.

How code coverage is calculated

There are several ways to track code coverage, but they all boil down to keeping a log of each line of code that was executed during the course of a test run and comparing that to your source tree. Each line that was hit during the test run is covered, everything else is not. To do this, your code must be instrumented with special instructions to enable an appropriate level of tracking. Some coverage tools require you to manually add instrumentation statements to your source code, use an alternate compiler, or run your compiled assemblies through a post processing utility. Not only are these extra steps slow and unwieldy, they discourage your from monitoring your coverage as frequently as you should. StudioTools works differently, dynamically instrumenting your assemblies at the start of each test run. No extra steps are necessary. The execution information collected during the test run is used to build the coverage reports and editor markup that shows you your code coverage.

Recognizing uncovered code in the editor

Uncovered lines of code are indicated by red bars in their gutter and executed statements are marked with a green bar, if you have enabled this option. (Note that some lines such as braces are never executed per say, and will never be marked as either covered or uncovered.

Bring up the Money class from our example. Assuming you have run all the tests in the solution, you can see that the conditional on line 82 has not been never tested with a passing condition, because lines 83 and 84 were never executed. Note that if you have run anything less than your solution's entire test suite the coverage will only reflect the tests that you actually run, giving you more apparently uncovered code than you may have expected. If you wish, you can change the color of the missed statement indicators through the Test Matrix options, or turn them off all together.

Finding all of your uncovered code

When you are working on the unit tests for a particular class or method, the immediate feedback provided in the editor is a great way to measure your progress towards complete code coverage. When you wish to review coverage across your entire solution however, it's time to break out the Coverage Profiler panel for a full report. It shows you the coverage percentage of every class in your solution so you you can no what steps are necessary to improve your coverage.

The Coverage Profiler window shows you the coverage you've achieved across the entire solution, broken down by class, method, and lines of code. The percentage scales indicate the proportion of executable statements that were hit during the most recent test run. You can sort by either class name or percentage coverage, making it easy to find the classes most in need over improved unit tests. By default the Coverage Profiler includes information about every class in your solution. You can use the Namespace Filter at the top of the panel to limit the view to selected namespaces and assemblies. The slider bar lets you hide classes that have reached a certain percentage of coverage, basically a tolerance control that determines what percentage of coverage is sufficient. Classes which meet the coverage indicated on the slider will be hidden.