TestRunner for Visual Studio 2005 Manual

Measuring Performance

Looking at how folks were testing with NUnit-Gui, it was apparent that test timings are often used as a poor-man's profiler. Considering how expensive profilers generally are, it's a practical decision. TestRunner is splitting the difference, by including a profiler in the package that can gather line level timings, total times, average times, and hit counts just like an expensive profiler, but in an easier to use (and afford!) format.

The basic scenario for performance profiling with TestRunner is to create a test case that exercises a potentially slow area of code. This is an alternative to writing a throw away 'hammer' exe program that calls you code, and more convenient because you can pack performance tests into a separate dll and save them, tracking performance over time.

The second scenario for performance profiling in TestRunner is the 'aha!' factor. By having each line report its share of runtime, you'll find surprising results as you test about what does and doesn't take a lot of time. It's educational, and will help you build up a measured foundation of performance tuning knowledge, rather than straight guessing or rules of thumb.

Performance Scenario

Everyone always seems to if not argue, at least ponder the relative performance difference between a string += and a StringBuilder. Let's set up a simple test case to evaluate this with TestRunner. Make a new library project and click Tools | TestRunner Performance. Here is the code, nothing complicated, just building up a string with 10000 appends with both approaches:

using System;
using System.Collections.Generic;
using System.Text;
using NUnit.Framework;

namespace ClassLibrary4 {
    [TestFixture]
    public class Class1 {
        private string StringBased(int count) {
            string ret = "";
            for (int i = 0; i < count; i++) {
                ret += i.ToString();
            }
            return ret;
        }
        private string StringBuilderBased(int count) {
            StringBuilder ret = new StringBuilder();
            for (int i = 0; i < count; i++) {
                ret.Append(i.ToString());
            }
            return ret.ToString();
        }

        [Test]
        public void Compare() {
            string one = StringBased(10000);
            System.Diagnostics.Trace.WriteLine(one);
            string two = StringBased(10000);
            System.Diagnostics.Trace.WriteLine(two);
        }
        
    }
}

Feedback

TestRunner provides you with an overall summary of performance in the tool window. Looking at the results, it seems pretty clear that our string based approach is taking up the bulk of the time. In fact, the string builder based approach is so fast for 10,000 appends it doesn't even take 100 milliseconds.

That's pretty conclusive, but it only tells us the method is slow, not what we did in that method that was slow. It's possible that it isn't actually the +=... well not really:

We can see in the editor that the += is clearly the culprit. What you can't tell from the screen shot is that if you hover the mouse in the left gutter, you'll get a tooltip that tells you the time, percentage, and hits for that line. Inline performance profiling, all without having to set up or learn another tool or do anything but run your unit tests.

Recap

TestRunner is a really fast and easy way to run tests and gather performance statistics. If you've use another profiler, you know they have a lot of overhead on the runtime, making them fairly aggravating to use. TestRunner won't slow you down, you'll be surprised how fast it is. If you can make your classes and components fast at the unit level, you are on your way to having a well performing application. Since tuning almost always comes down to the little things, like a += over a builder, unit based performance testing is a different, and precise, and easier method than setting up a large profiler. Now, TestRunner won't help you with load or scale testing, that's for other tools, but it will help you make sure your classes are as fast and accurate as they can be.

©2003-2006 Mailframe