Dynamic tests with mstest and T4

 Mar 5th, 2011 

 , , , , , ,

If you used mstest and NUnit you might be aware of the fact that the former doesn’t support dynamic, data driven test cases. For example, the following scenario cannot be achieved with the out-of-box mstest: given a dataset, create distinct test cases for each entry in it, using a predefined generic test case.

The best result that can be achieved using mstest is a single testcase that will iterate through the dataset. There is one disadvantage: if the test fails for one entry in the dataset, the whole test case fails.

So, in order to overcome the previously mentioned limitation, I decided to create a text template that will generate the test cases for me. As an example, I will write some tests for an integer multiplication function that has 2 bugs in it:

public int Multiply(int a, int b)
{
    //This conditions are simulating the 2 bugs
    if (a == 0 && b == 1)
        return 100;
    if (a == 1 && b == 0)
        return -100;
    return a * b;
}

The classical approach (no dynamic test)

Without using any ‘hacks’, one could write the tests for the Multiply function in the following way:

//Tuple description <value of param a, value of param b, expected result>
private static readonly Tuple<int, int, int>[] TestData = new Tuple<int, int, int>[]{
    new Tuple<int, int, int>(0,0,0),
    new Tuple<int, int, int>(2,3,6),
    new Tuple<int, int, int>(1,0,0), //These will trigger one of the bugs
    new Tuple<int, int, int>(-2,-3,6),
    new Tuple<int, int, int>(0,1,0) //These will trigger one of the bugs
};
[TestMethod]
public void TestMultiply()
{
    foreach (var data in TestData)
    {
        Assert.AreEqual(data.Item3, Multiply(data.Item1, data.Item2),
                        "Failed for input ({0}, {1})", data.Item1, data.Item2);
    }
}

Running the test will surface only one of the bugs, the one triggered by the input (1,0):

This is not only bad because it doesn’t give a complete overview of the bugs but it also violates the principle of one assertion per test because more than one assertion could be triggered in the test case above.

The T4 approach (dynamic test)

A text templates, from MSDN:

… is a mixture of text blocks and control logic that can generate a text file. The control logic is written as fragments of program code in Visual C# or Visual Basic. The generated file can be text of any kind, such as a Web page, or a resource file, or program source code in any language. Text templates can be used at run time to produce part of the output of an application. They can also be used for code generation, in which the templates help build part of the source code of an application.

Text templates are invoked before compilation in order to generate some code that will be used in the compilation process. Preprocessed text templates are used to generate templates that can be invoked at runtime in order to generate new files. I am going to use the former one.

So, the goal is to have one assertion per test and show all bugs. For this, different test cases, for each input, are needed. A possible approach would be to write manually each test:

[TestMethod]
public void TestMultiply_Input_0_0()
{
    TestMultiply(0, 0, 0);
}
[TestMethod]
public void TestMultiply_Input_2_3()
{
    TestMultiply(2,3,6);
}
[TestMethod]
public void TestMultiply_Input_1_0()
{
    TestMultiply(1, 0, 0);
}
[TestMethod]
public void TestMultiply_Input_Minus2_Minus3()
{
    TestMultiply(-2, -3, 6);
}
[TestMethod]
public void TestMultiply_Input_0_1()
{
    TestMultiply(0, 1, 0);
}
public void TestMultiply(int a, int b, int expected)
{
    Assert.AreEqual(expected, Multiply(a, b), "Failed for input ({0}, {1})", a, b);
}

While this approach doesn’t violate the two principles above, it creates a code that is hard to maintain and is a pain to write it for many inputs. Just imagine having 100 tuples in the dataset. The result is the expected one:

A smarter approach is to generate all those test methods. Can you see the pattern they follow?

  • The title of the method is composed of the string “TestMultiply_Input_” followed by the first input value, followed by the string “_” and then followed by the second input value
  • The body of the method is made by a call to a generic test method (TestMultiply) using the two input values and the expected result
  • For negative values, the minus sign is replaced by the literal “Minus”

So, a text template (not a preprocessed text template!) can be written to the test cases. It can be added to a Visual Studio 2010 project by adding a new item of type “Visual C# Items\General\Text Template”. I will name the file “GeneratedTestCases.tt” and the generated code file will be “GeneratedTestCases.generated.cs”.

<#@ template debug="false" hostspecific="true" language="C#" #>
<#@ output extension=".generated.cs" #>
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace Tests
{
    [TestClass]
    public class MultiplicationTests
    {
<#
//Tuple description <value of param a, value of param b, expected result>
Tuple<int, int, int>[] TestData = new Tuple<int, int, int>[]{
    new Tuple<int, int, int>(0,0,0),
    new Tuple<int, int, int>(2,3,6),
    new Tuple<int, int, int>(1,0,0), //These will trigger one of the bugs
    new Tuple<int, int, int>(-2,-3,6),
    new Tuple<int, int, int>(0,1,0) //These will trigger one of the bugs
};
foreach (var data in TestData)
{
#>
		[TestMethod]
		public void TestMultiply_Input_<#= data.Item1.ToString().Replace("-", "Minus") #>_<#= data.Item2.ToString().Replace("-", "Minus") #>()
		{
			TestMultiply(<#= data.Item1 #>, <#= data.Item2 #>, <#= data.Item3 #>);
		}
<#
}
#>
		public void TestMultiply(int a, int b, int expected)
        {
            Assert.AreEqual(expected, Multiply(a, b), "Failed for input ({0}, {1})", a, b);
        }
	//Just write this method here. In reality, this method will be somewhere else
	public int Multiply(int a, int b)
        {
            //This conditions are simulating the 2 bugs
            if (a == 0 && b == 1)
                return 100;
            if (a == 1 && b == 0)
                return -100;
            return a * b;
        }
    }
}

This text template will generate the test class. It looks quite ugly and writing it is not trivial because there is not intellisense or syntax coloring in Visual Studio.

Let me explain what it does:

  1. All the text that is not between <# #>, <#@ #> or <#= #> is just copied to the output file (ex: lines 3-12)
  2. Line 2 specifies the extension of the output file
  3. Code between <# #> is C# code and will be executed at generation time
  4. Code between <#= #> is C# code, will be executed at generation time and it’s output will be written to the generated file. It must be a single statement returning a non void value.
  5. For each tuple in the data set, a test method is written to the output file.

It can be improved a little by working with partial classes. Instead of writing all the code in the tt file, the C# code can be placed in a cs file. So, I’ll do some changes to the tt file by removing the last methods and making the class partial:

<#@ template debug="false" hostspecific="true" language="C#" #>
<#@ output extension=".generated.cs" #>
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace Tests
{
    [TestClass]
    public partial class MultiplicationTests
    {
<#
//Tuple description <value of param a, value of param b, expected result>
Tuple<int, int, int>[] TestData = new Tuple<int, int, int>[]{
    new Tuple<int, int, int>(0,0,0),
    new Tuple<int, int, int>(2,3,6),
    new Tuple<int, int, int>(1,0,0), //These will trigger one of the bugs
    new Tuple<int, int, int>(-2,-3,6),
    new Tuple<int, int, int>(0,1,0) //These will trigger one of the bugs
};
foreach (var data in TestData)
{
#>
		[TestMethod]
		public void TestMultiply_Input_<#= data.Item1.ToString().Replace("-", "Minus") #>_<#= data.Item2.ToString().Replace("-", "Minus") #>()
		{
			TestMultiply(<#= data.Item1 #>, <#= data.Item2 #>, <#= data.Item3 #>);
		}
<#
}
#>
    }
}

Then, create a class called “MultiplicationTests” that is also partial and contains the methods removed from the tt file:

using Microsoft.VisualStudio.TestTools.UnitTesting;
namespace Tests
{
    partial class MultiplicationTests
    {
        public void TestMultiply(int a, int b, int expected)
        {
            Assert.AreEqual(expected, Multiply(a, b), "Failed for input ({0}, {1})", a, b);
        }
        //Just write this method here. In reality, this method will be somewhere else
        public int Multiply(int a, int b)
        {
            //This conditions are simulating the 2 bugs
            if (a == 0 && b == 1)
                return 100;
            if (a == 1 && b == 0)
                return -100;
            return a * b;
        }
    }
}

Now regenerate the template and compile.

Conclusion

The presented approach is good and definetely has some advantages but is not perfect. Here are some dissadvantages:

  • Text Template editor, in VS, is just plain text with not syntax coloring or intellisense
  • The compilation errors are sometime ambiguous
  • Debugging a text template is something you would like to avoid :)
  • For a few test cases, more code is written
  • In case of new test cases, the code must be recompiled

Advantages:

  • Shows more bugs earlier
  • Allows one assertion per test even for large data sets
  • The code is compiled and not evaluated at runtime (maybe there is a performance gain)
  • Less manual code duplication

Advice: try to write as less code as possible in the tt file. Move as much as possible to cs files. Any assembly, except the one in which the tt file is, can be referenced at generation time.

If you have troubles going through this tutorial, download the complete C# project: Download IconT4_mstest (3.56 kB)