DevTalk.net

Most people are used to the idea of continuous integration environments running tests. However, I often find myself wanting to have the tests run all the time. Such a practice is called Continuous Testing, and in the .Net space we currently don’t have a solution that allows this out-of-the-box.

However, there is a small lifehack you can employ that will get you there (to some extent) provided you buy into the idea of using multiple computers for development. Why multiple computers? Well, the assumption is that some stuff needs to run ‘elsewhere’ to prevent interference with normal operation of the machine you’re developing on.

The Solution

So here’s how I do it. Let’s say I have a solution that I want to test continuously, i.e. I want the tests rebuilt and run so frequently that I always get that ‘dashboard’ feeling of knowing what’s going on. Here’s what I do to get this.

1. Place the source code in synchronized cloud storage

First, stick your code in cloud storage that syncs with many machines. DropBox is great – it gives you 2Gb free space, and syncs flawlessly on all your machines.

2. Optimize test-runner machine performance

Ensure that the machine that builds the tests and runs them is optimized for just that. Do the lightest possible OS install (e.g., Windows XP). Also, ensure disk access is lightning-fast: I swap to RAM, and you can’t really get any better speed than this. Seeing how most developers have 8-16Gb RAM nowadays, allocating a bit of that and fitting a solution in there isn’t such a big problem.

3. Set up the scheduler to build your solution

Pick a good time interval and set the Windows scheduler to run your build script. This will result in your solution being rebuilt over and over and over. If you followed my advice about keeping the solution in RAM (which, IMHO, totally fits with the cloud storage practice), builds should be somewhat faster than builds from your Raptor or whatnot. Don’t forget to fiddle MSBuild to do its multithreading right.

Of course, if you have a cloud build solution (IncrediBuild, Electric Cloud, etc.), just use that.

4. Set up the test runner to rerun all tests whenever binaries change

That’s the easy part – get your runner to rerun tests whenever your build artifacts change. That way, you will be constantly informed about how many tests you caused to fail. Naturally, this implies you have a separate screen that shows you tests and allows you to navigate them. To be able to navigate tests on a different computer, just use Synergy.

Discussion

The approach I describe above is workable, but it’s far from perfect. Here are some issues you will encounter with it.

First, there’s no way to avoid rebuilding the whole solution, because currently no software is able to intelligently detect which systems a particular tests affects. And even if you could, it would still not work because, currently, we simply have a blanket call for a single build script whatever happens, whereas to have smarter builds we would need to specify explicitly which test assemblies need to be rebuilt.

Another problem is that a solution like DropBox does not sync with our building, which sometimes causes problems when we try to do a build while the file is being updated. Because we cannot reasonably predict when DropBox will be syncing files, we just have to hope there’s no collision. Also, on a side note, I don’t recommend sharing DropBox between developers: use ordinary source control for that. If you try to use DropBox as source control, developers will be constantly interrupting each other’s work, causing uncompilable code, and so on. In other words: the solution I present currently works best for just one developer.

Finally, if you’re into things like coverage, you’re somewhat out of luck: you won’t be able to view coverage inside on your development machine because coverage calculations have actually happened elsewhere. Of course, nothing prevents you from running code coverage analysis and simply presenting aggregated results, but those are more or less useless without you actually being able to tell which spots your tests seem to miss.