I think it was in 199X when I got my first experience of ‘serious’ programming. One day, being fed up with toying about with Delphi (it was relevant at that time, unlike now), I went out and bought one of those book+CD packages. The package included two books on Programming Windows with Visual Studio 97, a refernce book (essentially, a very large reprint of a chunk of MSDN), and the CDs containing a trial version (the equivalent of today’s Express) of Visual Studio. I installed it on my 166MHz 16Mb RAM machine, and I was hooked.

The fact that I got to actually write MFC apps was amazing. After all, MFC with all its insane macros and types such as CString that supplanted ordinary STL types (was there an STL back then?) were about as appropriate for learning C++ as it is appropriate to lose virginity with a horse. (Sorry, really crude example, but true.)

At any rate, I seem to have somehow survived MFC (without writing one single meaningful app, of course) and moved on to just writing console apps. At the moment I was trying to implement the gaming rules of AD&D (2^nd edition, I beleive), and I distincly remember my excitement of learning about Microsoft’s __declspec(property) which, for those of you who do not know, allows one to have C#-style properties. Yes, we really had this capability ages ago (and still do).

However, at some point, C++ gave way first to Java, then to C#. I suppose that the real problem was the creating a good GUI app was much easier using .NET. That and the fact that C# had usable strings, memory management, and a whole host of other things. You dodn’t have to deal with arcane things such as Boost, and even STL which is arcane, especially if you consider its error messages.

Recently

Fast-forward a few years, and I’ve started using C++ again. Why? Well, I think the reason I got back into it was image processing. In C#, manipulating an image is easy (the API is very intuitive and System.Drawing.Bitmap is nice), but the speed basically kills the effort outright. So what I did is I started using P/Invoke to basically Lock() all the bytes in place, send it off to a C++ DLL to process, and then return the results back in.

Performance increased tenfold. And sure, you can fiddle around in C# with unsafe and pointers, but why bother?

Anyways, the real issue with C++ compared to C# to me, at least, seemed to be the fact that C++ is generally superior for parallelization. That’s why, at some point, I completely gave up on using the Microsoft compiler (which, to be fair, has improved in recent years) and focussed on using Intel Parallel Studio. The great thing about IPS is that, first, it’s a terrific set of compiler+libraries+tools, and is a real joy to use provided you invest some time in learning how it all works; and second, the Intel compiler is available on non-Windows OSs, which helps in terms of uniformity, even if I don’t use other OSs as much.

In terms of parallelization, .NET has TPL whereas C++ has

• PPL

• OpenMP

• Intel Threading Building Blocks (similar to PPL, by the way)

• MPI

Okay so maybe it’s not fair to bring all of these into the same category, but they all serve one purpose – speeding up code. And they do it pretty well. I’ve used OpenMP on a project, and this is basically ‘blind parallelization’, kind of like trusting TPL’s Parallel.For() to optimize your loop for multicore.

‘Modern’ C++

Today’s C++ is every bit as painful as the C++ of old. You still have to do the same menial things. In terms of what can seriously annoy a C#/Java developer, I can think of the following:

• Having to manage your own memory. Worse yet, there’s lots of ambiguous ways of using all those funny shared_ptr, unique_ptr etc. containers, and it’s nearly impossible to figure out even for experienced devs, let alone the uninitiated.

• The lack of proper string literals as well as C++‘s inability to just bite the bullet and create just a compiler flag for how to treat strings has led us to the insanity of char/wchar_t, various BSTR implementations, as well as the fact that I need to put the letter L in front of string literals.

• Truly insane compiler messages that Clang is supposed to be able to fix (err… I’ll beleive it when I see it).

• Really messed up mechanisms for declaring/initializing arrays, as well as the equally insane idea that an array is just a pointer to the first element. And don’t get me started on multi-dimensional arrays.

• The separation between header and implementation files results in huge amounts of wasted efforts. Personally, I try to avoid it by keeping everything in one single, huge .cpp file, but that’s not a good idea either.

On the other hand, there are reasons to use C++ nowadays. I’ve mentioned parallelization briefly, but there’s also C++ AMP, which is a way to transparently (ugh, more or less) execute C++ on a GPU. And of course, like it or not, interfacing with CUDA is a lot easier from C++ than .NET, though products are showing up all over the place that actually illustrate that it’s possible to just transcompiler. Still, if you go looking for jobs in the GPGPU space, you’re essentially looking at CUDA C which is, you guessed it, a C variant, so something closely related to C++.

Another advantage to C/C++ is that it’s a lot easier to integrate with .NET. I won’t talk about MC++ here, because this is a really ambigous product, but rather about P/Invoke – a technology that lets you transparently call C++ DLL functions right from C# (or indeed F#/VB.NET). Of course, you cannot use OOP structures from C#, but chances are you’re just optimizing a particular algorithm and you don’t need to import any hierarchies.

On the subject, there has been a few attempts to bring even OOP libraries to .NET. A good example is QuantLib which is a C++ library for quant finance (a topic very close to my heart). This library actually uses Boost for lifetime management, and of course it does use objects. Still, there are mechanisms which let you use it from .NET, though you must realize that you’re essentially getting a second-rate product.

Conclusion

There is no conclusion, since these are just my ruminations on a language that has largely gone out of relevance and now remains the niche of a few specialists. That’s not to say that it’s fundamentally bad, rather that it’s no longer something that warrants serious investment. In fact, I’d argue that knowing CUDA C is a more valuable skill than knowing C++, unless of course you’re job seeking in the quant space, where it’s still considered fresh.

About a year ago, while I was putting the polish on MathSharp (a MathML-to-code converter), I had yet another idea. I observed that for computational problems, despite the usefulness of MATLAB, Mathematica and other math applications, a great many people used Excel.

Excel really is convenient, and I also model things in it. But for executing code on ‘live’ server systems, rather than running an Excel workbook as a server (yes, it is possible), people typically prefer executable code.

And so with that in mind, I decided to put the two together and create an Excel add-in that would let me turn Excel spreadsheets (with formulae and all) into ready-to-compile C++ code. Thus, the X2C project was born.

High-Level Design

How can Excel and C++ constructs be related? My approach is to define mappings from cells to C++ code constructs. There are 5 different types of mapping:

• Scalar — a scalar maps a single cell to a variable. This means that a single A1 cell with the value 42 can be mapped to a global field defined as double A1 = 42. Of course, the name is customizable, and I left an option to not set the default value if it’s not needed.

• Function — this corresponds to a global function. For example, if cell A2 has a formula =A1+23, we generate a global function: double A2() { return A1+23; }

• Vector — this is a one-dimensional array of values. This means that a selection of A1:A3 can be mapped to a double Stuff[] array with 3 elements in it. If an array is subsequently used in a formula, we use the [] operator to extract the right element. Note that some advanced C++ is used here: for example if someone has a formula =SUM(A1:A3), we cannot create a temporary variable and a for loop to sum things up.

• Matrix — this is a two-dimensional array. It works just like a 1D array but its initialization is tricker and it’s tougher to understand the iteration code in case it’s used in a formula somewhere. Also, one can take 1D slices out of a 2D matrix, which turns things really nasty.

• Entity — this construct basically converts a selection of scalar/function definitions into a class! This lets you take a chunk of a spreadsheet and organize it in a way that its constituent parts now become part of a larger, named entity. Of course, lots of tricks are needed here too, because for example, a formula might use a cell mapped to an entity, so you need to add EntityName& to the parameters and dereference accordingly.

All of these mapping are created by an Excel add-in that lets you just click a button with the cell selected and edit the mapping. It’s really as simple as that, though mappings do have different code generation options for added customization.

Mathematical Issues

When dealing with math, there’s lots of issues caused by mismatches between real-world-math and computational math. Here are a few examples:

• Functions with known equivalents are easy: they are typically replicated by corresponding STL and Boost calls. In certain cases, though, it makes sense to create temporaries: for example, if a function uses NORM.DIST in many places, it makes sense to create the Boost normal distribution construct only once and then reuse it.

• Functions with no equivalents and easy implementation are typically done using best-available methods. For example, ODD and EVEN Excel functions require checking the last bit of an integer, so we can simply check e.g. (long int)foo & 1.

• Functions that are weird or just difficult to implement, like BAHTTEXT for example, are completely left out: not much we can do about those.

• Lots of matrix functions are meticulously implemented with cell-by-cell addressing. For example, a multiplication of two Excel matrices will be spelled out by creating real-life matrices from the data and then using Boost to perform the multiplication. The code for such a construct can look quite scary when generated, but the important thing is that it’s syntactically correct.

• Inefficient operations are rewritten when necessary. This includes things like repeated calculations. If you’re OK with having POWER(x,3) then that’s fine, but you can also get x * x * x or MyIntPow(x, 3) or even _IntPow(x,3) if you feel like it.

Things to Come

As it stands, X2C only supports C++, and it is capable if generating C++ files that you can preview and dump to disk. There’s rudimentary operations like removal and editing of mappings, and some customization functionality already in the box.

The following ideas might be worth pursuing (depending on demand for the program):

• .NET support is probably highest on the list. MathSharp already supports C# and F# (but not C++ — that might also happen at some point), so having at least F# in X2C would be nice. It would especially be nice to get continuous (re) compilation working for the generated code, so that the user can see immediately his model updating the code and the code being executed.

• Web service support is also another neat idea. Generating web server stubs isn’t exactly difficult, but would let users put up the calculations as a service and then share it accross the internet.

• UI generation is another very attractive feature, because once you get the model in a compilable state, the next thing you want to do is to is to start wiring up the UI so you can manipulate the data and observe the results. So why not generate the UI from the outset?

All in all, the development of X2C really depends on two things: my own personal needs as well as how much demand there is for such a tool. Meanwhile, if I got you interested, check out the product page (I promise to update the video as soon as I can) and let me know what you think.

In this post, I want to take a look at other ways of exploiting GPUs – specifically, how individual strings as well as string arrays can be processed on the GPU and what kind of performance benefit we would be able to get. For the purposes of my investigations, I will be using an ATI 6800 series GPU with an ordinary Core 4 Quad. I will use ordinary C++ and C++ AMP as the technologies being compared. I will use primary Latin characters (in a range that can be displayed in a console), but in Unicode (i.e., wchar_t-based) strings as befits a modern framework.

Please note that the examples use minimum optimization of C++ code, i.e., common STL algorithms and approaches are used and no attempt is made to improve or replace basic C++ constructs. This, I beleive, constitutes a fairer test than attempting to fine-tune C++ for peak performance.

Character histogram

We’ll begin with a simple case: given a rather lengthy text, how long would it take to build a histogram of all the characters present in the string? We’ll use a special function to randomly generate strings of different sizes:

// uses chars 32-127
void fill_string(wchar_t* chars, int count)
{
  for (int i = 0; i < count; ++i)
  {
    chars[i] = 32 + rand() % 95;
  }
}

We’ll perform the experiment on strings with length from 2 to 2²⁴ to ascertain how the algorithm performs under different conditions. First, we begin with a C++ implementation – without any constraints on the architecture, we’ll use a simple map-based approach:

typedef concurrent_unordered_map<wchar_t,int> histogram;
unique_ptr<histogram> cpu_histogram(wchar_t* str, size_t count)
{
  unique_ptr<histogram> result(new histogram);
  parallel_for((size_t)0, count, [&](int i)
  {
    (*result)[str[i]]++;
  });
      
  return result;
}

The GPU function is quite a bit more complicated. First of all, there is no real way of treating elements as wchar_t types since C++ AMP does not recognize elements smaller than an uint32_t. Thus, any view over a wchar_t array has to use the uint32_t data type.

In order to test the histogram, I used an existing implementation written by Daniel Moth. I won’t replicate the code here. All I changed in Daniel’s implementation is the definition of the data type, which I set to wchar_t, as well as an implementation of an alternative constructor that took actual data rather than the data size.

From then on, what I did is created strings of sizes 2ⁿ and filled them in with random printable characters:

int size = sizes[i];
wchar_t* target = new wchar_t[size+1];
ZeroMemory(target, (size+1) * sizeof(wchar_t));
fill_string(target, size);

Then, I simply timed the ‘cost’ of the CPU and GPU calls and averaged out results:

aggregate = 0.0;
for (int s = 0; s < 20; ++s)
{
  Timer gpuTimer;
  gpuTimer.Start();
  auto gh = gpu_histogram(target, size);
  gpuTimer.Stop();
  aggregate += gpuTimer.Elapsed();
}
wcout << (aggregate / 20.0) << endl;

Measuring Performance

For performance measurement I found yet another useful blog post describing the steps necessary to get the measurement just right. I also used a high-res timer from this article (isn’t the internet wonderful?).

The end result is the following performance measurements from the CPU and GPU. This is from a Release build with all optimizations enabled. Each iteration was repeated 20 times with the results averaged.

Figure 1. Comparison of GPU and CPU performance for building a string histogram. The X axis corresponds to a string of length 2^x, the Y axis represents the elapsed time in milliseconds, and is binary-log-scaled.

It’s clear that, up to some point, the CPU has an advantage, as the GPU is consistently paying a small but annoying start-up cost. As far as measuring the benefits, the parallel lines on the right hand side of the chart suggest a constant 8× performance improvement when using the GPU. Performance benefits of the GPU can only be appreciated when dealing with strings with length greater than 2¹⁶ (65536) characters.

Conclusion

This experiment is just a small performance investigation, probably full of various methodological errors and inaccuracies, but the above has certainly been benefitial in terms of figuring out that:

• The GPU kernel appears to have a fixed start-up cost associated with it. However, the start-up cost seems to be around 2ms, which is fairly insignificant, except maybe for things like high-frequency trading.

• The GPU appears to have a linear advantage over the CPU (about 3 orders of binary magnitude), which is surprising, because I would expect non-linear divergence in the results.

• The current approach seems to be entirely unfit for calculating histograms with unlimited character sets. The reason is that currently, a histogram array matches the size of all possible characters, and then there are equally many numbers of such arrays. Essentially, to make lots of arrays of size 2^{sizeof(wchar_t)} is impossible – the GPU just doesn’t have thas much memory.

I’ll certainly be playing more with C++ AMP and string processing specifically. Meanwhile, if you’d like to get the source code for this article, you can find it here. Who knows, maybe your performance measurements will be entirely different? Or maybe the algorithm won’t even run on your machine? At any rate, let me know. ■

Brief Overview of Agent Mulder

The principal idea behind Agent Mulder is to get ReSharper to support various IoC frameworks. To get a better appreciation for its features, take a look at the screencast or, better yet, download the plugin and try it out for yourself. But to sum things up, Agent Mulder augments ReSharper with intrinsic knowledge about the types registered in the IoC container and allows all sorts of wonderful things, such as e.g. the possibility to navigate to the point where a type is registered.

But seeing how our SDK adventures concern actual implementations of the various features, here’s what we’re going to talk about today:

• Programmatic use of Structural Search and Replace (SSR)

• Use of gutter marks

• Suppression of existing inspections

All of these are fairly advanced topics, but we’ll tread carefully and hopefully things will make sense.

Programmatic SSR

Structured Search and Replace is a ReSharper feature that helps people locate code based on some pattern. For example, you can go off looking for $1.Foo($2) where $1 and $2 have specific types. The Replace part of the SSR lets you then replace the found pattern with something else by presenting a context action.

So how does this work in Agent Mulder? Well, AM needs to locate places where a particular component has been registered. To do that, it creates and uses search patterns – definitions of type IStructuralSearchPattern which are object-orientated equivalents of what one would otherwise define in the SSR GUI. For example, here’s the pattern for a Castle Windsor container registration:

private static readonly IStructuralSearchPattern pattern =
  new CSharpStructuralSearchPattern("$container$.Register($arguments$)",
    new ExpressionPlaceholder("container", "Castle.Windsor.IWindsorContainer", false),
    new ArgumentPlaceholder("arguments", -1, -1)); // any number of arguments

The above is a simple search for $container$.Register($arguments$) where the first parameter has a type of IWindsorContainer and the second argument is actually variadic (i.e., there might be several arguments).

Now that we’ve got the pattern to search with, how do we perform the search? Well, to match an expression to a pattern we need a matcher embodies by the IStructuralMatcher interface. This matcher is effectively created from the defined pattern, for example:

var matcher = pattern.CreateMatcher();

With the matcher in hand, any particular ITreeNode can then be matched against the pattern using a QuickMatch() method. This method takes the tree node as a parameter and returned a bool indicating whether there is, in fact, a match. Keep in mind that a matcher works on a particular in vocation, not on the entire tree. This is why Agent Mulder also tries to get the tree node as an IInvocationExpression and, if successful, performs a match against all its subexpressions.

Gutter Marks

Gutter marks are little glyphs on the left-hand side of the editing pane. They typically show various useful bits of information, such as indicating that a type inherits from another type. Gutter marks are also clickable. In the case of Agent Mulder, a gutter mark is used to indicate that a type is registered in a container.

So what’s a gutter mark in terms of the API? Put simply, it is a class that inherits from IconGutterMark and is registered with an assembly-level RegisterHighlighter attribute. There are a few things the gutter mark class has to do. First of all, it has to call the base class constructor and provide an image that it is going to display:

public ContainerGutterMark()
  : base(ImageLoader.GetImage("Hat", Assembly.GetExecutingAssembly()))
{
}

The above loads the Hat.png image file from the current assembly. In order for the above invocation to work, an additional assembly-level attribute is required in AssemblyInfo.cs:

[assembly: ImagesBase("AgentMulder.ReSharper.Plugin.Resources")]

Now, once the image is loaded, there is also the IsClickable property of the gutter mark class to implement. Its function is predictable: if set to true, the OnClick() function must be defined.

public override bool IsClickable
{
  get { return true; }
}

Now, before we get to clickability itself, we need to discuss the way gutter marks are actually added to a file because, after all, their invocation (the OnClick() override) depends entirely on their position relative to code.

The answer to this question is rather simple: gutter mark positions are actually controlled by highlightings – yes, the same highlightings that are used to indicate warnings, errors, and so on. This exact mechanism allows the clickable gutter mark to actually get information about its location. Here’s how it works: the gutter mark is registered at the assembly level with an attribute similar to the following:

[assembly: RegisterHighlighter("Container Registration", 
  "{B57372C1-16C3-4CB5-8B68-A0FBEFB487AD}", 
  EffectType = EffectType.GUTTER_MARK, GutterMarkType = typeof(ContainerGutterMark), 
  Layer = 2001)]

The critical parameter in the above registration is the Id, and in this case it has a value of "Container Registration". Now, we can go ahead and create a simple highlighting with a severity of INFO. The critical piece of the puzzle is that the AttributeId parameter of the highlighting has to match the Id from above:

[StaticSeverityHighlighting(Severity.INFO, "GutterMarks", 
  OverlapResolve = OverlapResolveKind.NONE, AttributeId = "Container Registration", 
  ShowToolTipInStatusBar = false)]
public sealed class RegisteredByContainerHighlighting : IClickableGutterHighlighting
{
  // implementation here
}

But this doesn’t answer the most difficult of questions: how does the gutter mark know where to go? It knows because the gutter mark’s OnClick() override gets an IHighlighter as a parameter. This means that if we get a solution manager and this highlighter, we can get at the actual highlighting and invoke its own Click() method:

public override void OnClick(IHighlighter highlighter)
{
  ISolution currentSolution = Shell.Instance.GetComponent<ISolutionManager>().CurrentSolution;
  if (currentSolution == null)
  {
    return;
  }
  var clickable = Daemon.GetInstance(currentSolution).GetHighlighting(highlighter)
                  as IClickableGutterHighlighting;
  if (clickable != null)
  {
    clickable.OnClick();
  }
}

Put all of the above together and you get a clickable gutter mark.

Suppression of Inspections

If you turn on Solution-Wide Analysis (SWA), you’ll get container-registered classes as unused, which isn’t good because they are used, albeit in a container. Sure, you could just mark these as [UsedImplicitly], but this is extra work for the developer, so Agent Mulder’s idea is to handle this automatically.

How does it work? Well, one of the daemon processes in ReSharper’s ecosystem is the CollectUsagesStageProcess. This process handles usage information, and conveniently enough, this component has a SetElementState() method that can set the ‘usage state’ of a particular element (in the case of Agent Mulder, an IConstructor):

private void SetConstructorsState(ITypeElement typeElement, UsageState state)
{
  foreach (IConstructor constructor in typeElement.Constructors)
  {
    collectUsagesStageProcess.SetElementState(constructor, state);
  }
}

It really is that simple. Agent Mulder actually sets the usages in its daemon stage process as it goes through the files. This means that, for classes that are registered in the container, constructors are marked as used which prevents SWA from claiming that the class is unused.

Conclusion

I’ve gone over most of the core aspects of Agent Mulder, leaving out perhaps just one – the search & navigation aspect. This really warrants its own entry however, so stay tuned for more R# SDK adventures. Oh, and check out Agent Mulder if you haven’t done so already!

What are actions?

Actions are, effectively, commands that you can invoke and your plugin can respond to. To invoke an action, you can use any number of menus as well as keyboard shortcuts. Actions are actually published as commands in Visual Studio, so binding them to a particular key is no problem.

For example, let’s say we’re working with Excel and we want to cut-and-paste data from Excel into your C# file, naturally turning the data into some easily digestible form, like e.g., an array. What we can do is define an action to handle it. An action has to implement the IActionHandler interface and be decorated with the ActionHandler attribute whose parameter takes the action’s identifier.

[ActionHandler("PasteCSV")]
public class PasteCSV : IActionHandler
{
  // todo
}

Now, the IActionHandler interface has two methods that you need to implement.

First, there is the Update() method, which determines whether our action gets executed at all. This is a great place to check, in our case, if we’ve got the right data on the clipboard. Interestingly, when you copy a chunk of an Excel worksheet, data gets placed on the clipboard in multiple formats simultaneously, including a plain-text format (separated by tabs) and a CSV format (separated by commas). We’ll go for the CSV format here. Also, we need to check that there is an editor to work with, because if there isn’t, pasting is meaningless. Here’s the implementation we end up with:

public bool Update(IDataContext context, ActionPresentation presentation, DelegateUpdate nextUpdate)
{
  return Clipboard.ContainsText(TextDataFormat.CommaSeparatedValue) &&
    context.GetData(JetBrains.TextControl.DataContext.DataConstants.TEXT_CONTROL) != null;
}

The second method is the Execute() method where, as you may have guessed, execution of the action actually happens. This is where the bulk of our algorithm will reside.

The Algorithm

Now, we know text is in there, so let’s think a little about the algorithm that we might want to implement when pasting things. Data can be coming in as a single row, column, or a table of multiple rows and columns. It can also be homogeneous (e.g., all numbers) or heterogeneous, mixing text and numbers.

Without overcomplicating things, I’d argue that:

• A single row or column of numeric data can be declared as an array of fixed size.

• A table of purely numeric data can be treated as a rectangular array.

• Any ‘mixed’ data needs to be declared as an array of Tuples. The supported data types are double, DateTime or string, depending what parses.

We therefore start by getting the clipboard data and breaking it into lines:

var csv = Clipboard.GetText(TextDataFormat.CommaSeparatedValue);
var lines = csv.Split(new[]{Environment.NewLine}, StringSplitOptions.RemoveEmptyEntries);

We can now write a simple method checking that all data is numeric:

public bool IsAllDataNumeric(string[] lines)
{
  double dummy;
  return lines.SelectMany(line => line.Split(','))
    .All(element => double.TryParse(element, out dummy));
}

Let’s consider the numeric case first. If either the number of rows or columns is equal to zero, we get a 1-dimensional array; otherwise, we get a rectangular array:

if (rows == 1 || cols == 1)
{
  sb.AppendFormat("double[] foo = {{ {0} }};",
                  rows == 1 ? lines[0] : lines.Join(","));
}
else
{
  sb.Append("double[,] foo = {").AppendLine();
  foreach (var line in lines)
    sb.AppendFormat("{{ {0} }},", line).AppendLine();
  sb.Append("};");
}

Well, that was the easy part, now the tough part. How can we guess the type of a non-numeric data item? Well, we can try parsing it for whatever data structure we want. And once we’ve got the type, we can format things accordingly:

public string FormatForType(string input)
{
  DateTime dt;
  double d;
  if (DateTime.TryParse(input, out dt))
    return dt.ToAssemblyCode();
  else if (double.TryParse(input, out d))
    return input;
  else return input.Quoted();
}

In the above, the ToAssemblyCode() extension method creates a new DateTime(...) declaration corresponding to the actual DateTime object. Quoted() simply puts double quotes around a string.

Now, if we are making an array, we can still keep implicit typing just so long as data is, roughly the same. This can cause a few hiccups: for example, is 123 a double? According to type conversion rules, it may as well be. But according to type inferencing rules, it’s not, and Tuple.Create(123) is actually a Tuple<int> even if you’ve got a large array where some elements are of type double. You need to be explicit about this.

Anyways, the implementation of Tuple-based algorithm for both vectors and matrices is as follows:

if (rows == 1 || cols == 1)
{
  var data = rows == 1 ? lines[0].Split(',') : lines.ToArray();
  sb.AppendFormat("var foo = Tuple.Create({0});", data.Select(FormatForType).Join(","));
} else
{
  sb.Append("var foo = new[] {").AppendLine();
  foreach (var line in lines)
    sb.Append("Tuple.Create(").Append(line.Split(',').Select(FormatForType).Join(",")).Append("),");
  sb.Append("};");
}

And finally, we can put insert the generated text into the current document at the caret position:

var textControl = context.GetData(JetBrains.TextControl.DataContext.DataConstants.TEXT_CONTROL);
var doc = textControl.Document;
var pos = textControl.Caret.Position;
doc.InsertText(pos.Value.ToDocOffset(), sb.ToString());

And that’s all there really is to it.

Using Actions

There are two fairly obvious ways to use actions.

The first is to simply bind it to a shortcut. To do this, you go to Tools | Options and choose the Environment → Keyboard section:

The other option is to have the action display its own menu item, either in the top menu bar or in any number of context menus for, e.g., the solution, the project, etc. In order to implement this, you need to do three things:

• Create in your project a file called Actions.xml and set its Build Action to ”Embedded Resource“

• Edit the file, specifying the action and the menu you want it to appear under. For example, to have our action in the top-level menu under ReSharper | Foo, you would specify the following in the XML file:

  <actions>
    <insert group-id="ReSharper" position="last">
      <action-group id="Foo" text="Foo" shared="true">
        <action id="PasteCSV" text="Paste CSV"/>
      </action-group>
    </insert>
  </actions>
  

  A more comprehensive example of places where the menu item can be added is available in the SDK.

• Finally, you need to specify the location of the Actions.xml file in AssemblyInfo.cs, i.e., add a line similar to the following:

  [assembly: ActionsXml("Bar.Baz.Actions.xml")]
  

  In the above, Bar.Baz refers to the name of the assembly you’re working with. (We are referencing an embedded resource, after all.)

Conclusion

Once again, I’ve presented an example that would require a lot more rigor if it were to be deemed production-ready code. Actions are simple, but there are lots of useful things you can do with them. And in case you’re interested, the source code for this action can be found here.

Checking that caller is indeed System.Math

So far, we only checked that the function being called is Pow by getting the name of the reference:

bool functionIsCalledPow = false;
var e = element.InvokedExpression as IReferenceExpression;
if (e != null)
{
  if (e.Reference.GetName().Equals("Pow"))
    functionIsCalledPow = true;
}

This means that we can easily get a false positive with something like this:

double y = Foo.Pow(x, 2.0);

Now, to handle this situation properly, we’re going to scrap our previous check of the name and replace it with the following:

bool isOnMathPow = false;
var r = element.InvocationExpressionReference.Resolve();
var m = r.DeclaredElement as IMethod;
if (m != null)
{
  var parent = m.GetContainingType();
  if (parent != null)
  {
    isOnMathPow = parent.GetClrName().FullName.Equals("System.Math")
                  && m.ShortName.Equals("Pow");
  }
}

There’s quite a lot that’s happening in the above, so let’s go through it step-by-step:

• The first thing you’ll notice is that we now use the InvocationExpressionReference and Resolve() it. This typically yields us a ‘resolve result’, but can also fail in case the method doesn’t resolve to anything.

• To figure out if the resolution happened correctly, we simply take the result’s DeclaredElement and cast it to an IMethod, since Math.Pow() is a static method call.

• If the result is OK, we also get the type that contains the method. If this is the right call, this should point us to the System.Math class.

• To check that we do in fact have System.Math, we get the parent’s CLR name, and from that its FullName, which has the full namespace prefix.

• To check we have the right method, we simply check its name as before.

The above manipulations, though complex, rid us from false positives like that Foo.Pow() call.

Custom function handling

Changing Math.Pow(x, 2.0) to x*x makes sense. So does changing Math.Pow(y, 3) to y*y*y. However, after this threshold, things get a bit ridiculous: you really don’t want z*z*z*z*z*z in your code, particularly since there’s no way to read from this that this is z^6. Wouldn’t it be nice if the user could specify their custom function that we ought to use for powers, say, greater than 3?

Let’s do this. First of all, we’ll define a settings class that will house the user’s preferences, specifically whether they want to have a custom power function, and what its name is:

[SettingsKey(typeof(Missing), "General Settings")]
public class GeneralSettings
{
  [SettingsEntry(false, "Use Custom Power Function")]
  public bool UseCustomPowerFunction { get; set; }
 
  [SettingsEntry("", "Custom Power Function Name")]
  public string CustomPowerFunctionName { get; set; }
}

We can now create a settings page (a user control that implements IOptionsPage) and bind the properties as follows:

settings.SetBinding(this.lifetime, (GeneralSettings gs) => gs.UseCustomPowerFunction,
  WinFormsProperty.Create(this.lifetime, chUseCustomFunction, x => x.Checked, true));
settings.SetBinding(this.lifetime, (GeneralSettings gs) => gs.CustomPowerFunctionName,
  WinFormsProperty.Create(this.lifetime, tbInliningFunctionName, x => x.Text, true));

Now, we need to change our inlining fix. This is tricky. First of all, let’s define a method that would actually get the two values given a context function:

public Pair<bool, string> GetCustomFunctionSettings(Func<Lifetime, DataContexts, IDataContext> ctx)
{
  var ss = Shell.Instance.GetComponent<ISettingsStore>();
  var bs = ss.BindToContextTransient(ContextRange.Smart(ctx));
  var s = bs.GetKey<GeneralSettings>(SettingsOptimization.DoMeSlowly);
  return new Pair<bool, string>(s.UseCustomPowerFunction, s.CustomPowerFunctionName);
}

The above simply gets a settings store, binds it to a transient context that’s based on the function we provide (more on that in a sec), reads the settings and returns them in a Pair<>.

The function above may be a bit tricky, but it’s required to create a context. Typically, though, you can get the function directly from whatever code element you’re operating on. In our case, the highlighting passed into a quick-fix has an Expression, so we can simply use that:

public IntPowerInliningFix(IntPowerHighlighting highlighting)
{
  this.highlighting = highlighting;
  customNameSettings = GetCustomFunctionSettings(highlighting.Expression.ToDataContext());
}

Now that we’ve got the settings, all that remains is to use them! The approach I’m going to take here is to use the custom function, if available, for powers greater than 3:

protected override Action<ITextControl> ExecutePsiTransaction(ISolution solution, IProgressIndicator progress)
{
  var expr = highlighting.Expression;
  var arg = expr.Arguments[0];
  var factory = CSharpElementFactory.GetInstance(expr.GetPsiModule());
  ICSharpExpression replacement;
  if (customNameSettings.First && highlighting.Power > 3)
  {
    var template = "$0($1, " + highlighting.Power + ")";
    replacement = factory.CreateExpression(template, 
      customNameSettings.Second, expr.Arguments[0]);
  }
  else
  {
    replacement = factory.CreateExpression(
      Enumerable.Range(0, highlighting.Power).Select(i => "$0").Join("*"), arg.Value);
  }
  ModificationUtil.ReplaceChild(expr, replacement);
      
  return null;
}

Now, a user can specify a replacement function Maths.IntPow and have their Math.Pow(x, 4.0) call refactored to Maths.IntPow(x, 4). Note that it’s 4, not 4.0 — we assume that the target function takes an int as the second parameter. Note also that in order to get 4 to appear, we cannot use the $ notation – instead, we prepare the template by concatenating strings, putting the integer there manually.

Code Cleanup

Changing the Math.Pow() invocations one at a time is too slow if you’ve got hundreds of such instances. The mechanism to perform changes en masse in ReSharper is called Code Cleanup and that’s the mechanism we’re going to use to perform changes on all int-bearing instances of Math.Pow() that are found, be it in a file, project or whole solution.

So let’s start with the basics: what we need is a code cleanup module, a class that implements the ICodeCleanupModule interface and is decorated with [CodeCleanupModule]. Unfortunately, ICodeCleanupModule has six methods that we must implement. Some of these are similar to IQuickFix methods but actually serve different purposes. There’s also the thorny issue of descriptors, so let’s start with those.

A descriptor is basically an option of code cleanup. For example, the option of whether to change declarations to var is encapsulated within a descriptor. We need at least one descriptor to define whether we want to do the Math.Pow() replacement at all, so here goes:

[DefaultValue(false)]
[DisplayName("Replace Math.Pow() integer calls")]
[Category(CSharpCategory)]
private class Descriptor : CodeCleanupBoolOptionDescriptor
{
  public Descriptor() : base("ReplaceMathPowIntegerCalls") {}
}

All this does is provide an option to turn the feature on or off in various code cleanup profiles. Now, this feature can be instantiated and returned from the code cleanup module:

private static readonly Descriptor descriptor = new Descriptor();
public ICollection<CodeCleanupOptionDescriptor> Descriptors
{
  get { return new[] {descriptor}; }
}

Now, moving on, let’s implement the IsAvailable() method. This method takes an IPsiSourceFile and we’re supposed to determine whether it can be used. Well, our only restriction right now is that this has to be a C# file, so…

public bool IsAvailable(IPsiSourceFile sourceFile)
{
  return sourceFile.GetPsiFile<CSharpLanguage>() != null;
}

One small thing to note is that the above causes the action to also fire in injected PSI, i.e., in C# that’s part of MVC or Razor views. If you want to avoid this, we could use GetNonInjectedPsiFile() instead.

Next, let’s go after SetDefaultSetting(), which determines the default value of this option depending on the code cleanup profile that’s being used. Let’s assume that, by default,

• Our feature is on in the “Full Cleanup” profile.

• Our feature is off in the “Reformat” profile.

Thus, the following implementation of SetDefaultSetting() communicates our preferences:

public void SetDefaultSetting(CodeCleanupProfile profile, CodeCleanup.DefaultProfileType profileType)
{
  switch (profileType)
  {
    case CodeCleanup.DefaultProfileType.FULL:
      profile.SetSetting(descriptor, true);
      break;
    default:
      profile.SetSetting(descriptor, false);
      break;
  }
}

Now, there’s also the IsAvailableeOnSelection property, which is pretty self-descriptive. You can return true or false here based on applicability; it doesn’t matter much for our experiments.

Finally, we come to the meat of the problem: the Process() method where changes actually happen. But, annoyingly enough, this method needs to reuse our analyzer and quick-fix code. We don’t really want to run it all again, do we? As a result, we perform the following refactorings:

• In IntPowerProblemAnalyzer, we isolate the check on the right method into a static method called InvocationExpressionIsMathPowCall. This function now returns two values – a bool indicating whether this is the right function, and an int indicating the power that is being used.

• In IntPowerInliningFix, we ensure the function to acquire settings is static; we then isolate the code that performs the change into a separate (again, static) method.

We can now attempt to perform the code cleanup itself. This is no mean feat, considering that previously, in our quick-fix, a lot of the plumbing was handled by the BulbItemImpl class that we inherited from. Let’s do a few checks, first. Let’s check that we’ve got the right file and that we are allowed to do the change:

var file = sourceFile.GetPsiFile<CSharpLanguage>();
if (file == null) 
  return;
 
if (!profile.GetSetting(descriptor)) 
  return;

If these checks pass, there’s some legitimacy to doing the change. We may as well get the user settings for the quick-fix right now, because doing them every time we meet an invocation expression is a bad idea:

var settings = IntPowerInliningFix.GetCustomFunctionSettings(sourceFile.ToDataContext());

Now, regrettably, we must manually set up a transaction using the PsiManager. We’re going to also be using shell locks, specifically a write lock that is required to perform changes in a file. The IShellLocks variable can be injected into the constructor of our module:

var settings = IntPowerInliningFix.GetCustomFunctionSettings(sourceFile.ToDataContext());
file.GetPsiServices().PsiManager.DoTransaction(() =>
{
  using (shellLocks.UsingWriteLock())
  {
    // scary stuff here
  }
}, "Code cleanup");

So, what exactly is happening inside the above? Well, we have to go through all IInvocationExpression’s in the file recursively and, for each one that happens to be a correct Math.Pow() call, perform the replacement. This is where our refactorings come into play, because without them we’d be duplicating code or creating meaningless (and quite possibly broken) instances of our analyzer or quick-fix.

Here’s what the ‘scary stuff’ looks like:

var itemsToChange = new List<Pair<IInvocationExpression,int>>();
file.ProcessChildren<IInvocationExpression>(e =>
{
  int power;
  if (IntPowerProblemAnalyzer.InvocationExpressionIsMathPowCall(e, out power))
    itemsToChange.Add(new Pair<IInvocationExpression, int>(e, power));
});
foreach (var e in itemsToChange)
  IntPowerInliningFix.PerformChange(e.First, settings, e.Second);

Notice how we have to cache the items before they are processed — this is necessary because modifying these items as they are being iterated is not safe, just as it’s not safe to modify the loop iterator within the loop. Thus, we use a List<> to cache the items and their respective powers, and then reprocess them wholesale once the iteration has been completed.

Conclusion

In this post I’ve demonstrated how to dig into the invocation’s type name (no easy feat), how to define settings and use them. I’ve also demonstrated how to write a code cleanup module to perform large-scale changes on the code base. You can find the source code here.

Problem

Why does the plugin ecosystem exist in the first place? Why make the SDK? The motivation is, of course, the fact that individual features that you might need for your particular case aren’t available out of the box. For example, you are aware of a particular case where code is bad and you want to somehow correct this. (SSR helps in simple cases, but the one we’ll consider is a bit more complex.)

For example, the following is, arguably, bad code:

var y = Math.Pow(x, 2.0);

The performance overhead on calculating x² this way is massive. The reason is that Math.Pow() is tuned towards floating-point powers, which means that up to some extent, it is a lot more efficient to do one of the following:

• Write the expression inline, i.e., Math.Pow(x, 2.0) → x*x.

• Substitute an expression with your own implementation where the power is an integer, i.e., Math.Pow(x, 2.0) → Maths.Pow(x, 2), where Maths is your own utility class and Pow is a function that has an integer overload.

Now, the above is a problem, and I’d like this problem to be taken care of by using ReSharper’s extensibility mechanisms.

Identifying the Structure

If we are to somehow fix the above, we first need to be able to identify the ‘offending’ structure. We are going to assume the simplest possible case, Math.Pow(x, y) where x is an identifier and y is a numeric value that happens to be an integer.

We should immediately be able to realize the constraints that we are putting on this model. For example, we assume that x is an identifier and not, for example, a numeric value. In fact, if it was, we could precompute the whole thing. But what if it was a constant, something like Math.Pow(Math.PI, 2)? Clearly, in this particular case, we wouldn’t care whether the power was an integer or not, because in any case, the replacement would be a computed constant value.

So, let’s look at a typical call, Math.Pow(x, 2.0). Here’s how to get info about it:

1. First, fire up Visual Studio with the /ReSharper.Internal switch to get all the internal debug menus.

2. Make a bare-bones file that contains the Math.Pow() statement in it.

3. In the top-level menu, choose ReSharper | Internal | Go to PSI Viewer.

What you should end up with is something like this:

The PSI viewer is a really neat tool for figuring out what the code you’re after actually is. For instance, the above basically shows us that Math.Pow(x, 2.0) is

• An IInvocationExpression containing

• A reference expression referring to Math.Pow

• A list of arguments containing

• A reference expression with an identifier x

• A literal expression with a floating-point value of 2.0

And that’s just about all we need to know – for now. We can now create a simple problem analyzer to look for this particular scenario.

Problem Analyzer

A problem analyzer is a piece of code that ReSharper can use to find out if there’s a problem somewhere. Since we agree to consider integer-based Math.Pow expressions to be a performance problem, we are going to create an element problem analyzer that will help us identify it.

First of all, let us define a class called IntPowerProblemAnalyzer. This class is looking explicitly for IInvocationExpressions, thus it will be made to inherit from ElementProblemanalyzer<IInvocationExpression>. There’s also some metadata at the top that we’ll discuss in a moment.

[ElementProblemAnalyzer(new[]{typeof(IInvocationExpression)}, 
  HighlightingTypes=new[]{typeof(IntPowerHighlighting)})]
public class IntPowerProblemAnalyzer : ElementProblemAnalyzer<IInvocationExpression>
{
  protected override void Run(IInvocationExpression element, 
    ElementProblemAnalyzerData data, IHighlightingConsumer consumer)
  {
    // todo :)
  }
}

Okay, so the class we have right now has to implement a single method called Run(). This method is important because this is where we teach the plugin to identify and highlight code that conforms to our pattern. But how to actually identify the pattern? Using the PSI viewer above, we can deduce that a Math.Pow() call check would consist of two parts: checking that we are calling a Pow() function, and checking that the arguments are correct. The first condition is checked as follows:

bool functionIsCalledPow = false;
var e = element.InvokedExpression as IReferenceExpression;
if (e != null)
{
  if (e.Reference.GetName().Equals("Pow"))
    functionIsCalledPow = true;
}

Note that we’re not testing it was Math that this was called on – this is more complicated and a bit out-of-scope right now. The second part of the check is a little more convoluted:

bool firstArgIsIdentifier = false;
bool secondArgIsInteger = false;
if (element.Arguments.Count == 2)
{
  firstArgIsIdentifier = element.Arguments[0].Value is IReferenceExpression;
  var secondArg = element.Arguments[1].Value as ICSharpLiteralExpression;
  if (secondArg != null && (secondArg.IsConstantValue()))
  {
    double value = -1.0;
    var cv = secondArg.ConstantValue;
    if (cv.IsDouble())
      value = (double)cv.Value;
    else if (cv.IsInteger())
      value = (int) cv.Value;
    secondArgIsInteger = (value > 0.0) && value == Math.Floor(value);
  }
}

The above does a very basic, almost simplistic check on the structure of the code in question. Note that I’ve greatly simplified the identification of the reference, but it will do for now. The next thing that we have to discuss is how problematic code is highlighted.

Highlighting the Problem

We’ve got some code for finding the problem, but how to show it to the user? In the snippet above, we used the highlighting consumer and passed it an instance of a highlighting. So what is it?

A highlighting is simply a class that implements that IHighlighting interface and is decorated by appropriate attributes. All it does is define how a particular problem is highlighted by ReSharper. This highlighting is passed to the consumer and is also featured as part of the HighlightingTypes attribute parameter of the problem analyzer. We are currently using the following definition of IntPowerHighlighting:

[StaticSeverityHighlighting(Severity.WARNING, CSharpLanguage.Name)]
public class IntPowerHighlighting : IHighlighting
{
  private readonly IInvocationExpression expression;
  public int Power { get; private set; }
  public IntPowerHighlighting(IInvocationExpression expression, int power)
  {
    this.expression = expression;
    Power = power;
  }
  public bool IsValid()
  {
    return expression != null && expression.IsValid();
  }
  public string ToolTip { get { return "Inefficient use of integer-based power"; } }
  public string ErrorStripeToolTip { get { return ToolTip; } }
  public int NavigationOffsetPatch { get { return 0; } }
}

Together with the element problem analyzer, we can fire up our plugin and get a result similar to the following:

And that’s what we wanted in the first place, but just seeing the problem isn’t enough. How about letting the end user correct the problem?

Quick-Fix

A quick-fix is a fix associated with a highlighting. In our case, we want to let a user replace Math.Pow(x,2) with x*x, with the number of x’s corresponding to the power. Note that it is entirely unreasonable beyond a certain value, typically around 2-3, so our fix will only work for these values.

In order to implement an inlining fix, we need to create an additional class. This class will implement the IQuickFix interface and decorate it with a [QuickFix] attribute as follows:

[QuickFix]
public class IntPowerInliningFix : BulbItemImpl, IQuickFix
{
  private readonly IntPowerHighlighting highlighting;
  public IntPowerInliningFix(IntPowerHighlighting highlighting)
  {
    this.highlighting = highlighting;
  }
  protected override Action<ITextControl> ExecutePsiTransaction(ISolution solution, IProgressIndicator progress)
  {
    // todo
  }
  public override string Text
  {
    get { return "Inline integer power"; }
  }
  public bool IsAvailable(IUserDataHolder cache)
  {
    int power = highlighting.Power;
    return power == 2 || power == 3;
  }
}

Note how the fix is only available for powers 2 and 3. The ExecutePsiTransaction() method is where all interesting things occur. Specifically, this is where we actually replace the Math.Pow() call with an inlined multiplication. Since this is somewhat more complicated, let’s go through this step-by-step. First of all, we need to get our argument – the one we’re going to multiply a few times:

var arg = highlighting.Expression.Arguments[0];

Now we need to create an element factory to construct the replacement expression. We use the original expression to get at the PSI module:

var factory = CSharpElementFactory.GetInstance(highlighting.Expression.GetPsiModule());

Now, through a bit of LINQ magic, we create an expression that represents the multiplication. Essentially, we create lots of $0’s separated by multiplication signs and let CSharpElementFactory fill them in:

var replacement = factory.CreateExpression(
  Enumerable.Range(0, highlighting.Power).Select(i => "$0").Join("*"), arg.Value);

Finally, we replace the old expression with the new one:

ModificationUtil.ReplaceChild(highlighting.Expression, replacement);

And here is what it looks like:

Conclusion

This demonstration is, admittedly, very simple. I’ve taken a number of methodological shortcuts that would never be acceptable in production code, namely:

• Only checked that the function is called Pow() without verifying that it’s part of System.Math

• Only did a basic check against the first argument

• Assumed that the power is either a double or an int, whereas in reality it could be e.g., a short

• Didn’t write any tests

In the next part of the series we’ll continue with this theme and take a look at how the plugin can be enhanced and its operations can be more rigorously checked. Meanwhile, check out the source code. For more information on the ReSharper API, check out the ReSharper Plugin Development Guide. Happy coding!

I beg to differ. In fact, minus the fancy manipulation widgets, the demonstrated approach shown is already available to anyone wishing to have it.

The solution is to take a language that is either REPL-pliable or very quick to compile and then re-run the compiler/interpreter on every symbol change in the editor. For example, in F# we can wire up FSI.EXE to re-run our script on any change. This is feasible performance-wise, and, coupled with a little code to intercept the standard output/error streams, you get a complete solution for rapidly prototyping any piece of code without having to explicitly compile something, manage the project, etc.

Use Cases

One place where the realtime approach is useful is numerics. Now we know that tools such as Mathematica, MATLAB etc. all basically do REPL, but that’s not nearly good enough. But what you can do is simply feed a whole file to an application to process wholesale. Then, simply observe any result you get.

The great thing about this is error handling during compilation. Making a mistake and then not getting the right result is very costly when compilation takes a long time and the availability of static analysis tools is poor. As a result, this ‘realtime’ approach really does save a lot of time and energy that would have otherwise been spent recompiling and cursing at yourself/the compiler/the universe.

And by the way, languages that support REPL are a lot better at outputting objects as text. For example, F# lets you write printf "%A" xyz where xyz can be a list, sequence, whatever. And it does output nicely formatted lists in a manner that, say, MATLAB easily can.

Enhancements

Let’s get back to F#/FSI for a moment. What’s interesting about feeding the ‘intermediate code state’ into the interpreter (which is the same as the compiler, by the way) is that we can augment the code passed in by putting in additional diagnostic statements. Not in the code, but at the postprocessing stage.

For example, we can take all method parameters and output them to the console. Or we can find every let binding and output the object that’s being kept. And then, all of a sudden, all that ‘Inventing on Principle’ magic with illustrating changes in the intermediate variables suddenly becomes possible.

Conclusion

The applicability of this technique largely depends on the language. However, in many languages, it’s totally reasonable to adapt this approach. In particular, compiled languages that have a scripting counterpart (e.g., D) can be more amenable to this approach, whereas slow-to-compile languages such as C++ can benefit from substitution, the idea being that you prototype your code using a different language entirely. (Of course, extra effort will be required to make the conversion.)

Moral of the story: you already have capabilities for realtime programming, you just aren’t using them.

Examples

Let’s start with the simplest example. Say you want to add an item to the collection: this is typically written as

myCollection.Add(x)

If you pause for a moment and think about the above statement and voice it in English, what you’ll get is myCollection should add to itself item x which is very much unreadable. This is precisely the case where IoC is useful, and here’s how: let us define an AddTo() extension method:

public static T AddTo<T>(this T self, ICollection<T> collection)
{
  collection.Add(self);
  return self;
}

Now, instead of calling the Add() method on the collection, control is inverted, so the AddTo() method is called on the collection:

var aList = new List<int>();
2.AddTo(aList);

Translated to English, the above now reads 2 should be added to aList. Makes more sense, doesn’t it? Also, as an added bonus, the AddTo() operation is fluent because it returns the original argument. This means that a value can be added to more than one list at once, e.g.:

2.AddTo(someList).AddTo(someOtherList);

Let’s try something else. Suppose you want to check that a collection is empty. Typically, you’d write

if (myClass.Fields.Count == 0) { ... }
// or
if (!myClass.Fields.Any()) { ... }

Both of these approaches are okay, I guess, but the code reads strange. The first example says if myClass’s fields count is equal to zero whereas the second reads if not myClass’s fields are any in number, which is even worse.

So here’s a pair of extension methods:

public static bool HasSome<TSubject, T>(this TSubject subject, 
  Func<TSubject, IEnumerable<T>> propertyToCheck)
{
  return propertyToCheck(subject).Any();
}
public static bool HasNo<TSubject, T>(this TSubject subject, 
  Func<TSubject, IEnumerable<T>> propertyToCheck)
{
  return !HasSome(subject, propertyToCheck);
}

Now, a check for the presence of fields turns into

if (myClass.HasNo(c => c.Fields)) {}

which reads as myClass has no fields, which is precisely what we’re checking here.

Okay, one more example. Say you’re checking a string value, and you want to compare it with a set of values. This is typically represented as

if (myOp == "AND" || myOp == "OR" || myOp == "XOR") { ... }

This reads in a kind of OK, but fragmented, fashion. A much better way would be to group the variants into an array but, in C#, this just looks ugly:

if (new[]{"AND", "OR", "XOR"}.Contains(myOp)) { ... }

This is even less semantic. What if there was an IsOneOf() function instead?

public static bool IsOneOf<T>(this T self, params T[] variants)
{
  return variants.Contains(self);
}

Now, the above check would turn to

if (myOp.IsOneOf("AND", "OR", "XOR")) { ... }

The great thing about the above approach is that the arguments are declared as params T[], so we don’t have to initialize any arrays with new[] before using them as a test.

Summary

So here’s the basic premise of LIoC: given a function f(x,y...), it may in certain cases be more benefitial from the semantic and usability perspectives to instead define an extension method on x that applies the function f using the argument(s) y.

Additional benefits are provided by the fact that:

• Extension methods may take generic arguments, thus letting you define inverted operations on groups of similar objects.

• The params keyword lets your API behave in a ‘variadic’ fashion by unchaining you from explicit collection initialization.

• Additional level of expressiveness can be gained from passing in lambdas or expressions.

The disadvantages of this approach are:

• Having to manage separate classes containing extension methods.

• API pollution — any time you define an extension method on a generic type, all objects get an additional IntelliSense popup member.

• Possible performance overhead — for example, in cases where you use a lambda instead of direct member access.

I use LIoC all over the place, just like I use the Maybe monad. It makes the code a lot more readable, maintainable and amenable to refactoring. ▪

Unfortunately, there is no headless WebKit library for .Net and current approaches all either have Java or JavaScript bindings or, alternatively, suggest that you use IKVM. However, this got me thinking: if an IKVM-driven port works (albeit not very well, in terms of performance, at least), then how about going in the direction of JavaScript solutions such as Phantom.js?

And then it hit me: I’ve already good a transcompiler called SharpKit that knows how to translate C# to JavaScript. So what if I were to simply write my code in C#, then let Phantom.js execute it as JavaScript?

What is Phantom.js, anyway?

Phantom.js is a dual API: it contains both its own APIs as well as APIs exposed through the webpage and fs modules of CommonJS. As a result, to get this API within SharpKit, I would have to trail through these APIs and generate corresponding C# classes and methods for them. This is actually fairly trivial – all I have to do is to decorate the class with [JsType(JsMode.Prototype)] and the methods with [JsMethod/JsProperty] and everything would work.

As you can see, it’s impossible to redefine property names, so the C# API has lost some of its gloss. But that’s not critical. What’s important it that I’ve been able to replicate the API of both the phantom and WebPage objects in C#. I didn’t go for the CommonJS modules because it seemed that the existing features, together with the DOM and jQuery APIs provided by Sharpkit are more than enough.

Trying It Out

To try things out, I rewrote the Phantom pizza-searching example to use C#. Here it is, in all its glory:

var page = new WebPage();
string url = "http://lite.yelp.com/search?find_desc=pizza&find_loc=94040&find_submit=Search";
page.open(url, status => {
  if (status != "success") console.log("Unable to access network");
  else
  {
    var result = page.evaluate(() =>
    {
      var list = document.querySelectorAll("span.address");
      var pizza = new JsArray<string>();
      for (int i = 0; i < list.length; i++)
      {
        pizza.push(list[i].innerText);
      }
      return pizza;
    });
    console.log(result.As<JsArray<string>>().join("\n"));
  }
  Phantom.exit();
});

Surprisingly enough, there aren’t that many caveats to note here. One issue I had is how to return types from methods that should, in all fairness, be made generic. In the end, I ended up deciding that it’s better to leave the return type as JsObject and then using the As<>() method to cast the result to whatever I actually need. The explicit type would have to appear eventually, unless of course we went with dynamic – but then again, I don’t know if this even possible.

Needless to say, the example worked out of the box, which shouldn’t really be surprising considering that I actually peeked into the resulting JS file and saw that what was generated.

Persisting Data

There’s no way getting around the fact that, with this approach, your main data mining process will be spawning an extra process for PhantomJS to do its work. Given that Phantom seems to only support the fs module, the expected mechanism for communication between the mining service and phantom would appear to be file storage. (This implies that progress monitoring is, effectively, impossible.)

At any rate, I have replicated fs in SharpKit, adding just enough of the methods in order to get the ball rolling. Annoyingly enough, I’ve declared the require() instruction in a rather strange fashion:

[JsType(JsMode.Prototype, Export=false, Name = "require")]
public static class Require
{
  [JsMethod(Name="require", Global = true)]
  public static JsObject Module(string moduleName)
  {
    return null;
  }
}

There’s probably a better way out there, as this approach requires me to use the As<>() method again to get the right module type. Ah well.

// not very pretty
var fs = Require.Module("fs").As<FileSystem>();

Given that PhantomJS uses Qt, Python and who knows what else, it’s hardly surprising that its file system API doesn’t like standard Windows paths such as c:\somewhere.txt. Not a big deal, really, because we can just write to the current directory, read off the data by our web minder, and then delete the file when we’re done. The file name would typically be provided as a parameter to the phantomjs.exe process (we get the parameters as a collection, remember?).

It’s All Gone Pear-Shaped

Having verified that the file system calls work (somewhat), I’ve decided to jump in and rewrite one of my WatiN-driven workflows using this new PhantomJS framework. That’s when I hit a couple of fairly serious problems:

• First of all, Phantom refused to work over SSL. It took some searching to figure out that, in the latest dynamic release, the authors simply forgot to include one of the required DLLs. Luckily, that was easy to fix.

• The next annoying thing I discovered is that doing something wrongly within Phantom basically causes your program to either hang or exit without any explanation. There is no debugging as far as I can see, which makes tracking things down impossible (compare with WatiN, where debugging is sensible).

• The jQuery API turned out to be insufficient for the purposes of data mining. That’s when I realized where the business value of WatiN is – the ability to write something like browser.Forms[0].TextField(t => t.Name.StartsWith("test")) is what makes WatiN a lot more usable. True, you can try to get the same result with jQuery selectors, but those typically cause silent failure and are not statically checked the wame way WatiN-based LINQ expressions are.

That last bullet point is the killer, as far as I can tell. Essentially, what this means is that if I want to have a good LINQ-driven selection mechanism, I have to write my own APIs on top of jQuery to give me the collections in the format I want. Furthermore, were I to do this (and this is a fairly substantial task), there would still be a problem debugging into these collections as they are iterated and processed.

Well, it sure was an interesting idea. Let’s hope there is more where that came from. Meanwhile, if you want the source code, you can get it here. ▪

At the time, in order to get a speed-up from a graphics card, you basically had to ‘deceive’ it by presenting your data as a texture and letting a pixel shader – a programmable GPU component capable of determinining a pixel’s color – perform mathematical computations on your behalf.

Luckily, things have improved since then. GPU manufacturers realized that their processors were far more efficient than general-purpose CPUs, albeit for a certain category of tasks. Of course, given that this category featured any kind of math where large quantities of data could be processed in parallel, the universal applicability has given rise to things such as CUDA. CUDA is NVidia’s unified API for programming the graphics card – not just for graphics but also for any kind of computation where the presence of a few hundred processor cores is of benefit.

Complexity abound

GPU computation does come with a set of limitations. It’s predominantly geared towards float data types, with double support appearing only now. Some of its numeric behavior is different from equivalent standards on the CPU – different in very obscure, hard-to-understand ways. But what’s most important is that the design of an algorithm for a GPU is entirely different than for a CPU.

In a way, GPUs exemplify the ‘multicore crisis’ – the problem where developers are in their majority not smart enough to ‘grok’ the new paradigm. This is precisely why Microsoft brought out its TPL (Task Parallel Library) API – because TPL is the absolute simplification of both task-level and data-level parallelism. Suddenly, if we want to run several methods in parallel, we use Parallel.Invoke, if we want to process a loop in parallel, we use Parallel.For/ForEach, and should we perform any kind of heavy data processing, we simply inject the AsParallel() call into our LINQ method chain and that’s it – our algorithm is supposedly made parallel, almost by magic.

With GPUs, none of these tricks work. The developer doesn’t have the option of not knowing the architecture of the GPUs they code for. Also, the combined complexity of blocks, threads, streams — together with the problems related to marshalling data to & from the GPU – make the task of writing GPU-specific algorithms so much harder.

Simplifying things (somewhat)

Now, just to be clear, there is no antidote to the complexity of parallel algorithms. We have some ‘hacks’ for the CPU which lets us not worry about frameworks such as OpenMP, PPL/Intel TBB, but it is unlikely that a ‘classical’ (CPU-specific) implementation of an algorithm can suddenly be translated to the GPU automatically. There’s no escaping the GPU specifics.

On the other hand, there are frameworks such as Microsoft Accelerator or GPU.NET which make things more manageable by hiding the really gory bits from us. For example, GPU.NET lets you define kernels right in .Net code, which automatically takes care of the problem of marshalling data to/from the GPU by hand. Also, being a managed solution, it automatically benefits from having the support of tools such as ReSharper.

Personally, I see this persistent trend of trascoding certain languages (or their compiled form) into another representation. And if IL can be transcoded into a stream of NVidia-specific instructions, who’s to say it cannot be transcoded to, e.g., a netlist for synthesis on an FPGA?

If you cannot be bothered to read all of this, just go and see the demo, available at http://nesteruk.org/talk. Source code is, as always, available at https://bitbucket.org/nesteruk/restfulwebconversation.

Back to JSONP

So the first things I wanted to do is to get JSONP back. As it turns out, the Web API still doesn’t have this built-in (why, I wonder?), but the project site was helpful – I found this post, which basically gave me an implementation of JSONP on a silver platter. Or not.

In actual fact, things went wrong from the start because

• There’s still no support for the header which accepts any media type (i.e., Accept: */*)

• The default formatters ‘take over’ whatever you do

• The implementation I found on the CodePlex site is just plain wrong

So let’s deal with the above issues. It’s clear that we have to define our own JsonpFormatter. But how do we handle the */* header? The answer is that we specify a media range mapping such that an unspecified range maps onto application/json:

public class JsonPFormatter : MediaTypeFormatter
{
  public JsonPFormatter()
  {
    this.AddMediaRangeMapping(new MediaTypeHeaderValue("*/*"), new MediaTypeHeaderValue("application/json"));
    SupportedMediaTypes.Add(new MediaTypeHeaderValue("application/json"));
  }
  ...
}

The above basically ensures that:

• A request with */* in the header will return JSON

• A request with an accept value other than */* or application/json will be processed by another formatter. This ensures that XML support is still there.

Now, let’s finish out the formatter. First, we determine whther we can read or write a particular type. Since there is only one type in our app, this is somewhat easy:

protected override bool OnCanWriteType(Type type)
{
  return type == typeof (ConversationItemDTO);
}
protected override bool OnCanReadType(Type type)
{
  return false;
}

Now, we need to take care of the method that writes the resulting data to the stream. The original CodePlex sample made the mistake in that it assumed that a JSON request always had to be JSONP, but that’s not correct. So, I changed it to selectively return JSON or padded JSON depending on whether the callback parameter was provided:

...
var callback = queryString["callback"];
if (callback != null)
  sb.Append(callback + "(" + jsonString + ");");
else
  sb.Append(jsonString);
byte[] buffer = Encoding.UTF8.GetBytes(sb.ToString());
stream.Write(buffer, 0, buffer.Length);

So, given this, I hit a final problem: how to make sure that my formatter takes precedence over the default JSON formatter? After some searching, I found the solution: inserting it as the first formatter in the configuration:

// jsonp formatter comes first!
config.Configuration.OperationHandlerFactory.Formatters.Insert(
  0, container.Resolve<JsonPFormatter>());

And that’s it – I got my JSONP support! Consumption can now go back to normal:

JQuery.GetJSON(serverUrl + id + "?enabledOptions=" + !enabledOptions + "&callback=?",
  ...

Better conversation definitions

Admittedly, the current way of defining conversation lines with its enable/disable list is a bit archaic. Actually, it does serve its intended function, it’s just not very convenient – especially if I want to define a rather large conversation without using a graphical DSL.

My approach to this problem was to create a factory class that would take care of routine problems and would keep track of all the items for subsequent insertion. What I ended up is the following:

public class ConversationFactory
{
  private int id = 1;
  public List<ConversationItem> Items;
  public ConversationFactory()
  {
    Items = new List<ConversationItem>();
  }
  public ConversationItem Create(Party party, string speech, List<int> disableList = null, 
    params ConversationItem[] enabledItems)
  {
    var ci = new ConversationItem();
    ci.Id = id++;
    ci.Speech = speech;
    ci.PartyId = (int) party;
    ci.DisableList = disableList ?? new List<int>();
    ci.DisableList.Add(ci.Id);
    ci.EnableList = new List<int>();
    foreach (var i in enabledItems)
      ci.EnableList.Add(i.Id);
    Items.Add(ci);
    return ci;
  }
}

The above code generates the IDs and takes care of some of the obvious things, such as conversation items disabling themselves as soon as they are fired. Populating the site is then as easy as the following code:

var f = new ConversationFactory();
f.Create(Party.Server, "Welcome to my site! Feel free to ask questions.", null,
  f.Create(Party.Client, "Tell me about your professional life"),
  f.Create(Party.Client, "Tell me about your personal life"),
  f.Create(Party.Client, "Tell me about how this site works"),
  f.Create(Party.Client, "Actually, I think I better go now")
  );
repository.Upsert(f.Items);

The above set-up totally rocks for creating a simple state machine. Why? Because statements can be nested, meaning I can alternate server/client dialogue items for as long as I like.

Deployment

Deployment for the WCF REST service follows the typical protocols: you make a web application in the folder where the service is located, ensure it’s .Net 4, and specify wildcard mappings (yes, poor me, I’m on IIS 6). Subsequently updating the service is easy – you can just do ‘xcopy’ deployment via FTP. No restarts are necessary because generally IIS is smart enough to pick up changes on its own.

The great thing is that deployment of the client side is a breeze because it’s pure HTML. So you just copy over the files and everything works, though I’ve had to specify the default document (it’s an .html file, after all). Since the client side is totally detached, you can actually use your browser to debug those JSONP calls (I found an error this way), as opposed to the tedium of having to go to the server via RDP or something. This is in sharp contrast with the WCF service itself.

Conclusion & Futures

This series of posts has demonstrated some ways of working with a pure client-side web app that uses a REST backend. It is by no means complete, since I neglected to cover some of the important issues such as caching. However, it is my hope that this project serves as a reference application that one can build upon. ▪

Source code is, as always, available here: https://bitbucket.org/nesteruk/restfulwebconversation

JSONP, again…

Before doing anything, I mirrored the conversation data structure using the F# record type:

type ConversationItem = {
  Id : int
  PartyId : int
  Speech : string
  Responses : ConversationItem list
  }

So to try things out, I wrote a very simple function that simply did a GET of some data from the server:

[<JavaScript>]
let serverUrl = "http://localhost:51688/"
[<JavaScript>]
let Update(id:string) =
  JQuery.GetJSON(serverUrl + id,
    fun (result:ConversationItem, msg) ->
      JQuery.Of("#serverItem").Text(msg)) |> ignore

This raised the all too familiar error message:

XMLHttpRequest cannot load http://localhost:51688/. Origin null is not allowed by Access-Control-Allow-Origin.

For a moment I thought that maybe I’ve gone crazy and I’d have to implement JSONP support manually like I did for WCF REST (in a pre-.Net 4 scenario), so I tried to adjust the query string to explicitly invoke the callback:

[<JavaScript>]
let Update(id:string) =
  JQuery.GetJSON(serverUrl + id + "?callback=?",
    fun (result:ConversationItem, msg) ->
      JQuery.Of("#serverItem").Text(msg)) |> ignore

Now that I got this working, I hit another problem: the result I got from the service was XML! Now I know that JSON works fine with the new API, but when I opened Fiddler, what I saw is that the header had an Accept: */* instead! As it turns out, with JSONP you cannot even customize it, which means that my nice, resource-oriented server must be forced to return JSON by default.

While waiting for clarification on how to do this (typical fiddling in Web.config didn’t help one bit), I turned off Chrome web security and went with a JSON (as opposed to JSONP) approach. I suspect that the lack of JSONP support us due to the nature of the Web API, which isn’t ready yet.

(On a side note, I noticed that the entity names of my lists in XML had a wrong name. So using XML would have been impossible anyway without further fiddling the XML serializer.)

Consuming the Service

So to start with I create a bare-minimum HTML structure to house my UI:

[<JavaScript>]
let Conversation() =
  Div [
    P [Id "serverItem"] -< [Text "Server text here"]
    Div [Id "clientItems"]
    Button [Text "Hello"] |>! OnClick (fun x y -> Update(""))
  ]

And I subsequently try to project each piece of data onto the UI using the following piece of code:

[<JavaScript>]
let Update(id:string) =
  JQuery.GetJSON(serverUrl + id,
    fun (item:ConversationItem, msg) ->
      // what the server says
      JQuery.Of("#serverItem").Text(item.Speech).Ignore
      // what the client can respond with
      // hateoas but without atom:link
      if not (List.isEmpty item.Responses) then
        let listItems = 
          item.Responses |> List.map (fun r -> LI [ A [HRef "#"] -< [Text r.Speech]])
        let wholeList = OL [] -< listItems
        JQuery.Of("#clientItems").Empty().Append(wholeList.Dom).Ignore
    ) |> ignore

Now, we can turn to the next and most painful part of our site – the handling of links.

Links and HATEOAS

Okay, so what should those links contain? Common sense dictates that the URL they point to should contain the link to the conversation item that has just been pressed, e.g., http://mysite.com/123 where 123 is the ID of what I just said.

That’s all well and good but gets us nowhere, because we’re not interested in postbacks (it’s pure AJAX) and because even if we get the data at the URL, we’d still need to render it. Anyways, we can agree that we need the link, if only to do another AJAX call.

The moral of the story here is that you don’t get nice permalinks, you cannot add a bookmark to a conversation point (largely because the conversation has a context which we cannot preserve), but you can define AJAX-based link handling actions en masse. Essentially, we make the Update() function recursive and then wire up each link to call it:

let listItems = 
  item.Responses |> List.map 
    (fun r -> LI [ A [HRef "#"] -< [Text r.Speech]]
              |>! OnClick(fun x y -> Update(r.Id.ToString()))
    )

This works, but what about the progressive enabling/disabling of links? Our Empty() JavaScript call wipes all existing links but they must, presumably, be stored somewhere on the client in order to be useful.

If you remember my previous post, we agreed to make the list of enabled options a string such as 1,2,3. The server then repopulates the set of links each time we get a response, taking into account the links we already have. Now, how can we put together this string without creating any extra data structures? The answer is, of course, to use jQuery.

The agreement here is to assign each link its ID

fun r -> LI [ A [HRef "#"; Id (r.Id.ToString())] -< [Text r.Speech]]
...

and subsequently iterate over all of them, creating the query string. Now, due to F#‘s statelessness, the code below uses ref cells and is a bit ugly:

let enabledOptions = ref ""
 
JQuery.Of("#clientItems").Find("a").Each(
  fun (e:Dom.Element) -> let id = JQuery.Of(e).Attr("id")
                         enabledOptions := !enabledOptions + id + ",").Ignore

And then, of course, we pass enabledOptions as a parameter to the URL. Note that this could be considered bad by some REST purists used to clean URLs. However, this is something I can live with.

Conclusion

As you can see, I am not posting the full example online yet. There are a few reasons why I am not doing this:

• I am still not sure what the situation with JSONP is right now. I hope that I’m just missing something in the new API, as opposed to having to implement this myself. Same goes for having JSON as the default output type for the service.

• I am still looking for a way to define conversations more efficiently. The main problem right now is that the way I’m doing it with enable/disable list causes it to be a ’poor man’s HSM’ instead of a proper one. Perhaps the Stateless library could be of some use here.

• Another problem I’m having is that, due to the fact that HTML is generated via JavaScript, I can’t see a place where I can initialize the content of my HTML after it has been returned. This is mainly due to my lack of knowledge of WebSharper, but I hope to figure it out before I post the next part.

Overall though, I’m happy to say that I didn’t write a single line of JavaScript to get it all working. To be continued!

The source code for this project is available online at http://bitbucket.org/nesteruk/RestfulWebConversation

The Nature of Conversations

Conversations are great for trying out RESTful services. They are, at the same time, stateless (the conversation structure doesn’t change often on the server), and stateful in the sense that the user’s position along a conversation line must be preserved somewhere. As a result, conversations are uniquely positioned to test out the new REST Web API that Microsoft is coming out with.

So let’s get started. The first thing I’m going to do is once again define the conversation entities for persisting them in a database. In this case, there is only one entity: a ConversationItem.

public class ConversationItem
{
  [BsonId] public int Id;
  public int PartyId;
  public string Speech;
  public List<int> EnableList;
  public List<int> DisableList;
}

Now, we can easily create a ConversationRepository class that would implement the typically CRUD operations – I typically like to lump Create and Update into a single operation though.

With this structure defined, let’s talk about the way we are going to serve the data via RESTful endpoints. My first attempt at creating this service was somewhat confusing: I couldn’t figure out what project template we’re meant to use together with the new WCF Web API. I mean, the libraries themselves are on NuGet, but what type of project should it be? And how should it be configured?

Not having any particular idea, I took the WCF REST for .Net 4 template off the gallery, but ended upthrowing away most of what got generated and adding in the WebAPI stuff.

DTOs

In order to get the data accross the wire, I added a pair of DTO classes. The first is a ConversationItemDTO, which is essentially a DTO (data transfer object) for the conversation items. It mirrors the original entity completely, but has a different type of ‘noise’ attached to it – first, it uses properties instead of fields; second, it has prescriptive attributes that determine its layout should XML serialization be used (I care mainly about JSON, but still…).

[XmlRoot("ConversationItem")]
public class ConversationItemDTO
{
  [XmlAttribute("Id")]
  public int Id { get; set; }
  [XmlAttribute("PartyId")]
  public int PartyId { get; set; }
  [XmlAttribute("Speech")]
  public string Speech { get; set; }
  [XmlAttribute("EnableList")]
  public List<int> EnableList { get; set; }
  [XmlAttribute("DisableList")]
  public List<int> DisableList { get; set; }
}

The second DTO entity is a list aggregation of the above:

[XmlRoot("ConversationItems")]
public class ConversationItemList
{
  [XmlArray]
  public List<ConversationItemDTO> Items { get; set; }
}

Finally, I have added an entity-to-DTO extension method for the conversation items. It seems fitting that the transposition method is contained in neither the original entity nor the DTO, though we ould have easily stuck it into the DTO if need be:

// booooring :)
public static ConversationItemDTO ToDTO(this ConversationItem item)
{
  var dto = new ConversationItemDTO();
  dto.Id = item.Id;
  dto.Speech = item.Speech;
  dto.PartyId = item.PartyId;
  dto.EnableList = item.EnableList;
  dto.DisableList = item.DisableList;
  return dto;
}

This may seem a bit trivial, but that’s because the HATEOAS paradigm is not in play yet. In fact, we are currently cheating by returning enable/disable lists instead of telling the client which options are valid at a particular location. Let’s try to fix this.

HATEOAS

Okay, so before we get to returning links to available conversation options (that’s what HATEOAS is all about, in this case), let’s do a few simple things. The first would be to return a set of initial conversation items on the default URL. Just like the RestCart sample, I have defined a resource class, which is a typical WCF service contract, but with a lot less noise:

[ServiceContract]
public class ConversationResource
{
  private readonly IConversationRepository repository;
  public ConversationResource(IConversationRepository repository)
  {
    this.repository = repository;
  }
  [WebGet(UriTemplate = "")]
  public ConversationItemDTO GetInitial()
  {
    var result = repository.FindOne(1).ToDTO();
    result.Responses = repository.FindMany(new[] {2}).Select(r => r.ToDTO()).ToList();
    return result;
  }
}

This is really as good as it gets – no huge attribute stacks, no complicated Web.config manipulation. This is all that is needed to publish an entity. In the above Get() method, you can see that I’m simply using the original Mongo repository, taking the first 4 items, converting them to DTOs and returning them wrapped in a list.

So far so good, but no hypermedia links in sight. Let’s try to add a few. In fact, we are going to get rid of the enabled/disabled lists and instead return the items relevant at this particular point. In other words, our DTO becomes something like the following:

[XmlRoot("ConversationItem")]
public class ConversationItemDTO
{
  [XmlAttribute("Id")]
  public int Id { get; set; }
  [XmlAttribute("PartyId")]
  public int PartyId { get; set; }
  [XmlAttribute("Speech")]
  public string Speech { get; set; }
  [XmlArray("Responses")]
  public List<ConversationItemDTO> Responses { get; set; }
}

That’s better, but still confusing: how do we incorporate client state (i.e., where the client is in the conversation thread) into this model? The answer is that we can simply pass an array of currently enabled items in a GET request. (Note: I’m serious about this being a GET and not a POST. We are not changing server data, just providing some hints for the server to compute the next set of links.)

With that in mind, we define the following stub for a method getting a conversation item:

[WebGet(UriTemplate = "{id}?enabledOptions={enabledOptions}")]
public ConversationItemDTO Get(int id, string enabledOptions = null)
{
  ...
}

You’ll notice that there is a string argument with a comma-separated list of currently enabled items. The id here refers to the conversation item the user chose. We start by checking that it’s valid and returning a 404 if it’s not:

// get the chosen option from repository
var userResponse = repository.FindOne(id);
if (userResponse == null)
{
  var ctx = WebOperationContext.Current;
  ctx.OutgoingResponse.SetStatusAsNotFound(
    string.Format("Could not find a client response with id={0}", id));
  return null;
}

Next, we build a HashSet<int> with the relevant items and exclude the disabled ones:

// compute the set of relevant items
var items = new HashSet<int>();
if (enabledOptions != null)
  items.UnionWith(enabledOptions.Split(',').Select(s => int.Parse(s)));
items.UnionWith(userResponse.EnableList);
items.ExceptWith(userResponse.DisableList);

Finally, we get the actual items from the server, split them into the server and client parts, and return the result.

// get all the items
var all = repository.FindMany(items).OrderBy(i => i.Id);
var grouped = all.GroupBy(i => i.PartyId)
  .ToDictionary(k => k.Key, v => v.ToList());
      
// get server and client items
var server = grouped[(int) Party.Server].First();
const int clientKey = (int) Party.Client;
var client = grouped.ContainsKey(clientKey) ? grouped[clientKey] 
                                            : EmptyList<ConversationItem>.Instance;
// now - finally - build the response
var result = server.ToDTO();
result.Responses = client.Select(i => i.ToDTO()).ToList();
return result;

That’s all there is to it! Let’s discuss how it all works.

Using the Service

I will show the client-side usage of the service in the next post: probably when I get back to a computer that has a working WebSharper license on it. Meanwhile, let me explain how the service works.

When someone goes to http://myservice.com/, the service returns a representation of the default conversation items for the server (the outer ConversationItem) and the set of possible responses. This is completely in line with HATEOAS in the sense that we return a set of links to things that can be done.

When someone goes to http://myservice.com/X, we assume that the client chose conversation option with id=X. We calculate the server response and the next set of client responses and return them to the consumer.

When someone goes to http://myservice.com/X?enabledOptions=A,B,C, the server also includes the currently enabled items A, B and C in the set of possible responses. This is necessary because we are typically too lazy to include the full set of responses each time. Instead, we do things incrementally. I have not tested this with client-side jQuery invocation, but there should be no problem.

Conclusion

Building RESTful services is now somewhat easier with the Web API, but there are still several fundamental issues which are, overall, not addressed:

• First, there are no helpers for projecting data collections. The creation of DTOs is tedious, and I’ve spent a few hours testing the service to make sure that things are returned correctly.

• It is entirely unclear how to handle errors. Should we throw exceptions? Or just return 404 in the output stream and return null? It would be great to have some prescriptive guidance and a few mechanisms to ensure that if I’m projecting data, all the typical problems (such as accessing an entity with a non-existent ID) are handled without my involvment.

• The WCF Web API has no built-in mechanism for creating state machines and wiring them into a HATEOAS paradigm. It would be great to have this – possibly in the WF4 updates that have come out in the .Net Framework 4 platform update.

Overall, though, I’m happy with the results I got from the Web API, and am also very happy with the way NuGet let me add a pile of references to the projects. In the next post, I’ll build a single-page-AJAX front end and will probably hit an altogether different set of issues.

That’s all for now. Comments welcome!

That was then, but now I’m happy to announce ActiveMesa MathSharp: a tool to do just that! With MathSharp, you can take an equation from any MathML-producing application (such as Microsoft Word 2007/2010, MathType or Maple) and quickly convert it to C# or F# code. Here’s a video that demonstrates the way MathSharp works:

To learn more, visit the MathSharp page at http://activemesa.com/mathsharp

Well, admittedly, writing the pure JavaScript by hand is probably less than pleasant. However, seeing how I dabble in WebSharper from time to time, I figured this would be a good opportunity to try something out. In this case, I decided to build a conversation system that would let us define the conversation graph¹ and the site user can then iteract with a site in the form of a scripted dialogue.

Click here to see the demo

Defining Conversations

A conversation is a sequence of speeches (can’t think of a better word) that people deliver in turn. Each conversation has the text that someone says, but it also involves a couple of other things. Take a look at the definition of a conversation item:

type ConversationItem = {
  mutable Enabled : bool
  Id : int
  PartyId : int
  Speech : string
  EnableList : int list
  DisableList : int list
  }

Each item has an Id used to uniquely identify it among others. It also has an Enabled field used to indicate whether this conversation item is relevant at the current point in time. Then there’s the PartyId indicating who the conversation item belongs to (in our case – either the Client or the Server).

And then there are the two lists – EnableList and DisableList. These hold list of Ids of items that get enabled or disabled when this conversation item is ‘fired’ (said). Since people don’t typically repeat themselves, the item is itself disabled right after it is used².

With this model in mind, let’s try building the conversation engine.

Conversation Engine

The conversation engine consists of the conversation graph – a graph of all possible conversation items and the relationships between them – and a number of utility functions for getting the client and server responses or ‘speeches’.

Let’s look at the graph itself. The graph is simply a list of all possible conversation items:

[<JavaScript>]
let private ConversationGraph = [
    {server with Id = 1; Enabled = true; Speech = "Welcome to a demo of Conversations. Please select one of the options below"}
    {client with Id = 2; Enabled = true; Speech = "Tell me what this is all about."; EnableList=[6]; DisableList = [1]}
    {client with Id = 3; Enabled = true; Speech = "Tell me about the technology used to implement this"; EnableList=[7]; DisableList=[1]}
    {client with Id = 4; Enabled = true; Speech = "Actually... I'm not that interested"; EnableList = [5]; DisableList = [1;2;3]}
    {server with Id = 5; Speech = "Well, okay, fair enough"; DisableList = [2;3;4]}
    {server with Id = 6; Speech = "Well, essentially, it's all about using F# to generate awesome JavaScript"}
    {server with Id = 7; Speech = "This demo uses pure HTML and JavaScript courtesy of WebSharper and F#"}
  ]

The graph is built upon two prototypes called client and server. These simply set the correct PartyId fields and also a number of useful defaults.

[<JavaScript>]
let private ServerId = 1
[<JavaScript>]
let private ClientId = 2;
 
[<JavaScript>]
let private server = { Enabled = false; Id = ServerId; Speech = ""; PartyId = 1; EnableList = []; DisableList = [] }
  
[<JavaScript>]
let private client = { server with PartyId = ClientId; }

With the conversation graph fully defined, we can write methods to provide the server and client items relevant at a particular point in the conversation:

[<JavaScript>]
let ServerSpeech() =
  ConversationGraph 
  |> List.find (fun f -> f.PartyId = ServerId && f.Enabled)
 
[<JavaScript>]
let ClientSpeech() =
  ConversationGraph
  |> List.filter (fun f -> f.PartyId = ClientId && f.Enabled)

Last but not least, we need a method for when a speech item gets fired. This is where we disable the item just fired, and also apply its enable/disable lists to other items.

[<JavaScript>]
let SpeechItemFired id =
  let item = ConversationGraph |> List.find(fun f -> f.Id = id)
  item.Enabled <- false // assume all fired items are not re-used
  // enable and disable all items with this id
  ConversationGraph
  |> List.iter(fun f -> 
    // if contains in the enable list, enable it
    item.EnableList  |> List.iter(fun en -> if en = f.Id then f.Enabled <- true)
    item.DisableList |> List.iter(fun de -> if de = f.Id then f.Enabled <- false)
    )

And that concludes the ConversationEngine module in all its simplistic beauty. We now have a pure JS-driven conversation definition, so it’s time to make use of it on the page.

Presenting Conversations

Let’s start with a simple thing – the conversation layout. For brevity’s sake, we’ll have two Divs one on top of another.

[<JavaScript>]
let Conversation() =
  Div [Attr.Style "width: 700px; height: 450px; border: 1px solid gray"] -< [
    Div [Attr.Style "height: 300px"] -< [
      P [Id "serverSpeech"; ConversationEngine.ServerSpeech().Speech |> Text]
    ]
    Div [Attr.Style "height: 150px"] -< [
      WiredClientSpeech()
    ]
  ]

Notice how we’re using ServerSpeech() directly, but the client-side part refers to something called WiredClientSpeech(). This is necessary because server speech is simply paragraph text, but client speech is a bunch of links with OnClick() event handlers attached to them.

Here comes a major F# dilemma: links update conversations, but a conversation update forces the replacement of these links (and whatever is holding them). How can we handle this? Well, in F# there’s no other way but using mutually recursive functions, which sounds scary, but is actually fairly simple. So, we have a function called WiredClientSpeech() – let’s take a look at the way it’s defined:

[<JavaScript>] WiredClientSpeech() =
  let stuff =
    ConversationEngine.ClientSpeech()
    |> List.map(fun f ->
       let id = f.Id
       LI [ 
         A [HRef "#"] -< [Text f.Speech]
         |>! OnClick(fun x y -> UpdateConversation id |> ignore)
       ] )
  OL [Id "clientSpeech"] -< stuff

Okay, so what happens here is we get a list of LI elements which are then fed into an OL, which we also give a specific id. The slight peculiarity of the way the above is defined is due to the fact that you cannot simply place a list of LIs inside an OL – you will experience type inferencing failure. That’s right – F# isn’t perfect! Anyways, it’s easily solvable with the handy -< operator, or you could just pipe it via |> OL. Of course, we also needed to provide an Id, thus things are defined the way they are.

In the above, you see a call to UpdateConversation with a cached (to prevent closure mishaps) id parameter. UpdateConversation() is the function that uses the conversation manager and actually puts converation items into the DOM.

let rec UpdateConversation id =
  ConversationEngine.SpeechItemFired id
  let srv = ConversationEngine.ServerSpeech()
  JQuery.Of("#serverSpeech").Text(srv.Speech) |> ignore
  ConversationEngine.SpeechItemFired srv.Id
  JQuery.Of("#clientSpeech").ReplaceWith((WiredClientSpeech() :> Element).Body)

So, what’s going on here? Well, we tell the conversation engine that a speech item has been fired, we get the server item first and then simply replace paragraph text with the new item. Then, we also inform that a server item has been fired. Finally, we do something tricky – we replace the #clientSpeech element completely, substituting a Dom.Element that we create synthetically in WiredClientSpeech(). Note that a cast is necessary here because, once again, type inference seeks to mess with us.³

Conclusion

The above problem is something that is very difficult (but doable) in ordinary JavaScript. However, with a little help from WebSharper, we managed to orchestrate the whole conversation process in terms of F#. There are a few caveats though, as always:

• The site isn’t exactly small due to the JS infrastructure support. A large number of files was dragged in. They’re necessary though – especially when the site is more ‘real’, in the sense of communicating to servers via JSON or whatnot.

• The F#→JS transformation is very picky. You can easily mess it up and end up crashing the compiler. To be fair, Web# is beta, but it’s still important to remember that not everything can be translated to JavaScript.

• F#‘s preference for code to be ’in order’ is most inconvenient here. Mutually recursive functions are one thing, but I’ve also been forced to move the conversation engine into a separate module, because keeping it in the main client module caused JavaScript to be emitted in the wrong order, thus spoiling the party.

Overall, I’m happy with my first little foray into the world of HTML sitelets. Now that Web# actually supports all of this, it makes sense to experiment with client-server REST interoperation. I’m actually curious as to how Web# can help here, but for the server side there’s always WCF REST or OpenRasta and the like.

Please check out the demo! And stay tuned for more F#/WebSharper goodness!

Notes

1. ↑ I’m stretching things a little when calling a scripted conversation a graph. It’s really more like a state machine. The kind of conversations you have in games have much greater complexity, and they also have a measure of reentrancy and context-keeping that a simple graph doesn’t support.
2. ↑ It’s worth noting that in a real-world interaction scenario, conversation choices affect a lot more than whether some item is enabled or disabled. However, given that we are working with a virtually object-oriented representation, it’s fairly easy to attach any sort of context-oriented action to particular conversation items. I leave this as an exercise for the reader.
3. ↑ I must admit that the process of organizing this methods was far from intuitive. I had to try a number of things before I figured out how to externalize everything and get it all to work. Admittedly, the code I got at the end isn’t as readable or well-organized as it could be, but this is mainly due to F#‘s limitations.

Every once in a while we all suffer from the Not Invented Here syndrome. So do I. This time, despite of being able to use either one of available web servers or previously written ASP.NET MVC-like web serving framework, I decided to write my own.

My requirements are that the server should serve JSON, provide meaningful URLs, be easy to use (anything that you write is by design easy to use for you), it should be hostable in a simple console app, should not use reflection as ASP.NET MVC does, and it should be written in F#. Because “not-written-in-F#” is a fundamental flaw of most existing systems I know of.

Design

Since this project is going to be a quick and dirty prototype of a future system – let’s use HttpListener as a serving mechanism. It is pretty straightforward and easy to use. Ok. What’s next?

Any kind of ‘web serving’ starts with a URL. Those who tried Django already know how URL handling is performed there. For others, here are the basics. There is a single dictionary (I mean something like System.Collections.Generic.IDictionary here) that maps a URL to the function that should handle requests to that URL. I’m oversimplifying but that’s enough for our purposes. The request handling function should get something in and return back our JSON as a string, which then will be wrapped in a HTTP response and sent back to the user agent. The only question now is what should be passed to the handler function. If we were writing a real web server, it would require us to pass the whole response context into the function, but we want some REST here, therefore the requested URL would be enough. In Django, they use regular expression match groups to obtain arguments for handler functions from the requested URL. Why not do so as well? Let’s create a simple URL mapping for clarity.

[("^/hello/([a-zA-Z]+)", helloNameHandler);]

The above means that when a URL matching the pattern is requested, we should take the URL substring containing a person’s name and pass it to the handler function helloNameHandler. Let’s not forget about the URL query string. We’ll parse it as well and pass to the handling function as a list of (string * string) pairs.

Implementation

Enough said, lets start implementing. First of all we’ll describe all the types already discussed:

type RequestData =
  // first list is url substrings, second list is query string pairs
  | RequestData of string list * (string * string) list
 
type ResponseData =
  | Json of string
 
type HandlerFunction = (RequestData -> ResponseData)
type DispatchEntry = (string * HandlerFunction)
type DispatchEntryEx = (Regex * HandlerFunction)

Since we’re using regular expressions for our patterns we might have some speed gain if they are compiled before usage, hence the DispatchEntryEx type.

let makeDispatcher (pattern, handler) =
  let fullPattern = sprintf "%s/?.*" pattern
  let rx = Regex(fullPattern, RegexOptions.Compiled)
  (rx, handler)
let prepareDispatchers (table:DispatchEntry list) : DispatchEntryEx list =
  List.map makeDispatcher table
module Query =
  let decode : string -> string =
    HttpUtility.UrlDecode
  
  let processParams (s:string) =
    if String.IsNullOrEmpty(s) then
      []
    else
      let s' = s.TrimStart([|'?'|])
      s'.Split([|'&'|])
      |> Array.map (fun pair -> pair.Split([|'='|]))
      |> Array.map (fun  [|h;t|] -> (h, t))
      |> Array.toList
  
  let processPath (rx:Regex) (s:string) =
    rx.Match(s).Groups.Cast<Group>()
    |> Seq.skip(1) // skipping the overall group
    |> Seq.map (fun g -> g.Value)
    |> Seq.toList

Nothing interesting here: we are creating a dispatch table with compiled regular expressions and two functions to provide parameters to our handler functions. Now we are ready to implement a dispatch function, but first we should define its type.

The HttpListener class provides a GetContext() method of type HttpListenerContext that in turn has the Request and Response properties. Processing a request means reading from context.Request and writing something back into context.Response instance. That said, our dispatch function should have the following type: DispatchEntryEx list -> HttpListenerRequest -> HttpListenerResponse -> unit. It does not return any value, because the actual result is written directly to the response as a side effect (We could purify the approach if we wanted to, but not this time).

let dispatch (ds:DispatchEntryEx list) request response =
  let (!!) = Query.decode
  let absPath = !! request.Url.AbsolutePath
  let queryString = !! request.Url.Query
  
  printfn "(II) '%s' requested..." <| !! request.RawUrl
  
  let (rx, handler) =
    ds |> Seq.find (fun (r, _) -> r.IsMatch(request.Url.AbsolutePath))
  
  let path = Query.processPath rx request.Url.AbsolutePath
  let query = Query.processParams queryString
  
  let resultBytes =
    match handler (RequestData (path, query)) with
    | ResponseData.Json json ->
      let callback = Query.get "callback" "callback" query
      json |> Encoding.UTF8.GetBytes
  
  response.ContentLength64 <- int64 resultBytes.Length
  response.OutputStream.Write(resultBytes, 0, resultBytes.Length)

Yet again, nothing complex. The function does exactly what it supposed to. First it searches for the matching handler in dispatch table, then parses the request URL path and query string, after that applies handler function and finally writes everything back to response stream.

The last thing we should implement before we could test our server, is server initialization function.

let start (prefix:string) (table:DispatchEntry list) =
  let listener = new HttpListener()
  listener.Prefixes.Add(prefix)
  listener.Start()
  
  let tableEx = prepareDispatchers table
  
  let requestProcessor =
    let processContext (ctx:HttpListenerContext) = async {
      dispatch tableEx ctx.Request ctx.Response
      ctx.Response.Close()
    }
    async {
      while true do
      try
        let ctx = listener.GetContext()
        processContext ctx |> Async.Start
      with
      | ex -> printfn "(EE): %s" ex.Message
    }
  
  requestProcessor
  |> Async.StartAsTask
  |> ignore
  
  {new IDisposable with
  member this.Dispose () =
  listener.Close()
  }

Please note the usage of async and Async.Start with the processContext function. Without it the whole server will be able to only process requests one after another. HttpListener.GetContext blocks execution until the request is received, right after that we start a new processing thread with Async.Start, which returns immediately and the while loop continues to the next call of GetContext leaving context processing to a separate thread.

Another thing to note is the return type of the start function. It returns IDisposable instance which will stop the listener when disposed, so we don’t have to expose the fact that we use HttpListener to the user still having some control over listeners lifetime.

Some improvements

The code above seems to work (at least it should), but then we realize that JSON is not the only thing we’d like to be able to serve. We would also need some binary data such as images. To support this we have to modify the return type of our handler function. Thanks to discriminated unions, it won’t require too many changes in the code.

There are actually only two places in the whole code that are subjected to change in order to support serving images. First – the ResponseData type:

type ResponseData =
| Binary of byte[]
| Json of string

And second, the dispatch function:

let dispatch (ds:DispatchEntryEx list) request response =
//...
let resultBytes =
match handler (RequestData (path, query)) with
| ResponseData.Json json -> json |> Encoding.UTF8.GetBytes
| ResponseData.Binary bytes -> bytes
// ...

That’s all! We simply added support for a new case to all places where ResponseData is used. No need to change already developed handler functions.

Possible further improvements

Lets imagine that once this “web server” evolves into something more complex and we would need to support “areas” of site, where each area will process URLs separately so that dispatch table will become unbearably long and uncomfortable to use. F# discriminated unions can help us here as well, simply change the HandlerFunction into something like type UrlHandler = | HandlerFunction of (...) | SiteArea of (...) etc. Django also has this feature but here we don’t use dynamic types.

It is also possible to use some templating libraries (or view engines) to render responses from handler functions. This will require just another case in ResponseData type and a slight change dispatch function.

Conclusion

In this article we developed a dead simple yet working web serving solution that can be used to prototype URL base REST services to be hosted in your application. It has not been properly tested and surely has limitations, but the development process showed that it is possible and pretty easy to implement a web serving solution in a .NET environment without using reflection or dynamic typing.

The F# programming language is really fun to work with. It lets you improve your code almost infinitely, not only in terms of performance but also readability. Having finished reading through McConnell’s Code Complete I’ve decided to add some declarativeness to JSON processing code I developed for my little pet project Linq2vk. And since JSON is ubiquitous these days, I hope the technique described below might be of use to others as well.

Data

It is common for all modern online services to provide a public API with JSON as an option of data representation format and all the data we get in response from those services is either a huge JavaScript object or an array. Let me leave objects for further investigations and concentrate on arrays like the following one:

[123, "Maxim", "Moiseev",  "http://bitbucket.org/moiseev"]

This is an array (or list if you will) of simple objects, of which we are sure that it contains an integer for an Id and then three string values for first name, last name and homepage URL respectively. We are not going to actually parse JSON and represent it as a .NET list of tokens, it is better to use a great Json.NET library for that task. Our target is writing a function of type JToken[] -> Person, where JToken is a list of JSON tokens provided by Json.NET and Person is a simple user defined .NET class. In other words, we are going to deserialize JSON array into an instance of a .NET class.

Initial approach (naïve)

There are (as always) several ways of achieving this. The simplest one, which I have used until now is as follows (for simplicity I won’t define the Person class here, using a tuple instead):

let createPerson (ts: JToken[]) =
    let id = ts.[0] |> int
    let name = ts.[1] |> string
    let lastName = ts.[2] |> string
    let url = ts.[3] |> string
    (id, name, lastName, url)

Note that int and string here are type conversion functions, so we can use any other converter like for example this one:

let intBool (token:JToken) =
    0 <> (token |> int)

This approach works perfectly until, say, the URL becomes optional so that the source JavaScript list can sometimes contain 4 tokens and sometimes only 3.

let createPerson (ts: JToken[]) =
    let id = ts.[0] |> int
    let name = ts.[1] |> string
    let lastName = ts.[2] |> string
    let url =
        if ts.Length > 3 then
            ts.[3] |> string
        else
            String.empty
    (id, name, lastName, url)

Still, this works but looks rather ugly. Thanks to the if statement inside the almost declarative code. Let’s try to avoid ifs.

let optional<'T> (tokens:JToken[]) (index:int) convert =
    if index < tokens.Length then
        convert tokens.[index]
    else
        Unchecked.defaultof<'T>

This optional function lets us process our list of JTokens like this:

let createPerson (ts: JToken[]) =
    let id = ts.[0] |> int
    let name = ts.[1] |> string
    let lastName = ts.[2] |> string
    let url = optional<string> 3 string
    (id, name, lastName, url)

Much better now, but still field processing code does not look unified: normal fields are accessed directly while optional one works through a separate function, conversion function in former case is called as a part of pipe but is passed as a parameter in the latter. Still not good enough.

Better approach

Here comes the idea to use all the knowledge from multiple infinitely long and smart articles about monads and parser combinators. Luckily F# gives us its Computation Expressions feature, which is all about those monads.

Fine. Let’s go. We’ll start with the definition of our parser type:

type JsonArrayParser<'a> =
    JsonArrayParser of (JToken[] -> ('a * JToken[]) list

The parser is essentially a function that takes an array of tokens as its input and returns a list of pairs: an object of some type 'a and an input array. In case of success, the result is a singleton list, otherwise it is an empty list. (List is used here for simplicity, we could define a discriminated union with two value constructors for either case or use a standard Option<'a> type).

Let’s add some auxiliary code:

// running our parser
let runParser (JsonArrayParser p) tokens =
    p tokens
exception JsonParserException of string
// getting the parsing result
let getParsingResult = function
    | [] -> raise <| JsonParserException("Parsing failed")
    | [(res, _)] -> res
    | _ -> raise <| System.NotSupportedException("This should not normally happen")
// and a combinations of these functions
let parse parser tokens =
    runParser parser tokens
        |> getParsingResult

It is time now to define the Computation Expression builder:

type JsonArrayBuilder() =
    member this.Return(x) =
        JsonArrayParser (fun ts -> [(x, ts)])
    member this.Bind(p, f) =
        JsonArrayParser (fun ts ->
            match runParser p ts with
            | [(r, ts')] -> runParser (f r) ts'
            | _ -> []
        )
let jarr = JsonArrayBuilder()

That’s all!

• Return – well… returns a given value;
• Bind – given a parser A creates a new parser B that, depending on the result of A either calls a function to a A’s result or returns an empty list as a sign of failure.

Frankly speaking, this is not enough. We would also need some simple parsers:

let at idx =
    JsonArrayParser (fun ts ->
        if ts.Length <> 0 &amp;&amp; idx < ts.Length then
            [(ts.[idx]), ts]
        else
            []
    )
let tokenAtAs idx converter =
    jarr {
        let! t = at idx
        return (converter t)
    }
let (<@>) converter idx =
    tokenAtAs idx converter

Here is how we are now able to write our JSON processing:

let createPerson (ts: JToken[]) =
    jarr {
        let! id = int <@> 0
        let! name = string <@> 1
        let! lastName = string <@> 2
        let! url = string <@> 3
        return (id, name, lastName, url)
    }
// simple test
let tokens = JArray.Parse("[1,2]") |> Seq.toArray
let me = parse createPerson tokens

Finally we are at the point where we started and again we are missing support for optional fields. This should not be a big deal this time however, since every piece of data processing is a parser – what we need is just another parser, or more precisely a parser combinator (a thing that gets parser as its input and returns a new parser back) that will return a default value if initial parser fails.

let parseOrDefault parser defaultValue =
    JsonArrayParser (fun ts ->
        match runParser parser ts with
        | [] -> [(defaultValue, ts)]
        | res -> res
    )
let (<|>) = parseOrDefault

Voilà! Support for optionals is implemented:

let createPerson (ts:JToken[]) =
    // ...
    let! url = string <@> 3 <|> String.Empty
    // ...

Please note the fact that thanks to operator priority we don’t need to use parentheses.

A little more fun

Lets imagine that sometimes parsing of a field depends on some external condition, for example, if id field is greater than 1000 then URL will not be present and we should use some default value instead.

let parseOrFail condition parser =
    JsonArrayParser (fun ts ->
        if condition then
            runParser parser ts
        else
            []
    )
let (<?>) = parseOrFail

As a result of applying <?> to any parser A we get a new one – B – that, depending on a condition, does exactly what A does or returns empty list without even running A.

let createPerson (ts: JToken[]) =
    jarr {
        let! id = int <@> 0
        let! name = string <@> 1
        let! lastName = string <@> 2
        let! url = (id > 1000) <?> (string <@> 3) <|> String.Default
        return (id, name, lastName, url)
    }

See how using <?> and <|> in the same statement resembles usage of standard ternary operator ?:.

Conclusion

In the end we have a declarative way of processing JSON tokens and a simple parser framework capable of performing any kind of parsing task that may occur. As a next step we can implement a similar framework for processing JSON objects, with the only difference being the usage of field names instead of indices.

However, there is a small lifehack you can employ that will get you there (to some extent) provided you buy into the idea of using multiple computers for development. Why multiple computers? Well, the assumption is that some stuff needs to run ‘elsewhere’ to prevent interference with normal operation of the machine you’re developing on.

The Solution

So here’s how I do it. Let’s say I have a solution that I want to test continuously, i.e. I want the tests rebuilt and run so frequently that I always get that ‘dashboard’ feeling of knowing what’s going on. Here’s what I do to get this.

1. Place the source code in synchronized cloud storage

First, stick your code in cloud storage that syncs with many machines. DropBox is great – it gives you 2Gb free space, and syncs flawlessly on all your machines.

2. Optimize test-runner machine performance

Ensure that the machine that builds the tests and runs them is optimized for just that. Do the lightest possible OS install (e.g., Windows XP). Also, ensure disk access is lightning-fast: I swap to RAM, and you can’t really get any better speed than this. Seeing how most developers have 8-16Gb RAM nowadays, allocating a bit of that and fitting a solution in there isn’t such a big problem.

3. Set up the scheduler to build your solution

Pick a good time interval and set the Windows scheduler to run your build script. This will result in your solution being rebuilt over and over and over. If you followed my advice about keeping the solution in RAM (which, IMHO, totally fits with the cloud storage practice), builds should be somewhat faster than builds from your Raptor or whatnot. Don’t forget to fiddle MSBuild to do its multithreading right.

Of course, if you have a cloud build solution (IncrediBuild, Electric Cloud, etc.), just use that.

4. Set up the test runner to rerun all tests whenever binaries change

That’s the easy part – get your runner to rerun tests whenever your build artifacts change. That way, you will be constantly informed about how many tests you caused to fail. Naturally, this implies you have a separate screen that shows you tests and allows you to navigate them. To be able to navigate tests on a different computer, just use Synergy.

Discussion

The approach I describe above is workable, but it’s far from perfect. Here are some issues you will encounter with it.

First, there’s no way to avoid rebuilding the whole solution, because currently no software is able to intelligently detect which systems a particular tests affects. And even if you could, it would still not work because, currently, we simply have a blanket call for a single build script whatever happens, whereas to have smarter builds we would need to specify explicitly which test assemblies need to be rebuilt.

Another problem is that a solution like DropBox does not sync with our building, which sometimes causes problems when we try to do a build while the file is being updated. Because we cannot reasonably predict when DropBox will be syncing files, we just have to hope there’s no collision. Also, on a side note, I don’t recommend sharing DropBox between developers: use ordinary source control for that. If you try to use DropBox as source control, developers will be constantly interrupting each other’s work, causing uncompilable code, and so on. In other words: the solution I present currently works best for just one developer.

Finally, if you’re into things like coverage, you’re somewhat out of luck: you won’t be able to view coverage inside on your development machine because coverage calculations have actually happened elsewhere. Of course, nothing prevents you from running code coverage analysis and simply presenting aggregated results, but those are more or less useless without you actually being able to tell which spots your tests seem to miss.

Source code (VS10 solution): https://bitbucket.org/nesteruk/dynamicprototyping

The Problem

Software development tasks come in various sizes – some huge, some tiny. Depending on the functionality you’re trying to implement, a task may take a minute or a month. The minute-length tasks are very annoying, because their verification (let’s forget about unit tests for a moment) takes longer than the correction itself. They are even more annoying because they break my concentration.

As a result of this dissatisfaction, I started probing around the idea of getting rid of the roundtrip associated with recompile cycles. There are, essentially, just two options on how to handle this.

The first option is to rely on the DLR for code segments that need to be modified. However, this approach results in code that’s slower, and also gets us to use strange languages that we might not be willing to use in the first place.

The second option, and the one I favor, is in-process compilation. Since every .Net program has a capacity to compile other .Net programs from within, it is realistically possible to be able to modify a running program and – what’s more important – to be able to immediately see the results.

Basic Infrastructural Concerns

Since a diligent developer might instantly find several issues with this approach, I’m going to address some obvious concerns beforehand.

Speed

The first issue to address is: will the program be fundamentally slower because of this? Convention wisdom suggests that it will, quite simply because many of my readers are already thinking of cross-domain marshaling and things like that. In fact, you may have noticed that at no point in time did I suggest unloading assemblies. In actual fact, this is not necessary in order to get things running.

Why not? Well, mainly because nothing prevents you from loading several copies of the same assembly. Sure, you pay the price for that, and some additional fiddling is necessary, but the net result is that

• The assembly you change is compiled and loaded immediately after you’re done with it
• The newly loaded assembly is immediately put in use in lieu of the previous version

Of course, prudent planning and thought-out development practices are required to guarantee that things don’t go wrong when loading the new assembly and using it. We’ll talk about those in a while.

Roundtrip and Shipping

So the next concern is somewhat multi-faceted. Namely, the following questions come to mind:

1. How do you both package the source code and compile it into the program at the same time?
2. How do you ensure that in-memory changes actually ‘return’ back to the program?
3. How do you ensure that none of this hackery ever gets to the deployed program?

Let me address these concerns in turn:

First, the source code is both compiled and embedded in the application. As a result, you already have a binary representation to work with, plus a textual representation that you can also use.

Second, in-memory changes return to the program because, realistically, there’s nothing preventing you from serializing some text to ..\..\SomeClass.cs. Your breakpoints (hint!) will be broken, of course, but apart from that, everything will function normally.

Third, the proposed infrastructure is intended (not saying that’s what you should do) to be Debug-only. Realistically, though, it doesn’t have to be – you can persist the program’s source code somewhere. Of course, you can also encrypt it so that it’s not human-readable.

Just to expand on that last point, the idea is that source code is typically available in the solution folder, and is not packaged in any way. Rather, it is simply kept where it needs to be – in the project directory! Now it should (hopefully) be obvious as to why there’s no real roundtrip concern – because we’re using the original program files for implementation!

Architectural Change

Okay, so you might imagine that a new assembly loaded with the same types as before would require re-wiring every necessary type in the IoC container, and you’re certainly correct – that’s how it’s done. However, this requirement is in line with best practices anyway, and arguing against it is like saying we shouldn’t brush our teeth.

Implementation

So, time to write some code. For the purposes of this article, I’ll present a minimal implementation of a program which is used for HTML encoding. The idea is that the HTML-encoding part of the program is an editable service, for which we’ll create provisions.

Default Implementation

Okay, so let’s say I have some interface called IConverter:

public interface IConverter
{
  string Convert(string source);
}

And subsequently I create my (minimal) implementation for HTML encoding:

public class HtmlConverter : IConverter
{
  public string Convert(string source)
  {
    var sb = new StringBuilder(source);
    sb.Replace("<", "&lt;");
    return sb.ToString();
  }
}

Now, being a good boy, I inject an IConverter into my main window, and use it to perform the conversion:

public partial class MainFrame : Form
{
  private IConverter converter;
  public MainFrame(IConverter converter)
  {
    this.converter = converter;
    InitializeComponent();
  }
  private void btnConvert_Click(object sender, EventArgs e)
  {
    tbConverted.Text = converter.Convert(tbSource.Text);
  }
}

Finally, I configure the container…

static void Main()
{
  var uc = new UnityContainer();
  uc.RegisterType<IConverter, HtmlConverter>();
  
  Application.EnableVisualStyles();
  Application.SetCompatibleTextRenderingDefault(false);
  Application.Run(uc.Resolve<MainFrame>());
}

And the application is usable:

Showing the Editor

Okay, so for the purposes of this exercise, I’m going to assume (meaning I’m not going to implement this just yet) that everything related to dynamic prototyping is either wrapped in #if DEBUG or has a [Conditional("DEBUG")] in front of this. And with this in mind, we can show the editor.

If you’re wondering as to how we figured out which file to open, the answer is simple: typically, you would need to write debug-specific code to actually show this window, so you may as well hardcode the file name. If you want a more industrial-strength solution, feel free to create your own infrastructure with nice little menus letting you specify the type/file to edit.

Rebuilding the Type

Okay, so we got an editor that lets us edit a file right from the solution and, naturally, lets us persist it back to where it came from. What next? Oh, we still have to compile and load the thing. Let’s deal with compilation first.

Compilation is actually easy, the only caveat being that you need to have a reference to your original executable and any related DLLs. Here’s an example:

public class InProcessCompiler
{
  public object CompileAndInstantiate(string sourceCode)
  {
    var ps = new CompilerParameters {GenerateInMemory = true, GenerateExecutable = false};
    ps.ReferencedAssemblies.Add("System.Drawing.dll");
    ps.ReferencedAssemblies.Add("System.Core.dll");
    ps.ReferencedAssemblies.Add("DynamicPrototypingSample.exe");
    var po = new Dictionary<string, string> {{"CompilerVersion", "v4.0"}};
    var p = new CSharpCodeProvider(po);
    var results = p.CompileAssemblyFromSource(ps, new[] {sourceCode});
    if (results.Errors.HasErrors)
    {
      var sb = new StringBuilder();
      foreach (var e in results.Errors)
        sb.AppendLine(e.ToString());
      throw new Exception(sb.ToString());
    }
    var ass = results.CompiledAssembly;
    var mainType = ass.GetTypes()[0];
    return Activator.CreateInstance(mainType);
  }
}

Plenty of hackery up above – specifying the explicit DLLs and our EXE to add as references, compiling source code under 4.0 and, last but not least – creating the first of all available types. Actually, that last point is prudent because in this case I’m only interested in compiling one type, and that’s the type found first in the generated assembly.

Final Touches

Having created an instance of the generated type, we do one simple thing: assign it to the variable. Thus, the whole process becomes as follows:

• Throw the editing UI
• Try compile the type
• If successful, save the new instance
• Overwrite the .cs file with new source

Here’s the event handler for our edit button:

private void btnEditProgram_Click(object sender, EventArgs e)
{
  string path = @"..\..\HtmlConverter.cs";
  var ce = new CodeEditor(path);
  if (ce.ShowDialog() == DialogResult.OK)
  {
    try
    {
      converter = (IConverter) Compiler.CompileAndInstantiate(ce.SourceCode);
      File.WriteAllText(path, ce.SourceCode, Encoding.UTF8);
    } 
    catch (Exception ex)
    {
      MessageBox.Show(ex.ToString());
    }
  }
}

And that’s pretty much it!

Conclusion

Okay, so as you can see, the implementation is not too complex. Of course, it can be ‘pruned’ so that the instrumentation of this in-process editing is only available in Debug code. And, by giving more time to the way you handle the compilation and ‘uptake’ of the code, you can make your prototyping development a little easier.

One useful thing which I do (but cannot show in this article) is using a commercial control for C# editing instead of just editing it as plain text. Apart from that, the set-up I use is pretty much like I described here.

That’s it! Thanks for reading! If you liked this article, please vote for it on CodeProject.

What Was

Before we got continuations in the Async CTP (download it now if you haven’t – it’s a non-breaking incremental update), barring doing our own BeginXxx/EndXxx-related management we had two legitimate and somewhat understandable ways of doing continuations.

The first option was to use F# async workflows. The idea itself is rather primitive – an F# workflow is simply a named scope within which the behavior of certain operators – such as let! or return! is redefined to do something related specifically to that workflow. In the case of F#, we had a mechanism for defining our own Begin and End pairs…

type WebRequest with
  member x.GetResponseAsync() =
    Async.BuildPrimitive(x.BeginGetResponse, x.EndGetResponse)

…and subsequently within an async workflow we could use the let! and return! operators to wait for an operation to complete.

let private DownloadPage(url:string) =
  async {
    try
      let r = WebRequest.Create(url)
      let! resp = r.GetResponseAsync() // wait for the result
      use stream = resp.GetResponseStream()
      use reader = new StreamReader(stream)
      let html = reader.ReadToEnd()
      use fs = new FileStream(@"c:\temp\file.htm", FileMode.Create,
                              FileAccess.Write, FileShare.None, 1024, true);
      let bytes = Encoding.UTF8.GetBytes(html);
      do! fs.AsyncWrite(bytes, 0, bytes.Length) // wait while everything is written
    with
      | :? WebException -> ()
  }
 
Async.RunSynchronously(DownloadPage("http://devtalk.net"))

The second option for managing callbacks was given to us by Jeffrey Richet’s AsyncEnumerator (part of the PowerThreading library). The trick here is to use iterators and the yield keyword to ‘step out’ of a method, thereby instrumenting an artificial continuation. This option, while innovative, does have its shortcomings, such as an inability to work within try-catch blocks.

public IEnumerator<int> DownloadPage(string url, AsyncEnumerator ae)
{
  var wr = WebRequest.Create(url);
  wr.BeginGetResponse(ae.End(), null);
  yield return 1;
  var resp = wr.EndGetResponse(ae.DequeueAsyncResult());
  var stream = resp.GetResponseStream();
  var reader = new StreamReader(stream);
  string html = reader.ReadToEnd();
  using (var fs = new FileStream(@"c:\temp\file.htm", FileMode.Create,
                      FileAccess.Write, FileShare.None, 1024, true))
  {
    var bytes = Encoding.UTF8.GetBytes(html);
    fs.BeginWrite(bytes, 0, bytes.Length, ae.End(), null);
    yield return 1;
    fs.EndWrite(ae.DequeueAsyncResult());
  }
}

In fact, from what we heard on the PDC (and beforehand) it was Jeffrey’s consulting that helped Microsoft come up with what we now got as Async CTP.

What Is

So, we now have support for continuations in C# and VB, though the mechanisms are different. In actual fact, everything is painfully easy.

First of all, we got an ability to “await” something, and that’s what the new await keyword is for. As a parameter, it takes a Task, and simply waits for the task (beneath it is a TaskAwaiter – just generate a dependency diagram to explore this stuff). The real upside here is that through the Task-related infrastructure we can manage cancelations and exceptions better. And, of course, we’re already used to tasks since they’re part of TPL.

Second, we can now do async calls not via Begin-End pairs, but by simply marking the method as async and awaiting it. This saves us from IAsyncResult-related headaches. Now, all that is required to make an asynchronous method is to use the async keyword and change its return type to Task (more on that later).

By the way, the async and await keywords are intertwined – if a method uses await, then it must be marked as async.

Okay, so let’s look at a ‘live’ example with the same old scenario – downloading a webpage and saving it to a local file.

public static async Task DownloadInformation(int id)
{
  Contract.Requires(id >= 0);
  
  string filename = @"c:\temp\somesite.com\" + id + ".htm";
  if (File.Exists(filename))
    return;
  var ct = new CancellationToken();
  var r = new Reporter(id);
  string data = await new WebClient().DownloadStringTaskAsync(
    new Uri(@"http://somesite.com/" + id), ct, r);
  
  using (var fs = new FileStream(filename, FileMode.CreateNew, FileAccess.Write, FileShare.Write))
  {
    var bytes = Encoding.Default.GetBytes(data);
    await fs.WriteAsync(bytes, 0, bytes.Length);
    fs.Close();
  }
}

The first paradox of perception is that even though our method doesn’t return anything (can you see a return anywhere?), its return type is not void but rather Task. This is the Async CTP trick, though I’m not sure how adequate it is (c’mon guys, you’re breaking the natural match between a method’s return type that that which is actually returned). At any rate, the idea is that the method which uses await must be marked as async and must also have a return type of Task (if it is void) and, predictably enough, Task<T> if the method returns a T.

Unfortunately, out/ref parameters are not supported, which will cause a small outcry because, as you all know, working with tuples in C# is not exactly easy.

So… in addition to the typical methods, the Async CTP adds a DownloadStringTaskAsync() method to the WebClient, which is a pretty good hint as to what will happen there (though it can be confusing because WebClient already has a DownloadStringAsync() method). In addition, for example, the FileStream class gets a WriteAsync() method (where is the Task bit?), though my opinion is that it would be a lot more convenient if they added (hey, I’m sure they will) a File.WriteAllTextAsync() so I don’t have to bother with all the extra initialization. At any rate, both methods are usable with await.

Oh, speaking of which, one useful thing which I liked is the ability to use an IProgress<> implementation as a parameter to DownloadStringTaskAsync(). We often need to track the completion of a file download, and having this built-in is just great. I’m using it in the code above, with the definition of the Reporter class as follows:

private class Reporter : IProgress<DownloadProgressChangedEventArgs>
{
  private int id;
  public Reporter(int id)
  {
    this.id = id;
  }
  public void Report(DownloadProgressChangedEventArgs value)
  {
    Console.WriteLine("{0}:{1}", id, 
      string.Empty.PadRight(value.ProgressPercentage / 10, '*'));
  }
}

Now, here’s my way of invoking the above code:

static async void Main()
{
  var ct = new CancellationTokenSource();
  Console.CancelKeyPress += (s, e) => ct.Cancel();
  try
  {
    var po = new ParallelOptions();
    po.MaxDegreeOfParallelism = 64;
    po.CancellationToken = ct.Token;
    Parallel.For(1, 100, po, i =>
    {
      po.CancellationToken.ThrowIfCancellationRequested();
      DownloadInformation(i);
    });
  } 
  catch (Exception e)
  {
    Console.WriteLine(e);
  }
  finally
  {
    Console.ReadKey();
  }
}

What’s of note here is that we also had to mark Main() as async – there is a viral nature to async calls!

And now for the things I didn’t like. First, I’m as yet uncertain about what to do if the URL of the file to be downloaded doesn’t resolve to anything (i.e., page doesn’t exist). Right now, everything just stops and I get no exception or other indication of what’s going on. Another thing that worries me is that Debugging doesn’t work for me yet – right after the first await it bounces me to the caller, and that’s it. Then again, this is a CTP, so I guess I have to be patient.

Now What?

So now we got ourselves the ability to ‘lighten up’ on syntax, and code readbility has grown even compared to F#. We can now describe algorithms with continuations though, predictably enough, it makes little sense for singular calls – it’s mainly benefitial if you are making lots of these, either via Parallel.For() or, for example, by creating several tasks and then doing Task.WaitAll().

Since the Async CTP came out, the C# mailing lists have exploded – everyone’s experimenting with the ‘kit’ and trying things out. I actually put the code above into ‘live’ use – the risk is minimal and everything is working well, so I can afford it. The code above has already worked for a few hours, and I have no complaints so far. Then again, this use-case is fairly trivial.

Other PDC Goodies

It would be nice if Anders mentioned something mind-blowing, but instead I was somewhat surprised even that while talking about metaprogramming in C#, when asked about aspect-oriented programming, he said he wasn’t sure it’s a good idea and that C# vNext is unlikely to support it. I find this bizarre because my understanding of metaprogramming is having a capability to do exactly that. Of course, the differentiator here is that, e.g., PostSharp’s approach is to post-process existing assemblies, which might not be what the ‘compiler-as-a-service’ is attuned for.

Then again, I don’t need metaprogramming for AOP – I need it to auto-implement INotifyPropertyChanged, generatively create complex data structures or help create internal DSLs for better descriptions of algorithms. And C# vNext should be perfect for that. Unless of course I get the temptation to migrate everything to JetBrains MPS. Hey, it could happen. ■

Composite Extension Method

This type of method is created when you need to call two (or more) methods together, and you want to simplify the call. There’s a myriad of situations where this is needed – for example, the StringBuilder class has an AppendFormat() and an AppendLine(), but what if you want to output a formatted string with a line break? A composite extension method helps you do just that.

public static StringBuilder AppendFormatLine(
  this StringBuilder sb, string format, params object[] args)
{
  return sb.AppendFormat(format, args).AppendLine();
}

In actual fact, there are many use-cases for method combinations, but they typical one is sequential call (i.e., call A() then B()).

Recursive Extension Method

This type of method helps you avoid recursive calls by wrapping the ‘folding’ into an extension method. For example, the Path.Combine() method takes only two parameters, which isn’t very convenient if you have more than two elements in a path.

// we have to write this
Path.Combine(a, Path.Combine(b, c));
// but we _want_ to write this
new[]{a, b, c}.PathCombine();

Params Extension Method

Since in .Net T[] is not equivalent to params T[], we can’t simply make a bunch of T arguments and pass them arbitrarily to a method that takes an array. The solution here is to create extenion methods which take a params array and proxy the calls then and there. For example, we can define the following:

public static void AddRange<T>(this IList<T> list, params T[] objects)
{
  foreach (T obj in objects)
    list.Add(obj);
}

And here’s how one would use it:

var list = new List<int>();
// previously, you would have to write this
list.AddRange(new[] {1, 2, 3});
// but now you can write this
list.AddRange(1, 2, 3);

Anti-static Extension Method

Instead of calling a static method of class X, we instead call an extension method on this class (if this is possible, that is). The classical example to this approach is the way String.Format() gets turned into an extension method:

public static string ƒ(this string format, params object[] args)
{
  return string.Format(format, args);
}

Are you spooked by the curly f?

Factory Extension Method

These methods let you completely bypass the object constructor, hiding it behind more concise, intention-revealing methods. One popular example is the creation of a date from a fluent expression so that one can write 19.June(1986). Note that the name of the method itself is an implicit parameter used for the creation of the DateTime structure:

public static DateTime June(this int day, int year)
{
  return new DateTime(year, 6, day);
}

The factory method is not obliged to attach this method to the first parameter’s type. For example, if your goal is the creation of an object (say, from a line of Xml) from a key-value structure, you can pass the whle object. We just need to remind ourselves that C# supports anonymous types. Then, the creation of an arbitrary piece of Xml can look like this:

new { Name="Dmitri", Age=25}.ToXElements();
// will give us <Name>Dmitri</Name> and <Age>25</Age>

I won’t show the implementation of this method here – you can find it in MiscUtil – but I hope this is a good reminder to our readers that an anonymous class is de facto a key-value collection. And hey – Asp.Net MVC happily exploits this language feature.

Monadic Extension Method

Monadic extension methods let you implement what I call monadic syntax. My previous article shows fairly clear examples of what this is and what it’s for, but I’ll mention it here too for posterity – here’s an extension method that allows chain calls that only ‘happen’ if the argument is non-null:

public static TResult With<TInput, TResult>(
  this TInput o, Func<TInput, TResult> evaluator)
  where TResult : class where TInput : class
{
  if (o == null) return null;
  return evaluator(o);
}

I’d like to point out here that monadic extension methods are typically useful in domain areas where F# is more useful than C#. And, as you may remember, F# has not just extension methods but also extension properties, which adds an extra aspect usability-wise… but at the expense of all the headaches you’ll get related to immutability and other capricious behavior.

Proxy Extension Method

Sometimes, in order to create a fully-fledged fluent interface, it’s useful to keep some state associated with an object. Since in the conditions of staticity (everything is static) persistence is very difficult, we use a proxy object which can act as some sort of configurator or ‘smart parameter’ for something else. Besides that, it can implement a host of methods, which will not pollute the global scope were they extensions over object.

Here’s an exampe – SomeProxy<T> which acts as a convertible wrapper for T:

public class SomeProxy<T>
{
  private readonly T subject;
  private object something;
  internal SomeProxy(T subject, object something)
  {
    this.subject = subject;
    this.something = something;
  }
  public static implicit operator T(SomeProxy<T> obj)
  {
    return obj.subject;
  }
  // other useful operators here
}

This class can then be ‘substituted’ in lieu of the original object with an extension method:

public static SomeProxy<T> Configure<T>(this T subject, object something)
{
  return new SomeProxy<T>(subject, something);
}

Unfortunately, you won’t be able to ‘layer’ these mixin-like constructs over one another without having to perform casts.

Functionality Extension Method

This is the catch-all type for any type of extension method. A typical example is the (yet missing) ForEach() method for Linq, which Eric Lippert blogged about. It’s an interesting read – mainly concerned with immutability that everyone seems to be obsessing over these days.

The internet is positively teeming with various mini-libraries of functionality extension methods – not just ForEach(), but also really bizarre constructs that are often, upon close ovservation, very obscure. For example, the Step() method which is simply a for-loop replacement. You would use it like this:

// all numbers from 5 to 9 with a step of 2 are written to the console
5.Step (9, 2, i => Console.WriteLine (i));

Conclusion

I’m not sure if it’s fully valid to speak of ‘extension method patterns’ in the same architectural sense as, say, GoF design patterns, but… it’s something that caught my eye, and I guess putting it down on paper (so to speak) helps me better spot and classify these when I see them.

As soon as software systems become reasonably complex, it gets very hard to organize things. We’re lucky to have sorted out the problem of component creation (with Inversion of Control and Dependency Injection), but when it comes to reacting to events, the framework give us very few organizational hints. As a consequence, people naturally create various infrastructural solutions for organized event handling. This is what we’re going to discuss in this article.

 Get Source Code Here

Problem statement

For the vast majority of developers, the task of handling events either doesn’t exist, or exists ‘in full force’ but in a very predictable key – for example when someone handles a WinForms button_Click event.

That’s all well and good, but as soon as as things get complex, things break down. Here’s an (albeit synthetic) example – let’s try modeling the interaction between a football coach and player:

public class FootballPlayer
{
  public string Name { get;set; }
  public void Score()
  {
    var e = Scored;
    if (e != null) e(this, new EventArgs());
  }
  public event EventHandler Scored;
}
public class FootballCoach
{
  public void Watch(FootballPlayer p)
  {
    p.Scored += (_, __) =>
    {
      Console.WriteLine("I'm happy that {0} scored", p.Name);
    };
  }
}

In the above example, subscription and notification works just fine. But what if we do this?

var p2 = c.Clone(); // deep copy :)
p.Score();

The issue here is that when you copy an object (either via MemberwiseClone() or via deep copy using, say, BinaryFormatter), all subscriptions will be lost.

Now, truth be told, subscriptions can be restored by hand… or, I don’t know, you can try to abuse Expression<T> and the like. But that’s really only part of the problem.

The next issue is that, one day, you’re going to wish that your objects subscribed to events automatically. For example, as soon as a football player walks out onto the field, the coach starts observing him. Same goes for unsubscribing, too. As a consequence, you might end up with the following:

class FootballCoach
{
  public FootballCoach(FootballPlayer[] players)
  {
    foreach (var p in players) {
      p.EntersField += new EventHandler(StartWatchingPlayer);
      p.LeavesField += new EventHandler(StopWatchingPlayer);
    }
  }
}

And so on – in each StartXxx you would have to subscribe, and in each EndXxx you would unsubscribe. But even this is not all.

Okay, so now imagine that your system contains many such objects. And they all send messages to one another. If we try to manage the subscriptions with the += operator (as we’re used to), we’ll get really crazy coupling as well as virtually untestable code (event testing is a pain in any case).

And finally, let us not forget the whole ‘configuration’ issue. Because sometimes it makes sense to receive events of a specific type regardless of who sent them. For example, the coach doesn’t really care who is swearing at him – a player, the other coach, or the referee. Also, beleive it or not, sometimes it’s beneficial to perform various manipulations related to event handling, such as batching events in blocks of 100 and handling them only on Friday 13^th during a full moon. As you may have guessed, the simple event infrastructure is poorly suited to handle these sorts of demands.

Simple Event Broker

So given the requirements above, a sane developer would start writing something called an Event Arggregator or Event Broker – a separate component that can help us manage the madness of events. It all starts with a class similar to the following:

public class EventBroker
{
  private MultiDictionary<string, Delegate> subscriptions =
    new MultiDictionary<string, Delegate>(true);
  public void Publish<T>(string name, object sender, T args)
  {
    foreach (var h in subscriptions[name])
      h.DynamicInvoke(sender, args);
  }
  public void Subscribe<T>(string name, Delegate handler)
  {
    subscriptions.Add(name, handler);
  }
}

Thread safety notwithstanding, we got ourselves a broker that can manage subscriptions. Of course, you’re not getting any QoS (Quality of Service) here, and any aggregation/transformation logic related to events has to be done by hand, but we’re already improving on the original situation – for example, by using name as the classifier above, we got ourselves a situation in which one class can subscribe a handler for several different events.

So, with this implemented, our football player doesn’t publish events anymore:

public class FootballPlayer
{
  private readonly EventBroker broker;
  public string Name { get; set; }
  public FootballPlayer(EventBroker broker)
  {
    this.broker = broker;
  }
  public void Injured()
  {
    broker.Publish("LeavingField", this, new EventArgs());
  }
  public void SentOff()
  { // event args can be different for this one
    broker.Publish("LeavingField", this, new EventArgs());
  }
}

And the football coach now subscribes to events via the broker:

public class FootballCoach
{
  private readonly EventBroker broker;
  public FootballCoach(EventBroker broker)
  {
    this.broker = broker;
  }
  public void Watch(FootballPlayer player)
  {
    broker.Subscribe<EventArgs>("LeavingField",
                                new EventHandler(PlayerIsLeavingField));
  }
  public void PlayerIsLeavingField(object sender, EventArgs args)
  {
    Console.WriteLine("Where are you going, {0}?",
                      (sender as FootballPlayer).Name);
  }
}

In the above, we’re relying on polymorphism and the idea that all events, being good citizens, inherit from EventArgs. Strong typing is not critical here because you can always do a type case. And here’s how you could use it all in a client app:

var uc = new UnityContainer();
uc.RegisterType<EventBroker>(
  new ContainerControlledLifetimeManager());
var p = uc.Resolve<FootballPlayer>();
p.Name = "Arshavin";
var c = uc.Resolve<FootballCoach>();
c.Watch(p);
p.Injured();
p.SentOff();

One thing that you should note is that events got effectively replaced by messaging. If, like me, you work with frameworks such as NServiceBus, this might appear for you to be a very natural metamorphosis.

In the above example, the broker is registered as a singleton, so both the the coach and the player receive the same instance. What I’m hinting at is that in static scenarios, when you know in advance who subscribes to what, it makes more sense to describe subscriptions declaratively. We’ll get to that in a moment.

Issues

The solution presented here is not final for a number of reasons. First, it’s really not a good idea to have direct coupling between the component and the broker – this implies that you need to splice the broker through domain classes and use it explicitly. As we’ll see in the next section, this problem is easy to resolve.

The second problem, which I’ve already mentioned, is that event handling logic currently only uses one criterion – a string literal that acts as a ‘classifier’. To limit event processing logic thus is somewhat unwise, especially with the pressent of such a powerful instrument as LINQ.

Reactive Extensions Event Broker

Okay, so in the previous example I described an event broker implemented with plain C# by switching from events to message passing, leaving behind the event keyword and all the problems which typically accompany it. Now, let us try to exploit a library solution. We’re going to try and build an event broker using Reactive Extensions, which should allow us to apply complex selection logic to events, and would also simplify unsubscriptions via its IDisposable tie-in.

Subscription

To start with, we’ll begin by looking at a subscription not as some KeyValuePair, but rather an IDisposable, with the understanding that the subscription only exists while the subscriber exists, and that when the subscriber dies (e.g., gets disposed), it makes sense to unsubscribe from all events it was monitoring. Consenquently, we write the simplest possible implementation:

private class Subscription : IDisposable
{
  private readonly EventBroker broker;
  public IObserver<EventArgs> Subscriber { get; private set; }
  public Subscription(EventBroker broker, IObserver<EventArgs> subscriber)
  {
    this.broker = broker;
    this.Subscriber = subscriber;
  }
  public void Dispose()
  {
    broker.Unsubscribe(Subscriber);
  }
}

This is a private class that lives inside the broker. All it does is keep a reference to the subscriber. The subscription itself is stored in the broker (coming up!) and is served to the subscriber just in case the subscriber wants to die at a later stage.

The Broker

Our new Rx-driven broker implements IObservable<T>, which lets the subscriber perform various operations right on the broker before subscribing to anything. This may sound a bit surreal, but if you think of LINQ as a generator of some proxy between all of broker’s objects and the ones that we actually need, it starts to make more sense.

So, our broker has 2 fields – the list of subscribers (yes, a list, and not a HashSet) and a ReaderWriterLockSlim just in case the broker is used from several threads.

class EventBroker : IObservable<EventArgs>
{
  private readonly List<Subscription> subscribers = new List<Subscription>();
  private readonly ReaderWriterLockSlim myLock = new ReaderWriterLockSlim();
  ...
}

The broker has three methods. The first – Subscribe() – lets anyone subscribe and manipulate with LINQ a push-collection of events that come from this broker. Each subscription forces a check on the absence of existing subscriptions, which makes messy code even more messy:

public IDisposable Subscribe(IObserver<EventArgs> subscriber)
{
  Subscription sub = new Subscription(this, subscriber);
  myLock.EnterUpgradeableReadLock();
  try
  {
    if (!subscribers.Any(s => s.Subscriber == subscriber))
    {
      myLock.EnterWriteLock();
      try
      {
        subscribers.Add(sub);
      }
      finally {
        myLock.ExitWriteLock();
      }
    }
  } finally
  {
    myLock.ExitUpgradeableReadLock();
  }
  return sub;
}

As you can see, the return value as dictated by IObservable<T> has a type of IDisposable – that’s the reason why we had to create a separate Subscription class. The purpose of this is simple – if you need to unsubscribe from the event stream, you can just call Dispose().

Here’s what unsubscribing looks like:

public void Unsubscribe(IObserver<EventArgs> subscriber)
{
  myLock.EnterWriteLock();
  try
  {
    subscribers.RemoveAll(s => s.Subscriber == subscriber);
  }
  finally
  {
    myLock.ExitWriteLock();
  }
}

Nothing unusual here. You can call this method yourself, but it’s also called by Subscription during the moment of deletion. Whichever works for you.

Finally, the third method is the one to publish something.

public void Publish<T>(T args) where T : EventArgs
{
  myLock.EnterReadLock();
  try
  {
    var subs = subscribers.ToArray();
    foreach (var s in subs)
      s.Subscriber.OnNext(args);
  }
  finally { myLock.ExitReadLock(); }
}

There is a tiny hack in the above code. Can you see where? The issue above is that OnNext() can, theoretically, cause a destruction of a subscription, utterly invalidating the subscribers list. Which is why we make a copy.

Domain Objects

So, coming back to our ‘coach and player’ example. The player, unfortunately, continues to keep a reference to the broker. It uses EventBroker.Publish() where necessary to inform everyone that he has just scored a goal:

class FootballPlayer
{
  public string Name { get; set; }
  [Dependency]
  public EventBroker EventBroker { get; set; }
  public void Score()
  {
    Console.WriteLine("{0} scored!!!", Name);
    EventBroker.Publish(new GenericEventArgs(this, Name));
  }
}

As for the coach, he also has a dependency on the broker. In fact, he may as well take it as a constructor parameter (via DI, perhaps?) and subscribe to events then and there. Of course, since we are using Rx, we are at liberty to use LINQ to filter and manipulate the event stream.

class FootballCoach
{
  private readonly EventBroker broker;
  public FootballCoach(EventBroker broker)
  {
    broker.OfType<GenericEventArgs>().Subscribe(args =>
      Console.WriteLine("Well done, {0}!", args.Data));
  }
}

Just to be complete, here is the event argument type:

class GenericEventArgs : EventArgs
{
  public GenericEventArgs(object sender, string data)
  {
    Sender = sender;
    Data = data;
  }
  public object Sender { get; set; }
  public string Data { get; set; }
}

And, after all this is done, simple wireup with Unity finishes things off:

var uc = new UnityContainer();
uc.RegisterType<EventBroker>(new ContainerControlledLifetimeManager());
var p = uc.Resolve<FootballPlayer>();
p.Name = "Arshavin";
var c = uc.Resolve<FootballCoach>();
p.Score();

What is the Point?

Some of you might think that we haven’t really achieved anything – after all, just like with the simple broker, we have to propagate the broker through our entities. Even if this wasn’t a broker but rather some Subject<T> (an Rx interface for classes that both publish and subscribe), it’s still not very neat.

On the other hand, we got one very serious advantage – the ability to use Linq combinators for complex manipulations. For example, if the coach starts noticing a player only after he scores his third goal, you could describe it like so:

broker.OfType<GenericEventArgs>().Skip(2).Take(5).Subscribe(args =>
  Console.WriteLine("Well done, {0}!", args.Data));

“But wait,” you might say, “didn’t you promise to talk about declarative subscriptions?” Indeed I did. That’s what the next section is about.

Unity 2.0 Event Broker

So, moving on with our event broker discussion, let’s take a look at the use of an event broker in static scenarios (where all subscriptions are known at compile-time). We are going to abuse the Unity Application Block to implement declarative event brokerage via attributes and extensions to the Unity container.

The example presented here is originally taken from the corresponding example in the ObjectBuilder project, but have been adapted to support Unity 2.0.

The Broker

Let’s begin with the broker itself. Broker implmenetation imples two immortal constructs – the subscriber (or event sink) and someone who publishes the events (event source). In the Unity implementation of the event broker, these participants are divided into separate classes – somewhat differs from our previous example with a single Subscription class. Why? It’s simple – with Rx, we are attached to a particular interface, whereas what we’re trying to get right here is true loose coupling. All we know is that the publisher publishes an event (yes, an event, not a message), and as for the ‘receiving’ class, we only know the method that takes the predictable (object, EventArgs) pair that we’re all used to.

So, let’s take a look at the infrastructure that we need.

EventSource

This class encapsulates information pertaining to an event source – that’s why the constructor has an EventInfo parameter. Based on the information about an event, we dynamically create and add a subscription to that event. Also, by analogy with our previous example, this class is marked as IDisposable and its Dispose() method does… guess what! That’s right – it unsubscribes all handlers from this event.

internal class EventSource : IDisposable
{
  readonly string eventID;
  readonly EventInfo eventInfo;
  readonly MethodInfo handlerMethod;
  readonly EventBroker broker;
  readonly WeakReference source;
  public EventSource(EventBroker broker,
                      object source,
                      EventInfo eventInfo,
                      string eventID)
  {
    this.broker = broker;
    this.source = new WeakReference(source);
    this.eventInfo = eventInfo;
    this.eventID = eventID;
    handlerMethod = GetType().GetMethod("SourceHandler");
    Delegate @delegate = Delegate.CreateDelegate(eventInfo.EventHandlerType, this, handlerMethod);
    eventInfo.AddEventHandler(source, @delegate);
  }
  public object Source
  {
    get { return source.Target; }
  }
  public void Dispose()
  {
    object sourceObj = source.Target;
    if (sourceObj != null)
    {
      Delegate @delegate = Delegate.CreateDelegate(eventInfo.EventHandlerType, this, handlerMethod);
      eventInfo.RemoveEventHandler(sourceObj, @delegate);
    }
  }
  public void SourceHandler(object sender,
                            EventArgs e)
  {
    broker.Fire(eventID, sender, e);
  }
}

EventSink

This class represents an event ‘receiver’. It does two things – firstly, it ‘registers’ a relationship between a particular event and its handler – that’s why the constructor has a MethodInfo parameter. The constructor also creates an EventArgs type which is later passed to the handler.

The second thing the event sink is concerned with is the creation and invocation of the delegate that’s responsible for the handling of a particular event.

internal class EventSink
{
  readonly Type handlerEventArgsType;
  readonly MethodInfo methodInfo;
  readonly WeakReference sink;
  public EventSink(object sink,
                    MethodInfo methodInfo)
  {
    this.sink = new WeakReference(sink);
    this.methodInfo = methodInfo;
    ParameterInfo[] parameters = methodInfo.GetParameters();
    if (parameters.Length != 2 || !typeof(EventArgs).IsAssignableFrom(parameters[1].ParameterType))
      throw new ArgumentException("Method does not appear to be a valid event handler", "methodInfo");
    handlerEventArgsType = typeof(EventHandler<>).MakeGenericType(parameters[1].ParameterType);
  }
  public object Sink
  {
    get { return sink.Target; }
  }
  public Exception Invoke(object sender,
                          EventArgs e)
  {
    object sinkObject = sink.Target;
    try
    {
      if (sinkObject != null)
      {
        Delegate @delegate = Delegate.CreateDelegate(handlerEventArgsType, sinkObject, methodInfo);
        @delegate.DynamicInvoke(sender, e);
      }
      return null;
    }
    catch (TargetInvocationException ex)
    {
      return ex.InnerException;
    }
  }
}

Extending Unity

It won’t be a surprise to anyone that Unity can, in fact, be extended with new features. To create your own extension, you simply inherit from UnityContainerExtension and add the strategies that implement your extension. Here is the container extension for our broker:

public class BrokerExtension : UnityContainerExtension
{
  private readonly EventBroker broker = new EventBroker();
  protected override void Initialize()
  {
    Context.Container.RegisterInstance(broker,
      new ExternallyControlledLifetimeManager());
    Context.Strategies.AddNew<BrokerReflectionStrategy>(
      UnityBuildStage.PreCreation);
    Context.Strategies.Add(new BrokerWireupStrategy(broker),
                            UnityBuildStage.Initialization);
  }
}

In this example, the Context variable gives us access to the container itself (into which we add the broker itself) as well as the collection of strategies that will be applied. For each strategy we define the stage at which the build-up happens. There are two strategies for our broker. The first – BrokerReflectionStrategy uses reflection to find out about potential publishers and subscribers:

public class BrokerReflectionStrategy : BuilderStrategy
{
  public override void PreBuildUp(IBuilderContext context)
  {
    Type typeToBuild = context.BuildKey.Type;
    if (typeToBuild != null)
    {
      var policy = new EventBrokerPolicy();
      RegisterSinks(policy, typeToBuild);
      RegisterSources(policy, typeToBuild);
      if (!policy.IsEmpty)
        context.Policies.Set<IEventBrokerPolicy>(policy, context.BuildKey);
    }
    base.PreBuildUp(context);
  }
  static void RegisterSinks(EventBrokerPolicy policy,
                            Type type)
  {
    foreach (MethodInfo method in type.GetMethods())
      foreach (SubscribesToAttribute attr in method.GetCustomAttributes(typeof(SubscribesToAttribute), true))
        policy.AddSink(method, attr.Name);
  }
  static void RegisterSources(EventBrokerPolicy policy, Type type)
  {
    foreach (EventInfo @event in type.GetEvents())
      foreach (PublishesAttribute attr in @event.GetCustomAttributes(typeof(PublishesAttribute), true))
        policy.AddSource(@event, attr.Name);
  }
}

The second – BrokerWireupStrategy – is used to actually register subscriptions:

public class BrokerWireupStrategy : BuilderStrategy
{
  private readonly EventBroker broker;
  public BrokerWireupStrategy(EventBroker broker)
  {
    this.broker = broker;
  }
  public override void PreBuildUp(IBuilderContext context)
  {
    var policy = context.Policies.Get<IEventBrokerPolicy>(context.BuildKey);
    if (policy != null && broker != null)
    {
      foreach (KeyValuePair<string, MethodInfo> kvp in policy.Sinks)
        broker.RegisterSink(context.Existing, kvp.Value, kvp.Key);
      foreach (KeyValuePair<string, EventInfo> kvp in policy.Sources)
        broker.RegisterSource(context.Existing, kvp.Value, kvp.Key);
    }
    base.PreBuildUp(context);
  }
}

A strange thing is happening here – we’re using two strategies instead of one. Also, there’s a policy class called EventBrokerPolicy that’s used here – it allows us to pass data from one strategy eo another. One of the reasons behind this is that, in the context of strategies, there’s no direct access to the container itself. In other words, the EventBrokerPolicy class is simply a DTO.

public class EventBrokerPolicy : IEventBrokerPolicy
{
  readonly Dictionary<string, MethodInfo> sinks = new Dictionary<string, MethodInfo>();
  readonly Dictionary<string, EventInfo> sources = new Dictionary<string, EventInfo>();
  public bool IsEmpty
  {
    get { return sinks.Count == 0 && sources.Count == 0; }
  }
  public IEnumerable<KeyValuePair<string, MethodInfo>> Sinks
  {
    get { return sinks; }
  }
  public IEnumerable<KeyValuePair<string, EventInfo>> Sources
  {
    get { return sources; }
  }
  public void AddSink(MethodInfo method,
                      string eventID)
  {
    sinks.Add(eventID, method);
  }
  public void AddSource(EventInfo @event,
                        string eventID)
  {
    sources.Add(eventID, @event);
  }
}

Putting It All Together

So, we’re going to get Unity to find subscriptions via Publishes and SubscribesTo attributes. In our example, we once again used a string identifier to describe events – thus, we get a form of loose coupling in the sense that a competely arbitrary component that gets plugged into the system can easily subscribe to a particular event – so long as it knows its name.

Let’s take a look at actual usage. First thing you should notice is that our coach and player classes got a lot simpler:

public class FootballPlayer
{
  [Publishes("score")]
  public event EventHandler PlayerScored;
  public string Name { get; set; }
  public void Score()
  {
    var ps = PlayerScored;
    if (ps != null)
      ps(this, new EventArgs());
  }
}

Naturally, event arguments in the above example could be a more copmlex type – we have already seen how Unity determines this type and builds a delegate based on it. As for the coach, here’s the new definition:

public class FootballCoach
{
  [SubscribesTo("score")]
  public void PlayerScored(object sender, EventArgs args)
  {
    var p = sender as FootballPlayer;
    Console.Write("Well done, {0}!", p.Name);
  }
}

In other words, we got a purely declarative event handling infrastructure. That’s it! All that remains is to actually use it!

var uc = new UnityContainer();
uc.AddNewExtension<BrokerExtension>();
var p = uc.Resolve<FootballPlayer>();
p.Name = "Maradona";
var c = uc.Resolve<FootballCoach>();
p.Score();
p.Score();

Conclusion

In this article, I have presented three distinct approaches to implementing an event broker – a simple Dictionary-based broker, a broker based on Reactive Extensions, and a declarative broker infrastructure as a Unity 2.0 extension. There are many more possible broker implementations out there – the one you choose is up to you. ■

If you’re like me, you are already fed up with people throwing the term ‘DSL’ around without showing a good example of how it’s done and where they are used – not to mention giving a decent, human-readable description of DSLs that doesn’t allude to extraneous concepts (I’m talking about things like Oslo or MPS, mainly).

Okay, what on Earth is a DSL? A DSL is a way of defining domain-specific (you could say industry-specific, but it can be much narrower) logic using English instead of a programming language. The obvious benefit is that non-technical people can edit the DSL, without worrying about curly braces, semicolons, that sort of thing.

In this article, I’m going to show how to use the F# programming language to make a simple DSL that helps with software estimation. Just so you know, it’s a real-life example, with a more complex version of the DSL being used at our company. All right, let’s hit it!

Just to say upfront, this article doesn’t present book-perfect F# code, the reason being that it doesn’t really matter since F# code is not our end-product. I can think of a dozen ways to improve the F# code presented, but, as I said, that’s not the purpose of this exercise.

Problem Statement

When someone wants software written, they typically contact a software development company with something called an RFP, short for “Request for Proposal”. Depending on the level of detail of this RFP, the code shop can either do a detailed estimate or give a ballpark figure. Barring cases of clients wanting dedicated teams or having vague requirements, the estimate the code shop makes is a fixed-price project timeline. Yeah, I know it doesn’t sound particularly agile to people, but in situations where the prospective client has done lots of up-front design, it actually makes sense.

Anyway, someone has to do the estimate, i.e. partition the project into tasks, give them duration, assign resources (= people) to tasks, define milestones, et cetera. This sort of estimate can be done in a program such as Microsoft Project – a good choice in our case, since automating Office from an F# app is easy.

So why a DSL then? Well to be honest, you can probably do an estimate at the 10^th of the time it takes to draw up the GANTT chart with all the reorderings and point-and-click mechanics of Project. Not only that, but typically you need to apply some sort of logic (i.e. use your brain, ugh!) to make sure the project plan is nice and balanced (I mean resource utilization and such). Having a DSL means you can optimize and autogenerate the plan. With a DSL, you can stick as much business logic into your planning procedure as you want – for example, if your company has a process database (this is CMMI Level 4-ish, btw), you could try validating the plan against empirical data.

Are you sold on the DSL idea for estimates? If not, here’s another boon: integration. You can integrate Project with other systems to get more value from existing data. For example, if you run Dynamics CRM, you can tweak resource pricing for a particular client in order to put better developers on the features they consider important. It all sounds very pompous and BI-ish, but that’s life of a code shop.

End Users

In most code shops, barring a few exceptions, estimates are done by project managers (PMs). These guys are sometimes techies, sometimes not, so you can’t expect them to know programming. But you can certainly expect them to be able to work with a DSL and then press some magic key to generate a project plan or otherwise involve their DSL scribblings in an estimation BI scenario.

Continuous Process Improvement

At the risk of sounding cheeky, if you are constantly working on your (say) estimation DSL, improving it and tailoring it, it’s an excellent opportunity for improving your business processes. Think about it – as a code shop, you can use idle dev resources to improve the efficiency of your business. Is this great, or what? I think it is, anyway.

Choosing a Language

With the problem out of the way, let’s think of a solution. You can certainly make a free-form DSL language and write a parser, but it’s kind of tedious. A simpler solution is to use a programming language which looks so English (no cultural bias here – feel free to use Japanese or any other language) that the end user won’t know the difference. Of course, some language syntax will creep into the DSL, but its level varies.

Popular languages for DSLs include Boo (which Ayende is trying to make popular), Ruby and F#. Boo is extremely powerful, and for cases where you need metaprogramming support (not our scenario), it’s fantastic. Ruby I know next to nothing about, so no comment. Now, F# is a popular language, and a first-class citizen of .NET infrastructure (as of VS2010 especially). So we’re going to look at making an infrastructure in which PMs can easily write project estimates.

Let me issue a small disclaimer: being a language geared towards immutability, F# looks a bit weird when used with highly mutable concepts, such as a project which can accumulate tasks or milestones. This weirdness can be easily compensated by writing the DSL data structures in C# and then using them from F#. However, for this example, I’ll use F# exclusively.

First DSL Statement

I’ll try to keep this simple. Let’s start by defining a project with a name and start date:

project "Write F# DSL Article" starts_on "16/8/2009"

The above is a completely legal F# statement. It’s basically a function call to a function called project which takes 3 parameters. The first parameter is a project name – no, you can’t avoid the quotes here unless you want to parse stuff yourself. The second parameter is a dummy – a constant whose value is unimportant; its only purpose here is to make the spec readable and BDD-esque. You’re welcome to expand the starts_on keyword into two separate parts, but I prefer not to overdo it, especially when there’s a risk of keywords creeping in. The third parameter is the project starting date as a string.

Believe it or not, the DSL uses OOP constructs in order to manage the constructed project. For example, we have a Project class which is our DSL representation of a project. It’s shown below. Skipping ahead a bit, make sure your types don’t collide with other assemblies’ types. After all, Microsoft Project assemblies might just have a Project type too.

type Project() =
  [<DefaultValue>] val mutable Name : string
  [<DefaultValue>] val mutable Resources : Resource list
  [<DefaultValue>] val mutable StartDate : DateTime
  [<DefaultValue>] val mutable Groups : Group list

I did warn about the strange syntax, did I not? The above is F#’s way of making public fields. You’ll notice also that I use list types instead of the System.Collections.Generic ones. It doesn’t really matter what you use for the DSL so long as it works.

Our DSL will support just one project which will be at ‘global scope’, so to speak:

let mutable my_project = new Project()

The naming convention is a bit ad hoc here save for the fact that, when we get to the end of the spec, we need a statement to actually do something and my_project is a nice identifier name for that action. But now, we can finally show the project statement from earlier.

let project name startskey start =
  my_project <- new Project()
  my_project.Name <- name
  my_project.Resources <- []
  my_project.Groups <- []
  my_project.StartDate <- DateTime.Parse(start)

There. I have probably revealed 90% of what DSL construction is like. You can close this article and go off exploring right now, since you already know how it all works. In the rest of this article, I’ll be showing a few F# implementation details.

Handling Lists

Work in projects is done by resources (not very gratifying, is it?). A resource is a particular person (“John”) with a particular job title (“Junior DBA”) and a particular hourly rate ($65). A project keeps references to resources via a Resource list (see, F# is human-readable). Let’s look at the definition of Resource:

type Resource() =
  [<DefaultValue>] val mutable Name : string
  [<DefaultValue>] val mutable Position : string
  [<DefaultValue>] val mutable Rate : int

I’m abusing F# once again, but at least the structure is easy to work with. Now, a resource definition is also part of our DSL, and might look as follows:

resource "Dmitri" isa "Project Manager" with_rate 140

The above statement employs the same trickery as project except that it acts on an already-existing global variable my_project:

let resource name isakey position ratekey rate =
  let r = new Resource()
  r.Name <- name
  r.Position <- position
  r.Rate <- rate
  my_project.Resources <- r :: my_project.Resources

Resources and all other lists we use end up being listed in reverse order. It’s not a problem though – we reverse them when the time comes. If you don’t like it, use List<T> instead.

Referencing with Strings

The next concept of our DSL I want to introduct is a group of tasks. A group of tasks is typically done by one person to maintain, ahem, cognitive cohesion. We define a group as follows:

group "Project Coordination" done_by "Dmitri"

To put things in context, let’s look at the Group class:

type Group() =
  [<DefaultValue>] val mutable Name : string
  [<DefaultValue>] val mutable Person : Resource
  [<DefaultValue>] val mutable Tasks : Task list

A group references a particular resource, which our DSL specifies only as a string. Problem? I don’t think so:

let group name donebytoken resource =
  let g = new Group()
  g.Name <- name
  g.Person <- my_project.Resources |> List.find(fun f -> f.Name = resource)
  g.Tasks <- []
  my_project.Groups <- g :: my_project.Groups

Notice how, unlike with LINQ, we don’t have to call Single() after searching for the right resource.

Greater Fluency

Last but not least, we define tasks. Now, isn’t it great when you can say, for example, the following:

task "PayPal Integration" takes 2 weeks

In fact, you can. This type of fluency is achieved by judiciously defining timespan constants so that their values are meaningful. For example:

let hours = 1
let hour = 1
let days = 2
let day = 2
let weeks = 3
let week = 3
let months = 4
let month = 4

The values do not matter so long as they are distinct. Now we can define a task…

type Task() =
  [<DefaultValue>] val mutable Name : string
  [<DefaultValue>] val mutable Duration : string

… and add it to the project:

let task name takestoken count timeunit =
  let t = new Task()
  t.Name <- name
  let dummy = 1 + count
  match timeunit with
  | 1 -> t.Duration <- String.Format("{0}h", count)
  | 2 -> t.Duration <- String.Format("{0}d", count)
  | 3 -> t.Duration <- String.Format("{0}wk", count)
  | 4 -> t.Duration <- String.Format("{0}mon", count)
  | _ -> raise(ArgumentException("only spans of hour(s), day(s), week(s) and month(s) are supported"))
  let g = List.head my_project.Groups
  g.Tasks <- t :: g.Tasks

Notice that for each timespan, I slightly change the way the duration is phrased so that Project is capable of eating the spec. The dummy expression above tells F# that count is an integer – I could have defined it explicitly, of course, but I’m just too lazy. Oh, by the way, notice how easy it is for us to find the current (i.e., last) task. Because we’re using F#, this task is actually first in the list, so we can just call List.head.

Generating the Project

We’ve got everything and are ready to generate the project. The following (somewhat cheesy) command does it:

prepare my_project

Now, I’m about to show you the whole prepare definition, which uses the Project API to make the, umm, project. Notice how succinct F# is:

let prepare (proj:Project) =
  let app = new ApplicationClass()
  app.Visible <- true
  let p = app.Projects.Add()
  p.Name <- proj.Name
  proj.Resources |> List.iter(fun r ->
    let r' = p.Resources.Add()
    r'.Name <- r.Position // position, not name :)
    let tables = r'.CostRateTables
    let table = tables.[1]
    table.PayRates.[1].StandardRate <- r.Rate
    table.PayRates.[1].OvertimeRate <- (r.Rate + (r.Rate >>> 1)))
  // make root task with project name
  let root = p.Tasks.Add()
  root.Name <- proj.Name
  // add groups
  proj.Groups |> List.rev |> List.iter(fun g -> 
    let t = p.Tasks.Add()
    t.Name <- g.Name
    t.OutlineLevel <- 2s
    // who is responsible for this group?
    t.ResourceNames <- g.Person.Position
    // add tasks
    let tasksInOrder = g.Tasks |> List.rev
    tasksInOrder |> List.iter(fun t' ->
        let t'' = p.Tasks.Add(t'.Name)
        t''.Duration <- t'.Duration
        t''.OutlineLevel <- 3s
        // make task follow previous
        let idx = tasksInOrder |> List.findIndex(fun f -> f.Equals(t'))
        if (idx > 0) then 
          t''.Predecessors <- Convert.ToString(t''.Index - 1)
      )
    )

Yep, we finally reverse those backward lists with List.rev – probably not the fastest operation in the world, but I don’t care. All that matters is that the script runs and gives us the results we want – resource definitions, group names and tasks which are grouped and linked. What more could a PM ask for? (Quite a bit actually, but that’s another story.)

A complete project definition can therefore look like this:

project "F# DSL Article" starts "01/01/2009"
resource "Dmitri" isa "Writer" with_rate 140
resource "Computer" isa "Dumb Machine" with_rate 0
group "DSL Popularization" done_by "Dmitri"
task "Create basic estimation DSL" takes 1 day
task "Write article" takes 1 day
task "Post article and wait for comments" takes 1 week
group "Infrastructure Support" done_by "Computer"
task "Provide VS2010 and MS Project" takes 1 day
task "Download and deploy TypograFix" takes 1 day
task "Sit idly while owner waits for comments" takes 1 week
prepare my_project

And here is the project plan after it is generated:

Conclusion

This article shows that making a DSL in F# is really simple. Of course, the thing about DSLs is they are domain-specific, so for the domain you choose you might encounter a lot more challenges. Have fun!

 Get Source Code Here

Problem Description

So, to start with, let’s look at the way to get a person’s post code (just imagine you’re working with XML or something). The code shown below does several null checks and assigns the value only if it is available.

string postCode = null;
if (person != null && person.Address != null && person.Address.PostCode != null)
{
  postCode = person.Address.PostCode.ToString();
}

What you’ve got up there is some fairly unreadable (and un-maintainable) code. Actually, we’re lucky to have all of our code fall under a single if – something that might not be possible in a more complex scenario. Let’s imagine a more complicated situation – say we need to perform some operation between the if evaluations. What do we get? That’s right – a chain of ifs.

string postCode;
if (person != null)
{
  if (HasMedicalRecord(person) && person.Address != null)
  {
    CheckAddress(person.Address);
    if (person.Address.PostCode != null)
      postCode = person.Address.PostCode.ToString();
    else
      postCode = "UNKNOWN";
  }
}

The code presented above contains a lot of excess data – for example, person.Address.PostCode is mentioned twice. There’s nothing incorrect about the code per se, it just has a bit too many symbols. To sum up, we want our code to communicate better that

• If the value is null, no further evaluations should be done; if the value is not null, then this is the value we’re going to work with
• If we perform some action, it only happens on a valid object

So what am I suggesting? I propose that we create a fluent interface that will satisfy the above conditions without any nesting. To do that, we are going to employ the Maybe monad.

For those of you who know F#, the Maybe monad will be familiar as the Option type. For C# developers, let’s just assume that you can have variable that either have Some value or no value (None). Of course, C# doesn’t directly support this none-some duality except by using null. Which is precisely why I’m proposing the chained extension solution presented below.

With

Our primary concern is to do the null checks to ‘shorten’ them so they don’t pollute our code. For that, we’ll define a With() extension method:

public static TResult With<TInput, TResult>(this TInput o, Func<TInput, TResult> evaluator)
  where TResult : class where TInput : class
{
  if (o == null) return null;
  return evaluator(o);
}

The above method can be attached to any type (because TInput is effectively object). As a parameter, this method takes a function which defines the next value in the chain. If we passed null, we get null back. Let’s rewrite our first example using this method:

string postCode = this.With(x => person)
                      .With(x => x.Address)
                      .With(x => x.PostCode);

I suppose, in the above example, we could replace Func<> with Expression<> and try to pull properties, but I’ve seen this done and the resulting code is too slow and it’s also somewhat limiting – it assumes that you’re working with just one object, whereas my Maybe chains can (and do) drag in many objects.

Return

Here comes another piece of syntactic sugar – the Return() method. This method will return the ‘current’ value just like Where() does, but in case null was passed, it will return a different value that we supply. Consider this a kind of «Where() with fallback» method.

public static TResult Return<TInput,TResult>(this TInput o, 
  Func<TInput, TResult> evaluator, TResult failureValue) where TInput: class
{
  if (o == null) return failureValue;
  return evaluator(o);
}

So let’s assume now that, with the absense of a postcode, we want to return, say, string.Empty. Here’s how:

string postCode = this.With(x => person).With(x => x.Address)
                      .Return(x => x.PostCode, string.Empty);

By the way, you could rewrite the extension method so that failureValue would also be computed via a Func<> – I am yet to meet a scenario where this is required, though. It is typically the case that we never know at which stage the chain failed (and yielded null), so the terminal Return() is typically an indicator (either true/false or null/not null).

If & Unless

Going through the call chain, you sometimes need to do checks not related to null. Theoretically, you could suspend the chain and use an if, or you could use an if in one of the delegates, but… you can simply define an If() extension method (and an Unless() if you feel like it) and plug it into the chain:

public static TInput If<TInput>(this TInput o, Func<TInput, bool> evaluator) 
  where TInput : class
{
  if (o == null) return null;
  return evaluator(o) ? o : null;
}
 
public static TInput Unless<TInput>(this TInput o, Func<TInput, bool> evaluator)
  where TInput : class
{
  if (o == null) return null;
  return evaluator(o) ? null : o;
}

Do

Seeing how we’re having a party here, let’s add yet another method that simply calls a delegate – and that’s it. Of course, this method is best used for one-line calls and not for evaluating 20-line algorithms with convoluted logic. Nevertheless, the call is quite useful in practice.

public static TInput Do<TInput>(this TInput o, Action<TInput> action) 
  where TInput: class
{
  if (o == null) return null;
  action(o);
  return o;
}

So, we’re done: we’ve got the infrastructure we need to get our post code extraction to be a bit more readable. Here is the end result:

string postCode = this.With(x => person)
    .If(x => HasMedicalRecord(x))]
    .With(x => x.Address)
    .Do(x => CheckAddress(x))
    .With(x => x.PostCode)
    .Return(x => x.ToString(), "UNKNOWN");

As you can see, the depth of nesting has fallen to zero – no more curly braces!

Discussion

I use these Maybe-monadic-chain-null-extension-methods (call them how you will) in my R2P software product. Here’s an example of real-life use of these constructs:

public override void VisitInvocationExpression(IInvocationExpression expression)
{
  base.VisitInvocationExpression(expression);
  string typeName = this.With(x => expression)
    .With(x => x.InvokedExpression)
    .With(x => x as IReferenceExpression)
    .With(x => x.Reference)
    .With(x => x.Resolve())
    .With(x => x.DeclaredElement)
    .With(x => x.GetContainingType())
    .Return(x => x.CLRName, null);
  this.If(x => Array.IndexOf(types, typeName) != -1)
    .With(x => ExpressionStatementNavigator.GetByExpression(expression))
    .Do(x =>
          {
            var suggestion = new SideEffectSuggestion(typeName);
            var highlightInfo = new HighlightingInfo(
              expression.GetDocumentRange(),
              suggestion);
            context.HighlightingInfos.Add(highlightInfo);
          });
}

I have to point out here that, at any point, you can stop the chain and start a new one. Why would you want that? Well, for example, you cannot define shared variables within the chain (unless you refactor it all to have a Dictionary<string,object>-like parameter).

By the way, quite frequently I find myself making additional, domain-specific methods to plug into this chain. For example:

public static IElement IsWithin<TContainingType>(this IElement self) 
  where TContainingType: class, IElement
{
  if (self == null) return self;
  var owner = self.GetContainingElement<TContainingType>(false);
  return owner == null ? self : null;
}

One more thing: this type of notation is actually light obfuscation because, as I’m sure you’ve guessed, each extension method call will be shown as a static method call in Reflector:

public override void VisitInvocationExpression(IInvocationExpression expression)
{
    base.VisitInvocationExpression(expression);
    string typeName = this.With<SideEffectAnalyser, IInvocationExpression>(
    delegate (SideEffectAnalyser x) {
        return expression;
    }).With<IInvocationExpression, ICSharpExpression>(delegate (IInvocationExpression x) {
        return x.InvokedExpression;
    }).With<ICSharpExpression, IReferenceExpression>(delegate (ICSharpExpression x) {
        return (x as IReferenceExpression);
    }).With<IReferenceExpression, IReference>(delegate (IReferenceExpression x) {
        return x.Reference;
    })
    ⋮
    // and so on
}

This approach is easily extensible – for example, a colleague of mine does try-catch checks in his chains, too. Hey, this is kind of like AOP, but without post-build or dynamic proxies. Oh, and the performance hit for these chains is negligible compared to if statements.

That’s it! Comments are, as always, welcome! Oh, and if you like this article, please vote for it on CodeProject.  ■