# DevTalk.net

A blog on programming and quantitative finance

with one comment

## Intro

Every once in a while we all suffer from the Not Invented Here syndrome. So do I. This time, despite of being able to use either one of available web servers or previously written ASP.NET MVC-like web serving framework, I decided to write my own.

My requirements are that the server should serve JSON, provide meaningful URLs, be easy to use (anything that you write is by design easy to use for you), it should be hostable in a simple console app, should not use reflection as ASP.NET MVC does, and it should be written in F#. Because “not-written-in-F#” is a fundamental flaw of most existing systems I know of.

## Design

Since this project is going to be a quick and dirty prototype of a future system – let’s use HttpListener as a serving mechanism. It is pretty straightforward and easy to use. Ok. What’s next?

Any kind of ‘web serving’ starts with a URL. Those who tried Django already know how URL handling is performed there. For others, here are the basics. There is a single dictionary (I mean something like System.Collections.Generic.IDictionary here) that maps a URL to the function that should handle requests to that URL. I’m oversimplifying but that’s enough for our purposes. The request handling function should get something in and return back our JSON as a string, which then will be wrapped in a HTTP response and sent back to the user agent. The only question now is what should be passed to the handler function. If we were writing a real web server, it would require us to pass the whole response context into the function, but we want some REST here, therefore the requested URL would be enough. In Django, they use regular expression match groups to obtain arguments for handler functions from the requested URL. Why not do so as well? Let’s create a simple URL mapping for clarity.

[("^/hello/([a-zA-Z]+)", helloNameHandler);]


The above means that when a URL matching the pattern is requested, we should take the URL substring containing a person’s name and pass it to the handler function helloNameHandler. Let’s not forget about the URL query string. We’ll parse it as well and pass to the handling function as a list of (string * string) pairs.

## Implementation

Enough said, lets start implementing. First of all we’ll describe all the types already discussed:

type RequestData =
// first list is url substrings, second list is query string pairs
| RequestData of string list * (string * string) list

type ResponseData =
| Json of string

type HandlerFunction = (RequestData -> ResponseData)
type DispatchEntry = (string * HandlerFunction)
type DispatchEntryEx = (Regex * HandlerFunction)


Since we’re using regular expressions for our patterns we might have some speed gain if they are compiled before usage, hence the DispatchEntryEx type.

let makeDispatcher (pattern, handler) =
let fullPattern = sprintf "%s/?.*" pattern
let rx = Regex(fullPattern, RegexOptions.Compiled)
(rx, handler)
let prepareDispatchers (table:DispatchEntry list) : DispatchEntryEx list =
List.map makeDispatcher table
module Query =
let decode : string -> string =
HttpUtility.UrlDecode

let processParams (s:string) =
if String.IsNullOrEmpty(s) then
[]
else
let s' = s.TrimStart([|'?'|])
s'.Split([|'&'|])
|> Array.map (fun pair -> pair.Split([|'='|]))
|> Array.map (fun  [|h;t|] -> (h, t))
|> Array.toList

let processPath (rx:Regex) (s:string) =
rx.Match(s).Groups.Cast<Group>()
|> Seq.skip(1) // skipping the overall group
|> Seq.map (fun g -> g.Value)
|> Seq.toList


Nothing interesting here: we are creating a dispatch table with compiled regular expressions and two functions to provide parameters to our handler functions. Now we are ready to implement a dispatch function, but first we should define its type.

The HttpListener class provides a GetContext() method of type HttpListenerContext that in turn has the Request and Response properties. Processing a request means reading from context.Request and writing something back into context.Response instance. That said, our dispatch function should have the following type: DispatchEntryEx list -> HttpListenerRequest -> HttpListenerResponse -> unit. It does not return any value, because the actual result is written directly to the response as a side effect (We could purify the approach if we wanted to, but not this time).

let dispatch (ds:DispatchEntryEx list) request response =
let (!!) = Query.decode
let absPath = !! request.Url.AbsolutePath
let queryString = !! request.Url.Query

printfn "(II) '%s' requested..." <| !! request.RawUrl

let (rx, handler) =
ds |> Seq.find (fun (r, _) -> r.IsMatch(request.Url.AbsolutePath))

let path = Query.processPath rx request.Url.AbsolutePath
let query = Query.processParams queryString

let resultBytes =
match handler (RequestData (path, query)) with
| ResponseData.Json json ->
let callback = Query.get "callback" "callback" query
json |> Encoding.UTF8.GetBytes

response.ContentLength64 <- int64 resultBytes.Length
response.OutputStream.Write(resultBytes, 0, resultBytes.Length)


Yet again, nothing complex. The function does exactly what it supposed to. First it searches for the matching handler in dispatch table, then parses the request URL path and query string, after that applies handler function and finally writes everything back to response stream.

The last thing we should implement before we could test our server, is server initialization function.

let start (prefix:string) (table:DispatchEntry list) =
let listener = new HttpListener()
listener.Start()

let tableEx = prepareDispatchers table

let requestProcessor =
let processContext (ctx:HttpListenerContext) = async {
dispatch tableEx ctx.Request ctx.Response
ctx.Response.Close()
}
async {
while true do
try
let ctx = listener.GetContext()
processContext ctx |> Async.Start
with
| ex -> printfn "(EE): %s" ex.Message
}

requestProcessor
|> ignore

{new IDisposable with
member this.Dispose () =
listener.Close()
}


Please note the usage of async and Async.Start with the processContext function. Without it the whole server will be able to only process requests one after another. HttpListener.GetContext blocks execution until the request is received, right after that we start a new processing thread with Async.Start, which returns immediately and the while loop continues to the next call of GetContext leaving context processing to a separate thread.

Another thing to note is the return type of the start function. It returns IDisposable instance which will stop the listener when disposed, so we don’t have to expose the fact that we use HttpListener to the user still having some control over listeners lifetime.

## Some improvements

The code above seems to work (at least it should), but then we realize that JSON is not the only thing we’d like to be able to serve. We would also need some binary data such as images. To support this we have to modify the return type of our handler function. Thanks to discriminated unions, it won’t require too many changes in the code.

There are actually only two places in the whole code that are subjected to change in order to support serving images. First – the ResponseData type:

type ResponseData =
| Binary of byte[]
| Json of string


And second, the dispatch function:

let dispatch (ds:DispatchEntryEx list) request response =
//...
let resultBytes =
match handler (RequestData (path, query)) with
| ResponseData.Json json -> json |> Encoding.UTF8.GetBytes
| ResponseData.Binary bytes -> bytes
// ...


That’s all! We simply added support for a new case to all places where ResponseData is used. No need to change already developed handler functions.

## Possible further improvements

Lets imagine that once this “web server” evolves into something more complex and we would need to support “areas” of site, where each area will process URLs separately so that dispatch table will become unbearably long and uncomfortable to use. F# discriminated unions can help us here as well, simply change the HandlerFunction into something like type UrlHandler = | HandlerFunction of (...) | SiteArea of (...) etc. Django also has this feature but here we don’t use dynamic types.

It is also possible to use some templating libraries (or view engines) to render responses from handler functions. This will require just another case in ResponseData type and a slight change dispatch function.

## Conclusion

In this article we developed a dead simple yet working web serving solution that can be used to prototype URL base REST services to be hosted in your application. It has not been properly tested and surely has limitations, but the development process showed that it is possible and pretty easy to implement a web serving solution in a .NET environment without using reflection or dynamic typing.

Written by Maxim Moiseev

January 17th, 2011 at 8:02 pm

Posted in FSharp