Rash thoughts about .NET, C#, F# and Dynamics NAV.

My friend Björn Rochel (@BjoernRochel) built a really cool generic model for using fakes and automocking on top of Machine.Specifications (or MSpec) called Machine.Fakes (or mfakes).

Today he released version 0.1.0.0 (get it from nuget.org) and I want to talk a bit about the build process. Björn and myself thought a tool called “Machine.Fakes” should of course be built with a tool called “FAKE – F# Make”. In order to do so I had to fix some stuff in Fake which resulted in the new Fake version 1.50.1.0.

Here are some things I fixed:

• New task DeleteDirs allows to delete multiple directories.
• New parameter for NuGet task which allows to specify dependencies.
• Bundled with docu.exe compiled against .Net 4.0.
• Fixed docu calls to run with full filenames.
• Added targetplatform, target and log switches for ILMerge task.
• Added Git.Information.getLastTag() which gets the last git tag by calling git describe.
• Added Git.Information.getCurrentHash() which gets the last current sha1.

Machine.Fakes build setup

The build script for Machine.Fakes performs the following steps:

1. It retrieves the version no. via GitHubs REST API
2. It cleans all directories from old stuff
3. It compiles the app and test projects
4. It uses MSpec to test the application
5. It merges StructureMap and StructureMap.AutoMocking into the app by using ILMerge
6. It generates the XML documentation using the .NET 4.0 version of docu.exe
7. It zips the app and the documentation files
8. It builds and deploys a Nuget package with just Machine.Fakes
9. It builds and deploys bundled Nuget packages in the following flavors:
   1. Machine.Fakes.FakeItEasy
   2. Machine.Fakes.RhinoMocks
   3. Machine.Fakes.Moq

Now you can start using this awesome project by calling:

install-package Machine.Fakes.{Flavor}

Most build tools allow to run a specific target via a special parameter in the command line. FAKE.exe doesn’t have such a parameter, but with a combination of the Run function and getBuildParamOrDefault we can achieve the same thing:

 1: // include Fake libs
 2: #I @"tools\FAKE"
 3: #r "FakeLib.dll"
 4: open Fake
 5: 
 6: // *** Define Targets ***
 7: Target "Clean" (fun () ->
 8:     trace " --- Cleaning stuff --- "
 9: )
10: 
11: Target "Build" (fun () ->
12:     trace " --- Building the app --- "
13: )
14: 
15: Target "Deploy" (fun () ->
16:     trace " --- Deploying app --- "
17: )
18: 
19: // *** Define Dependencies ***
20: "Build"  <== ["Clean"]
21: "Deploy"  <== ["Build"]
22: 
23: // *** Start Build ***
24: Run <| getBuildParamOrDefault "target" "Default"

Now we can start the Deploy target by simply calling FAKE without a target parameter and if we want to start only the Clean target we could write “tools\Fake.exe build.fsx target=Clean”.

Today I released “FAKE – F# Make” version 1.40.5.0. This release fixes some issues with CruiseControl.NET compatibility and improves the MSBuild task.

Important links:

• Project page
• Download
• HTML Documentation
• Repository

Changes for CruiseControl.NET

Daniel Nauck created a FAKE task for CC.Net. This task allows a much easier configuration of FAKE. Please download the latest CC.NET build if you want to use it.

Daniel also created a FAKE build project on his server which showed us some compatibility issues:

• In FAKE:
  • Race condition in FAKE’s trace functionality
  • Better XML tracing if something fails
  • The tracing buffer will be flushed after every build target
  • Every task has a Timeout property (Default: 5 min.)
  • Setting of environment variables fixed
  • Showing labels in NUnit is now the default
• In Docu:
  • Unhandled exception in Docu.exe
• In CC.NET:
  • Implemented support for git submodules (CCNET-1922)

@Daniel: Thank you very much for helping me on this CruiseControl.NET stuff. I really appreciate this.

MSBuild task changes

The MSBuild task in FAKE gets a sequence of project files and compiles them in the given order. This might be slow if you have lots of dependent projects. Then MSBuild might analyze the dependencies over and over again. To fix this issue I currently see two possible solutions:

1. Generate a temporary solution file and use this for compilation.
2. Analyze the given projects and remove all dependent projects from the list.

FAKE 1.40.5.0 implements the second idea. If you have a better idea please contact me.

Yesterday I released “FAKE – F# Make” version 1.33.0. This release has lots of small bug fixes and a couple of new features.

Important links:

• Project page
• Download
• Repository

Git helpers -Fake.Git.dll

Git is a distributed revision control system with an emphasis on speed. Git was initially designed and developed by Linus Torvalds for Linux kernel development. Every Git working directory is a full-fledged repository with complete history and full revision tracking capabilities, not dependent on network access or a central server.

[Wikipedia]

In the last couple of months I worked on small helper library for controlling Git. This library is now released as part of “FAKE – F# Make”. You can find the source code at http://github.com/forki/FAKE/tree/master/src/app/Fake.Git/.

Features:

• Repository handling
  • init, clone
• Submodules
  • init, clone, information about submodules
• Branches
  • checkout, create, delete, merge, rebase, tag, pull, push, reset, commit, …
• SHA1 calculation

Documentation generation with James Gregory’s docu tool

What’s a docu? A documentation generator for .Net that isn’t complicated, awkward, or difficult to use. Given an assembly and the XML that’s generated by Visual Studio, docu can produce an entire website of documentation with a single command.

[docu website]

Fake comes bundled with James Gregory’s documentations generator “docu”, which converts XML-Documentation comments into HTML files. All you need to do is downloading a template and calling the docu task inside your build script:

!+ (buildDir + "*.dll")
     |> Scan
     |> Docu (fun p ->
         {p with
             ToolPath = docuPath + "docu.exe"
             TemplatesPath = templatesSrcDir
             OutputPath = docsDir })

Since FAKE builds its own documentation with docu I started to add more (and hopefully better) XML doc comments. My plan is to describe more and more of the internal FAKE functions in the next releases. An updated HTML-document (generated via a docu task) can be found at http://www.navision-blog.de/fake/.

Side by side specification

For Test-driven development (TDD) it’s sometimes nice to have the specifications next to the implementation files since the specs are considered as documentation.

By using a tool like VSCommands it is possible to group the implementation with the specs (see also http://gist.github.com/457248).

This “side by side specification” makes TDD a lot easier but of course we don’t want to deploy the specification classes and the test data.

==>

FAKE has a new feature which automatically removes all specification files and test framework references according to a given convention:

Target "BuildApp" (fun _ –>
      !+ @"src\app\**\*.csproj"
         |> Scan
         |> Seq.map (
             RemoveTestsFromProject
                 AllNUnitReferences      // a default references convention
                 AllSpecAndTestDataFiles // a default file convention
                 )
         |> MSBuildRelease buildDir "Build"
         |> Log "AppBuild-Output: "
)

The conventions are simple functions and can be customized e.g.:

/// All Spec.cs or Spec.fs files and all files containing TestData
let AllSpecAndTestDataFiles elementName (s:string) =
     s.EndsWith "Specs.cs" ||
       s.EndsWith "Specs.fs" ||
       (elementName = "Content" && s.Contains "TestData")

Miscellaneous

• SQL Server helpers are moved to Fake.SQL.dll
  • Additional functions for attaching and detaching databases.
• FileHelper.CopyCached function was added
  • Copies the files from a cache folder. If the files are not cached or the original files have a different write time the cache will be refreshed.
• EnvironmentHelper.environVarOrDefault added
  • Retrieves the environment variable or a given default.
• Fixed Issue 3: toRelativePath calculates paths with ..\..\ if needed
• Added a build time report to the build output.
• XPathReplace and XMLPoke tasks added.
  • Replaces text in an XML file at the location specified by an XPath expression.
• ILMerge task added
• Windows Installer XML (WiX) task added

Earlier today I released “FAKE – F# Make” version 1.10.0. This new release contains a lot path issue fixes for Mono and a new architecture for logging and tracing.

A guy named Joel Mueller had an awesome idea and sent me some patches. He noticed that TraceHelper.fs writes all messages synchronously to the console and/or a XML output file, which means the actual build operations must wait on the writing of hundreds of trace messages, slowing down the actual build.

His idea was to use a MailboxProcessor to buffer up the trace messages and write them out asynchronously, so that the actual build can proceed at full speed.

Now all internal logging and tracing functions can post their their messages to the inbox of the MailboxProcessor:

This idea reduces the build time of FAKE’s self build from 3 min. to 2 min. Which is really amazing, since I didn’t have to change anything in the build script. This version is compatible to the last released version.

Please download “FAKE – F# Make” version 1.10.0 and tell me what you think.

The new version 0.27 of “FAKE – F# Make” comes with new syntactic sugar for build targets and build dependencies. Don’t be afraid the old version is still supported – all scripts should still work with the new version.

The problem

Consider the following target definition:

As you can see we are having a lot of “magic strings” for the target names and the dependency definitions. This was always a small shortcoming in FAKE, since this doesn’t allow refactoring and may result in runtime errors.

One of my goals for “FAKE – F# Make” is to remove these strings in future versions. Unfortunately this is not that easy, because it causes a lot of internal issues. In particular logging to the build server is much harder if you don’t have a target name.

The first step

As posted in a bitbucket comment by cipher we could use the “dynamic lookup operator” to remove some of the magic strings without breaking any internal code.

As a result we can rewrite the above sample as:

All magic strings suddenly disappeared. I think this syntax looks really nice, but unfortunately the strings are not really gone, since the token Default is only checked at runtime.

The idea for future versions

Since the new syntax is really just syntactic sugar I’m always interested in a better solution. Currently I’m working on a syntax using monads. The result could look like this:

This way the magic string are really gone, but my current problem is retrieving the target name from the let-binding name. Please leave a comment if you have an idea to solve this issue.

Yesterday I released “FAKE – F# Make” version 0.14 with xUnit.net support. The usage is very easy and similar to the usage of NUnit:

This sample works perfectly with TeamCity and creates a html-page per test project in addition:

If you want to publish the xUnit.net test results in CruiseControl.NET just modify the build script a little:

Now follow the steps in the CrusieControl.NET documentation. You will need to download the xUnitSummary.xsl file and save it to your webdashboard directory. If everything works correctly you should see something like this:

Since version 0.12 the FAKE build system provides an easy way to setup build configurations for CruiseControl.NET.

“CruiseControl.NET is an Automated Continuous Integration server, implemented using the Microsoft .NET Framework.”

[thoughtworks.org]

In this article I will show you how you can set up a FAKE build script in CruiseControl.NET. We will use the CalculatorSample which you can download from the FAKE Download page.

If you want to know how this build script works and how you could create one for your own projects please read the “Getting started with FAKE”-tutorial.

If you want to set up “FAKE – F# Make” build configurations in TeamCity please read “Integrating a "FAKE – F# Make" build script into TeamCity”.

Installing CruiseControl.NET

You can download CruiseControl.NET from thoughtworks.org. After the installation process (further instructions) you should see an empty dashboard:

Installing Subversion

The CalculatorSample is using Subversion (SVN) for SourceControl, so we need to download SVN from subversion.tigris.org. I’m using the “Windows MSI installer with the basic win32 binaries” and installed them under “c:\Program Files (x86)\Subversion\”.

Installing FxCop

The CalculatorSample is using FxCop, so we need to download and install it next.

“FxCop is an application that analyzes managed code assemblies (code that targets the .NET Framework common language runtime) and reports information about the assemblies, such as possible design, localization, performance, and security improvements. Many of the issues concern violations of the programming and design rules set forth in the Design Guidelines for Class Library Developers, which are the Microsoft guidelines for writing robust and easily maintainable code by using the .NET Framework.”

[MSDN]

Creating a FAKE Project

Now create a new folder for the CalculatorSample sources. I’m using “d:\Calculator\” for the rest of the article.

The next step is to modify the CruiseControl.NET config file (“c:\Program Files (x86)\CruiseControl.NET\server\ccnet.config” on my machine):

In this configuration I set up a trigger which checks every 30 sec. for changes in my CalculatorSample project.

If SVN finds changes FAKE.exe is called with my build script (completeBuild.fsx).

After the build I want to merge the FxCop and NUnit output files with my build results to create a build report.

Configuring the dashboard

In order to provide a nicer output on the dashboard we need to modify the BuildPlugins section in the dashboard.config file (“c:\Program Files (x86)\CruiseControl.NET\webdashboard\dashboard.config” on my machine):

As you can see we use the nant.xsl to transform the FAKE output to HTML.

Starting the build

Now if everything is configured correctly, you can tell CruiseControl.NET to run your build (press “Start” for your build project on the dashboard):

Now CruiseControl.NET should use SVN to checkout the CalculatorSample sources and run the build. The output on the project page for build 1 should look like this:

You can also inspect the NUnit and FxCop results:

Please feel free to give feedback if you have any problems with this article.

I just released a new version of my Open Source Build Automation Framework “FAKE – F# Make”. You can read more about FAKE on the project website or in the Getting started with "FAKE – F# Make"-article.

Although the new release contains many bugfixes, I only want to show the two major improvements here.

1. FAKE 0.10 uses FSI instead of FSC

From now on FAKE uses the “F# Interactive” (fsi.exe) instead of the F# Compiler (fsc.exe) to run the build scripts, which brings two major improvements.

No TempPath for compiled binaries needed

Due to the fact that FAKE scripts are no longer compiled at the beginning of the build process, we don’t need a temporary folder for the created binaries.

Loading modules at runtime

The #load command in F# scripts allows us to load modules at runtime. Now we are able to put reusable Targets or TargetTemplates (see below) into external build script files.

2. TargetTemplates

TargetTemplates provide an easy way to reuse common Targets. Let’s consider a (very) small sample:

This Target “TraceHello” traces a “Hello World” string into our build log. Now we want it to be slightly more generic and to trace a custom string. We can do this by using a TargetTemplate:

Now we have a template (or a function which generates targets) that gets a string for the target name and a string for the trace text and generates  a usable target:

Of course the TargetTemplate function is generic and can be used with any tuple as parameter:

One of the nice properties of functional programming languages is the easy extensibility of custom functions. Let’s consider a simple F# function (from “FAKE – F# Make”) for a recursive directory copy:

open System
open System.IO

/// Copies a directory recursive
/// Thanks to Robert Pickering http://strangelights.com/blog/
///  param target: target directory : string
///  param source: source directory : string
let CopyDir target source =
  Directory.GetFiles(source, "*.*", SearchOption.AllDirectories)
    |> Seq.iter (fun file ->
      let newFile = target + file.Remove(0, source.Length)
      printf "%s => %s" file newFile
      Directory.CreateDirectory(Path.GetDirectoryName(newFile)) |> ignore
      File.Copy(file, newFile, true))

If we want to allow users to set custom file filters, we can add a third parameter:

/// Copies a directory recursive
/// and allows to filter the files
/// Thanks to Robert Pickering http://strangelights.com/blog/
///  param target: target directory : string
///  param source: source directory : string
///  param filterFile: FilterFunction: string -> bool
let CopyDirFiltered target source filterFile =
  Directory.GetFiles(source, "*.*", SearchOption.AllDirectories)
    |> Seq.filter filterFile
    |> Seq.iter (fun file ->
      let newFile = target + file.Remove(0, source.Length)
      printfn "%s => %s" file newFile
      Directory.CreateDirectory(Path.GetDirectoryName(newFile)) |> ignore
      File.Copy(file, newFile, true))

Now we can define some filter functions:

/// Exclude SVN files (path with .svn)
/// excludeSVNFiles: string -> bool
let excludeSVNFiles (path:string) = not <| path.Contains ".svn"

/// Includes all files
/// allFiles: string -> bool 
let allFiles (path:string) = true

Now it is possible to use CopyDirFiltered in the following ways:

/// Copies all files <=> same as CopyDir
CopyDirFiltered "C:\\target" "C:\\source" allFiles

/// Copies all files except SVN files
CopyDirFiltered "C:\\target" "C:\\source" excludeSVNFiles

/// Copies all files only if random number <> 2
let r = new Random()
CopyDirFiltered "C:\\target" "C:\\source" (fun path -> r.Next(5) <> 2)

Extensibility of functions in C#

Of course we can do the same thing in C# 3.0:

/// <summary>
/// Copies a directory recursive
/// and allows to filter the files
/// </summary>
/// <param name="target">The target.</param>
/// <param name="source">The source.</param>
/// <param name="fileFilter">The file filter.</param>
public static void CopyDirFiltered(string target, string source,
                                   Func<string, bool> fileFilter)
{
    string[] allFiles = Directory.GetFiles(
        source, "*.*", SearchOption.AllDirectories);
    foreach (string file in from f in allFiles
                            where fileFilter(f)
                            select f)
    {
        string newFile = target + file.Remove(0, source.Length);
        Console.WriteLine("{0} => {1}", file, newFile);
        Directory.CreateDirectory(Path.GetDirectoryName(newFile));
        File.Copy(file, newFile, true);
    }
}

Now it is easy to use the C# function with lambdas:

“A lambda expression is an anonymous function that can contain expressions and statements, and can be used to create delegates or expression tree types.”

[MSDN]

Func<string, bool> filterSVN = x => !x.Contains(".svn");
Func<string, bool> allFiles = x => true;

/// Copies all files <=> same as CopyDir
CopyDirFiltered("C:\\target", "C:\\source", allFiles);

/// Copies all files except SVN files
CopyDirFiltered("C:\\target", "C:\\source", filterSVN);

/// Copies all files only if random number <> 2
var r = new Random();
CopyDirFiltered("C:\\target", "C:\\source", path => r.Next(5) != 2);

Keeping this simple technique in mind allows to create very flexible functions.