Rash thoughts about .NET, C#, F# and Dynamics NAV.


"Every solution will only lead to new problems."

Monday, 23. February 2009


Introducing NaturalSpec – A Domain-specific language (DSL) for testing – Part I

Filed under: C#,F#,NaturalSpec,Tools — Steffen Forkmann at 11:31 Uhr

Test-Driven development (TDD) is a well known software development technique and follows the mantra “Red-Green-Refactor”. Behavior-Driven Development (BDD) is a response to TDD and introduces the idea of using natural language to express the Unit Test scenarios.

There are a lot of popular testing frameworks around which can be used for BDD including xUnit.net ,NUnit, StoryQ, MSpec, NSpec and NBehave. Most of them can be used with fluent interfaces and therefore provides a good readability of the sources. Some of them even provide the possibility to generate a spec in natural language out of passed Unit tests.

What is a spec?

“A specification is an explicit set of requirements to be satisfied by a material, product, or service.”

American Society for Testing and Materials (ASTM) definition

A spec is an important document for the communication process – it enables domain experts to communicate with developers. But how can you verify the compliance with the spec? The answer is: you have to write unit tests. Even with the mentioned frameworks there is a lot of work to do in order to translate a spec scenario into a Unit Test.

Question 7 in the famous Joel Test is “Do you have a spec?”.

The idea of NaturalSpec is to give domain experts the possibility to express their scenarios directly in compilable Unit Test scenarios by using a Domain-specific language (DSL) for Unit Tests. NaturalSpec is completely written in F# – but you don’t have to learn F# to use it. You don’t even have to learn programming at all.

Example 1 – Specifying a list

Let’s consider a small example. If we want to test a new List implementation a spec could look like this:

[<Scenario>]
let When_removing_an_3_from_a_small_list_it_should_not_contain_3() =
  Given [1;2;3;4;5]              // “Arrange” test context
    |> When removing 3           // “Act|> It shouldn't contain 3    // “Assert|> It should contain 4       // another assertion
    |> Verify                    // Verify scenario

I used BDD style here and expressed my scenario in a quite natural language. As the comments are indicating the scenario is following the Arrange Act Assert (“AAA”) pattern.

With the Keyword “Given” I can create a test context (the objects I want to test). In this sample I created a list with 5 elements. With the keyword “When” I call a function which does something with my test context. In this case I want to remove the value 3. In the Assert section (keywords “It should” or “It shouldn’t”) I can give some observations, which should hold for my manipulated test context.

When I run this scenario via a NUnit runner (i am using TestDriven.Net) I get the following output:

Scenario: When removing an 3 from a small list it should not contain 3

– Given [1; 2; 3; 4; 5]
– When removing 3
=> It should not contain 3
=> It should contain 4
==> OK

Example 2 – Specifying a factorial function

If you implement factorial function the spec could look like this:

[<Scenario>]
let When_calculating_fac_5_it_should_equal_120() =
  Given 5
    |> When calculating factorial
    |> It should equal 120
    |> Verify    

[<Scenario>]
let When_calculating_fac_1_it_should_equal_1() =
  Given 1
    |> When calculating factorial
    |> It should equal 1
    |> Verify          

[<Scenario>]
let When_calculating_fac_0_it_should_equal_0() =
  Given 0
    |> When calculating factorial
    |> It should equal 1
    |> Verify

And the output of NaturalSpec would look like this:

Scenario: When calculating fac 0 it should equal 0

– Given 0
– When calculating factorial
=> It should equal 1
==> OK

Scenario: When calculating fac 1 it should equal 1

– Given 1
– When calculating factorial
=> It should equal 1
==> OK

Scenario: When calculating fac 5 it should equal 120

– Given 5
– When calculating factorial
=> It should equal 120
==> OK

Getting started

Of course you can use NaturalSpec to specify C# objects. I see my post "Using NaturalSpec to create a spec for C# projects" for a small sample.

You can download NaturalSpec at GoogleCode and follow the “Getting started” tutorial in order to write your first automatically testable spec.

I am very interested in your feedback. Do you like the syntax? What should I change? Do you consider using a spec tool like NaturalSpec?

Tags: , , , , , , , , , , ,

6 Comments »

  1. It seems as if we hooked you up with BDD at our .NET Boot Camp. Nice job so far. But one issue: do you really need the Verify function? It looks like “noise”, cause it is used in every scenario. The same with [] and []. I’m not the F# guy – so don’t hit me. I’m not sure if it is possible, but in my eyes this will look much more elegant:


    module FactorialSpec =
    let When_calculating_fac_5_it_should_equal_120() =
    Given 5
    |> When calculating factorial
    |> It should equal 120

    Comment by Marcel Hoyer — Tuesday, 24. February 2009 um 1:41 Uhr

  2. The first thing that jumps out at me (coming from the Ruby world) is the nasty punctuation. Most people in the Ruby world have moved to this kind of thing:

    test “When removing blah blah blah” do
    your test goes here
    end

    Much easier to read (and type) than the old-fashioned

    def test_when_removing_blah_blah_blah

    end

    The method “test” takes two parameters, a string containing the name of the test and a block, and adds the block to the list of tests to be executed.

    I’m very new to F#, but it seems like you could get to nicer test names using the same idea.

    Something vaguely like:

    let test (test_name:string, test_function: (unit -> bool)) =
    appendThisNameBlockPairToListOfTests test_name, test_function

    Lets you do:

    test “fails, no matter what”, (fun -> false)

    Comment by James Martin Moore — Monday, 2. March 2009 um 4:07 Uhr

  3. I am new to F# so bear with my ignorance if my request is stupid!
    When considering your last example, it seems that it would be useful to have the scenario name, inferred from the contents of the scenerio, instead of given as an identifier. This would eliminate some typing and probably also some errors.

    []
    let When_calculating_fac_0_it_should_equal_0() =
    Given 0
    |> When calculating factorial
    |> It should equal 1
    |> Verify

    In the example the identifier incorrectly states that factorial of 0 is 0, however in the spec is correct states that factorial of 0 is 1.

    Comment by Peter Lundsby — Sunday, 23. August 2009 um 12:06 Uhr

  4. Oups.

    Thanks Peter.

    Comment by Steffen Forkmann — Monday, 24. August 2009 um 9:07 Uhr

  5. Curious, if you prefer the Ruby way with the name of your tests wrapped in quotes as opposed to snake case, why not use the F# syntax:

    let “When calculating fac 5 it should equal 120“ =
    Given 5
    |> When calculating factorial
    |> It should equal 120
    |> Verify

    Will NaturalSpec not support this valid F# syntax?

    Comment by ninegrid — Monday, 25. October 2010 um 21:28 Uhr

  6. […] encountered leave a big gap between the code syntax and the specification syntax. Then I discovered NaturalSpec, which leverages F#’s DSL-friendly syntax to produce very readable tests, and I was totally […]

    Pingback by F# Testing DSL | GET http://localhost…200 OK — Monday, 18. March 2013 um 22:28 Uhr

RSS feed for comments on this post. | TrackBack URI

Leave a comment

XHTML ( You can use these tags): <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> .