internal fragmentation

a personal journal of hacking, science, and technology

Mercurial: adding ‘unified’ tests

Wed, 4 Aug 2010 16:26 by mpm in Uncategorized (link)

Mercurial’s testing framework just got better. Since the beginning, the basic approach has been to run a test script and compare its output with known good output. This works great most of the time, but when things do change, it can be quite hard to figure out where because there’s no easy way to match the test commands with their output. If, for instance, you had a test script that said:

hg frob a
hg frob b

and the output:

abort: something bad happened

..you wouldn’t be able to tell which command caused the error.

Based on an idea from the 1.5 sprint, I’ve implemented a way to combine the input and the output in the same file in a ‘literate programming’ style. So the above example would look like this:

Try frobbing a:

  $ hg frob a

Try frobbing b (it shouldn't succeed):

  $ hg frob b
  abort: something bad happened

The test engine finds all of the lines prefixed with ”  $ ” and assembles a script to run and then reassembles an output test with the results of the test. The real trick of course, is lining up the input with the output. To do that, we simply inject line markers into the script of the form:

echo SALT <some random number> <line number>

..which we can easily locate in the output and strip out.

Another problem this new approach addresses is cleaning up non-repeatable output such as dates and names of temporary files. Our traditional approach is for the test script to filter output through sed or grep to clean it up or redirect it. But as we’re already massaging output, it’s very easy for us to simply throw in regular expressions on the bits we don’t care about. For instance:

Check the log:
  $ hg log -r 1
  changeset:   1:.*
  user:        test
  date:        .*
  summary:     commit test

With any luck, this change will also make tests slightly faster, as we’ll need to run fewer copies of sed.

Here’s an example of a full-blown test before and after.

1 Comment

  1. See also http://pypi.python.org/pypi/cram (which has a nice short description of it’s syntax — which is mostly the same as mercurial’s unified tests).

    Comment by Adrian Buehlmann — Tue, 28 Sep 2010 @ 02:22

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.