internal fragmentation

a personal journal of hacking, science, and technology

Mercurial: adding ‘unified’ tests

Wed, 4 Aug 2010 16:26 by mpm in Uncategorized (link)

Mercurial’s testing framework just got better. Since the beginning, the basic approach has been to run a test script and compare its output with known good output. This works great most of the time, but when things do change, it can be quite hard to figure out where because there’s no easy way to match the test commands with their output. If, for instance, you had a test script that said:

hg frob a
hg frob b

and the output:

abort: something bad happened wouldn’t be able to tell which command caused the error.

Based on an idea from the 1.5 sprint, I’ve implemented a way to combine the input and the output in the same file in a ‘literate programming’ style. So the above example would look like this:

Try frobbing a:

  $ hg frob a

Try frobbing b (it shouldn't succeed):

  $ hg frob b
  abort: something bad happened

The test engine finds all of the lines prefixed with ”  $ ” and assembles a script to run and then reassembles an output test with the results of the test. The real trick of course, is lining up the input with the output. To do that, we simply inject line markers into the script of the form:

echo SALT <some random number> <line number>

..which we can easily locate in the output and strip out.

Another problem this new approach addresses is cleaning up non-repeatable output such as dates and names of temporary files. Our traditional approach is for the test script to filter output through sed or grep to clean it up or redirect it. But as we’re already massaging output, it’s very easy for us to simply throw in regular expressions on the bits we don’t care about. For instance:

Check the log:
  $ hg log -r 1
  changeset:   1:.*
  user:        test
  date:        .*
  summary:     commit test

With any luck, this change will also make tests slightly faster, as we’ll need to run fewer copies of sed.

Here’s an example of a full-blown test before and after.

Mercurial Fellowship status for July

Sun, 1 Aug 2010 15:31 by mpm in mercurial (link)

A quick summary of what I’ve been up to in July:

  • 1.6 release
  • reviewed and merged 171 changesets
  • authored 29 changesets
  • 83 mailing list messages
  • major refactoring of Mercurial protocol support
  • in-depth investigation of the notorious issue1327
  • lots of bug triage
  • started 1.7 sprint planning
  • started developing 2010 user survey
  • 4 GSoC meetings
  • daily IRC office hours
  • interview with students from ETNA