internal fragmentation

a personal journal of hacking, science, and technology

Mercurial and Unit Testing

Thu, 12 Feb 2009 23:35 by mpm in Uncategorized (link)

Mercurial doesn’t do unit testing.

This is something of a shocker to some people, who think that unit testing is the one true way to test code. But in fact, it’s a bad fit for Mercurial.

First off, unit testing implies stable APIs. Unit tests test the internal APIs and make sure they perform as expected. If you want to evolve your APIs fluidly over time (and we do!), the tests have to be evolved along with them. Not only does this mean a lot of work, this can mean accidentally breaking or losing test cases and having previously fixed bugs reappear.

Second, the data structures we work with most are entire project histories with backing files and directory trees of working files. All but the simplest APIs take an entire repository as an input. Many produce a changed repository or other complex object (diff, bundle, log, changed working directory). Generating these objects programmatically is complex, tedious, and subject to change.

Third, we’re very serious about backwards compatibility of our user-visible interface (aka the command line). People get upset if their tools stop working the way they used to and break their build process. So it’s actually more important that we get the same correct end result than that any particular module work in a particular way.

So instead we take a more holistic approach to testing. Our tests consist of simple shell scripts exercising all the visible (and some invisible) behavior of Mercurial by building repositories and running commands against them just as a user would. To test, we compare the output of each of these scripts against known good output. This lets us freely refactor our internal systems without having to revisit old tests while maintaining confidence that we haven’t broken anything.

Another advantage of this approach is that it has a great synergy with bug reports. If a users sends us a list of commands they ran to produce a bug, we have a ready-made test case. A good percentage of our test suite was developed this way: directly by users.

At this point, our test suite is fairly comprehensive. We’ve got over 300 test scripts, some of them quite extensive, weighing in at about 20k lines (not counting results). That’s more than half as big as the Mercurial code itself (a mere 37k lines). And many of these tests have been unchanged for years even though much of the underlying code has changed completely.

(All that said, we do actually do a small amount of unit testing: about 3% of our test scripts are written in Python to exercise some of our simplest isolated interfaces. Unit testing is sometimes the right answer.)

No Comments

No comments yet.

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.