internal fragmentation

a personal journal of hacking, science, and technology

March development summary

Fri, 27 Apr 2012 17:52 by mpm in Uncategorized (link)

What I was up to in March:

  • released Mercurial 2.1.1
  • reviewed and merged 68 changesets
  • authored 39 changesets
  • wrote 157 mailing list messages
  • worked on 110 bug tracker issues
  • daily office hours
  • fixed a few long-standing case-folding and merge bugs
  • did various performance tweaks
  • started working on new consensus merge algorithm
  • started planning for 2.3 sprint

Probably the most interesting development this month was research into a more powerful merge algorithm, which I’m calling consensus merge. With any luck, this should be ready for testing in 2.3.

February development summary

Fri, 2 Mar 2012 17:14 by mpm in Uncategorized (link)

What I was up to in February:

  • released Mercurial 2.1
  • reviewed and merged 103 changesets
  • authored 27 changesets
  • wrote 160 mailing list messages
  • worked on 156 bug tracker issues
  • begin work on generic templating for all commands
  • various bug fixes
  • daily IRC office hours

The generic templating feature is an idea I’ve been kicking around for a while. The idea is be able to add templating support to commands with minimal code impact, while simultaneously adding support for generic formats like XML and JSON. You can read about the basics in this thread.

(The other big recent feature, of course, is phases, which were released in Mercurial 2.1.)

December and January development summaries

Wed, 29 Feb 2012 16:42 by mpm in Uncategorized (link)

Catching up on my “paperwork” from the run-up to the 2.1 release.

December:

  • released Mercurial 2.0.1
  •  reviewed and merged 123 changesets
  •  authored 15 changesets
  •  wrote 183 mailing list messages
  •  worked on 98 bug tracker issues
  •  worked on design and review for phase support
  •  improved bookmark handling for pull and update
  •  daily IRC office hours

January:

  •  released Mercurial 2.0.2 and python-hglib 0.2
  •  reviewed and merge 210 changesets
  •  authored 56 changesets
  •  wrote 277 mailing list messages
  •  worked on 138 bug tracker issues
  •  worked on design and review for phase support
  •  rewrote copy detection for diff and status
  •  improved handling of large revsets
  •  added support for full fileset queries in revsets
  •  numerous bug fixes
  •  daily IRC office hours

 

November update and Mercurial release numbering

Mon, 26 Dec 2011 17:18 by mpm in Uncategorized (link)

As always, things have been busy here, and it’s taken a while to get all the paperwork for this year’s fellowship lined up so my updates have dropped off for a few months. But work has continued apace and here’s what happened in November:

  • released Mercurial 2.0 and python-hglib 0.1
  • reviewed and merged 136 changesets
  • authored 43 csets
  • 138 mailing list messages
  • worked on 80 bug tracker issues
  • work on case folding, revsets, test harness, error messages, phase support
  • daily IRC office hours

The big item above, of course, is the release of Mercurial 2.0. So what are the big exciting new features in 2.0? You may be surprised to learn that 2.0 is actually a pretty boring release and that’s the way we like it.

For many projects, an x.0 release often means a major rethink/rewrite/break with the past. And if you look at projects like Python 3.0, Gnome 3.0, KDE 4.0, and so on, you’ll see that that kind of approach is often extremely unpopular with existing users.

Mercurial has a different philosophy: we recognize that an SCM is something that people plan on using for years if not decades. People often build elaborate frameworks and build systems around them and breakage here can mean losing many developer-days of time. So we take care to avoid affecting people’s existing work flows and automation. Regressions get more attention than other bugs.

Further, Mercurial takes an evolutionary approach to development.: steady accumulation of reliable small improvements, with regular releases. So almost all of the changes in the three and a half years since 1.0 have been in users’ hands for a while now. Our 2.0 release is simply 1.9 + .1. So why not just call it 1.10? First, we already know we will never do a release of the traditional 2.0 style, so there’s no point sticking on 1.x forever. Second, the difference between 1.0 and 2.0 is actually huge, notwithstanding the fact that most of the intermediate development has already been released to the public.

Similarly, I can tell you today when Mercurial 3.0 will be released: May 1st, 2014. And it will contain a huge number of interesting new features. But you won’t have to wait for 2014 to use them: they’ll all be released as soon as they’re ready. And none of them will require you to “upgrade” your repositories or all your users’ installs to keep working.

How to find rename data in Mercurial

Wed, 24 Aug 2011 11:27 by mpm in Uncategorized (link)

A fairly common question from new Mercurial users is “how do I figure out where/when a file got renamed?”

As an example, let’s look at the history of a file in the main Mercurial source that got renamed in the distant past. Start with something like this:

$ hg status --rev 10 --copies mercurial/localrepo.py
A mercurial/localrepo.py
  mercurial/hg.py

This says localrepo.py was copied from hg.py some time between the checkout and rev 10. We can even figure out where that happened:

$ hg log -v -r 'limit(follow("mercurial/localrepo.py") and file("mercurial/localrepo.py"), 1)'
changeset:   1089:142b5d5ec9cc
user:        mpm@selenic.com
date:        Sat Aug 27 14:21:25 2005 -0700
files:       mercurial/changelog.py mercurial/dirstate.py
mercurial/filelog.py mercurial/hg.py mercurial/httprepo.py
mercurial/localrepo.py mercurial/manifest.py mercurial/node.py
mercurial/remoterepo.py mercurial/repo.py mercurial/revlog.py
mercurial/sshrepo.py
description:
Break apart hg.py

- move the various parts of hg.py into their own files
- create node.py to store node manipulation functions

The magical -r argument can be read as:

“Find all revisions in the history of localrepo.py where it still has that name and show just the first one”

See ‘hg help revsets‘ for more info.

But we usually don’t need anything nearly that complicated. This simple command works where this file hasn’t existed multiple times:

$ hg log -l 1 -r 0: mercurial/localrepo.py
changeset:   1089:142b5d5ec9cc
user:        mpm@selenic.com
date:        Sat Aug 27 14:21:25 2005 -0700
summary:     Break apart hg.py

In other words “starting at rev 0, list one commit mentioning localrepo.py”

Here we can see exactly what happened in that crazy commit: lots of files copied from hg.py. These are all ‘copies’ and not ‘renames’ because hg.py continues to exist.

$ hg status --change 1089 -C
M mercurial/hg.py
M mercurial/revlog.py
A mercurial/changelog.py
  mercurial/hg.py
A mercurial/dirstate.py
  mercurial/hg.py
A mercurial/filelog.py
  mercurial/hg.py
A mercurial/httprepo.py
  mercurial/hg.py
A mercurial/localrepo.py
  mercurial/hg.py
A mercurial/manifest.py
  mercurial/hg.py
A mercurial/node.py
A mercurial/remoterepo.py
  mercurial/hg.py
A mercurial/repo.py
A mercurial/sshrepo.py
  mercurial/hg.py

And lastly, we can do:

$ hg diff --git -r 10 mercurial/localrepo.py
diff --git a/mercurial/hg.py b/mercurial/localrepo.py
copy from mercurial/hg.py
copy to mercurial/localrepo.py
--- a/mercurial/hg.py
+++ b/mercurial/localrepo.py
@@ -1,575 +1,2058 @@
-# hg.py - repository classes for mercurial
+# localrepo.py - read/write repository class for mercurial
 #
-# Copyright 2005 Matt Mackall
...

The  ‘–git’ flag asks Mercurial to use git’s non-standard diff format to show copy/rename info, and Mercurial thus shows a diff against hg.py from revision 10 rather starting with an empty file.

We obviously can’t do all of this in the web interface, but we can still easily track down file origins:

May and June maintainer updates

Thu, 7 Jul 2011 16:13 by mpm in Uncategorized (link)

Momentum from the 1.9 sprint carried on pretty well right through the 1.9 release. Now that things are a bit quieter, time to catch up on paperwork.

May:

  • 1.8.3 release
  • reviewed and merged 303 changesets
  • authored 42 csets
  • wrote 304 mailing list messages
  • weekly GSoC meetings
  • daily IRC office hours

June:

  • 1.8.4 release
  • reviewed and merged 237 csets
  • authored 55 csets
  • wrote 222 mailing list messages
  • created the new filesets feature
  • weekly GSoC meetings
  • daily IRC office hours

April maintainer update

Wed, 4 May 2011 14:42 by mpm in Uncategorized (link)

Mercurial development in April was quite busy, ending with our 1.9 sprint.

  • Released Mercurial 1.8.2
  • Reviewed and merged 214 changesets
  • Authored 37 changesets
  • 222 mailing list messages
  • Worked on refactoring merge and rebase
  • Worked on threading of our test harness
  • Worked on encoding and revset features
  • 1.9 sprint!
  • GSoC application review
  • Daily office hours

Mercurial 1.9 sprint, day three

Tue, 3 May 2011 23:33 by mpm in Uncategorized (link)

The last day of the sprint was Sunday, but a day without net access, a transatlantic flight, and a full inbox have delayed this update a bit. But this sprint was probably our most productive yet! Sunday was again about 10 hours of solid hacking. We finally shut down at about 8pm after half the team had left for the airport and others were on the verge of collapse from exhaustion.

As the project leader, I spent the bulk of my time in consultation on designs, debugging, code reviews, and merging patches.  This meant that in the whole course of the sprint, I managed to only write a total of 6 patches myself. And on Sunday I was extra-busy, because May 1st meant it was time for me to cut our monthly stable release (1.8.3).

Overall during the sprint, over 150 changesets were merged and a huge amount of exciting new work started at the sprint is still in the works. Highlights from Sunday include:

  • New, much faster changeset discovery protocol
  • PyFlakes-based code tests
  • More progress on new bundle format and lightweight copies
  • Test-suite speedups
  • Big file handling cleanup for PyPy
  • tons of bug fixes

But the real goal of the sprint is to build momentum for our biggest projects. Some of these got a lot of traction over the weekend. The most exciting is the stuff we’ve been discussing under the Liquid Hg banner, which is actually a collection of different features that will allow for painless and safe “evolution” of changesets. Various pieces of this will probably be appearing over the course of the next year.

We’ve been alternating sprints between the US and Europe every two releases, so the next sprint should be somewhere in the US around January 2012 as we prepare for the 2.1 release.

Mercurial 1.9 sprint, day two

Sun, 1 May 2011 02:36 by mpm in Uncategorized (link)

From left to right: Levi, Peter, Benoit, Natosha, Brodie, Patrick, Sune, Nicolas, Timeless, Matt, Dan, Augie, Pierre-Yves, Martin, Henrik, Mads, Alexander, and Kevin

 

The second day of the sprint was spent in an intense flurry of coding. We started at 9AM and ailed right through to 9PM when we finally stopped for dinner. Edlund’s onsite catering staff had breakfast and lunch on-hand for us.

During the day, I reviewed and merged over 50 changes from sprinters including:

  • improvements to graphlog
  • new ^ and ~ operators and last() function for revset
  • progress on lightweight copies
  • progress on new discovery protocol
  • work on improving case-collision smarts
  • work on PyPy support
  • fixes for Zeroconf support
  • progress on new bundle format
  • performance fix for the branch cache
  • numerous code clean-ups and assorted bug fixes

We’ve got sprint participants from Minneapolis, Helsinki, San Francisco, Århus, Gronigen, New York, Chicago, Paris(2), Zurich (4), and Copenhagen (6!). And we’ve also gotten a steady stream of patches from contributors at home.

Mercurial 1.9 sprint, day one

Sat, 30 Apr 2011 02:31 by mpm in Uncategorized (link)

I’m in Copenhagen this weekend for the Mercurial 1.9 sprint. We’ve got a team of 19 contributors meeting for three days at the beautiful offices of Edlund A/S.

Yesterday, we ran through a rather lengthy agenda, covering:

  • Google Summer of Code
  • Improving the templating engine
  • Filesets (a feature to complement revsets)
  • PyPy support and the difficulties of Python 3
  • Integrating various extension features into the core
  • Bundling some external extensions
  • Speeding up the test suite and testing on Windows
  • In-memory patching and interactive commit
  • Updating Mercurial: The Definitive Guide
  • Replacing our BTS
  • An extended discussion of frameworks for cleanly changing history (what we’ve been calling Liquid Hg)
Older Posts »