I've made a proof of concept patch that uses the longpath Windows file \\?\.. API.
I can clone and checkout long and reserved paths on windows (example repo:
http://www.cadifra.com/cgi-bin/repos/long2/)

I may update this patch for a while at
http://www.cadifra.com/cgi-bin/repos/hg-longpath/file/tip/longpath.patch.

Problem with this patch is, that explorer can't copy nor delete such
repo directories. As a workaround, robocopy could be used for copy
and I made a small python script for deleting long path trees
(available at http://www.cadifra.com/cgi-bin/repos/winrm/ ).

See also Issue793