cvs2hg problem with CVS vendor branch (was Re: cvsps wrapper script)

[John Hein]

John Hein wrote at 00:11 -0600 on Oct  5, 2009:
 > Traceback (most recent call last):
 >   File "/2/pub/jhein/tmp/cvs2hg/cvs2svn/cvs2hg", line 90, in <module>
 >     hg_main(os.path.basename(sys.argv[0]), sys.argv[1:])
 >   File "/.amd_mnt/brain/uv/homes/jhein/tmp/hgt/zz/1/cvs2hg/cvs2svn/cvs2svn_lib/main.py", line 125, in hg_main
 >   File "/.amd_mnt/brain/uv/homes/jhein/tmp/hgt/zz/1/cvs2hg/cvs2svn/cvs2svn_lib/main.py", line 86, in main
 >   File "/.amd_mnt/brain/uv/homes/jhein/tmp/hgt/zz/1/cvs2hg/cvs2svn/cvs2svn_lib/pass_manager.py", line 181, in run
 >   File "/.amd_mnt/brain/uv/homes/jhein/tmp/hgt/zz/1/cvs2hg/cvs2svn/cvs2svn_lib/passes.py", line 1803, in run
 >   File "/.amd_mnt/brain/uv/homes/jhein/tmp/hgt/zz/1/cvs2hg/cvs2svn/cvs2svn_lib/svn_commit.py", line 243, in output
 >   File "/2/pub/jhein/tmp/cvs2hg/cvs2svn/cvs2svn_lib/hg_output_option.py", line 289, in process_primary_commit
 >     svn_commit, [parent1, parent2], filenames, getfilectx, lod)
 >   File "/2/pub/jhein/tmp/cvs2hg/cvs2svn/cvs2svn_lib/hg_output_option.py", line 699, in _commit
 >     return self._commit_memctx(mctx)
 >   File "/2/pub/jhein/tmp/cvs2hg/cvs2svn/cvs2svn_lib/hg_output_option.py", line 705, in _commit_memctx
 >     node = self.repo.commitctx(mctx)
 >   File "/site/dist/mercurial-py25-1.3.1/lib/python2.5/site-packages/mercurial/localrepo.py", line 925, in commitctx
 >     linkrev = len(self)
 >   File "/site/dist/mercurial-py25-1.3.1/lib/python2.5/site-packages/mercurial/localrepo.py", line 129, in __len__
 >     return len(self.changelog)
 >   File "/site/dist/mercurial-py25-1.3.1/lib/python2.5/site-packages/mercurial/util.py", line 150, in __get__
 >     result = self.func(obj)
 >   File "/site/dist/mercurial-py25-1.3.1/lib/python2.5/site-packages/mercurial/localrepo.py", line 104, in changelog
 >     c = changelog.changelog(self.sopener)
 >   File "/site/dist/mercurial-py25-1.3.1/lib/python2.5/site-packages/mercurial/changelog.py", line 94, in __init__
 >     revlog.revlog.__init__(self, self._delayopener, "00changelog.i")
 >   File "/site/dist/mercurial-py25-1.3.1/lib/python2.5/site-packages/mercurial/revlog.py", line 468, in __init__
 >     d = self._io.parseindex(f, i, self._inline)
 >   File "/site/dist/mercurial-py25-1.3.1/lib/python2.5/site-packages/mercurial/revlog.py", line 381, in parseindex
 >     data += fp.read()
 > MemoryError

This was happening in the last pass (pass 16), by the way.  So close.

The largest repo I was able to convert with cvs2hg was about like so:

cvs2svn Statistics:    [that should read "cvs2hg Statistics", I suppose]
------------------
Total CVS Files:               555
Total CVS Revisions:          2923
Total CVS Branches:           5351
Total CVS Tags:              25608
Total Unique Tags:             671
Total Unique Branches:         141
CVS Repos Size in KB:        73637
Total SVN Commits:            2601
First Revision Date:    Fri Jan  7 16:52:14 2000
Last Revision Date:     Wed Sep 30 13:55:36 2009

[No, there aren't really that many real cvs branches, but that is what
cvs2hg is reporting - cvs2svn on the same cvs repo is not the same,
but has a similarly large number (6620 branches)]

If I try to convert a cvs repo that is much larger than that, I was
hitting the MemoryError.  You can see it start climbing up to hundreds
of MB memory usage in top(1) once cvs2hg hits the last pass.

I tried to split the cvs repository into pieces and try using --existing-hgrepos
to build up the hg repo in multiple conversions with cvs2hg.  It's not clear
how well this will work (cvs2svn doesn't do well to do incremental conversions
- "File already exists... path 'trunk'"), but I thought I'd try it.

But when I try to use --existing-hgrepos, I get:

ERROR: directory '--hgrepos=/tmp/foo-hg' does not contain a Mercurial repository

(and, yes, there is an hg repo there)

Is cvs2hg supposed to support --existing-hgrepos at this point in its life?