Issue948

Title convert killed for consuming too much memory on svn repo
Priority bug Status chatting
Superseder Nosy List kirr, mpm, pmezard, vadiml
Assigned To Topics convert

Created on 2008-01-28.21:42:56 by pmezard, last changed 2008-06-29.11:40:31 by kirr.

Messages
msg5057 (view) Author: pmezard Date: 2008-01-31.13:21:12
Well, first it would be good to compare boths. Can you make the git conversion
available somehow ?

One thing might explain mercurial conversion to be much larger: renaming of
large files. That would not be surprising given the number of branches and the
amount of binary contents. One possibility is the convert extension detects
rename incorrectly and duplicates files uselessly. Or this is a mercurial
shortcoming, see http://www.selenic.com/mercurial/bts/issue883 for details.

BTW, your 188MB is a bit strange because when converted under Windows or MacOSX
I ended with something around 300MB. That may be a case-sensitivity issue, I
will have to check again.
msg5056 (view) Author: vadiml Date: 2008-01-31.13:05:26
Then i've resumed the conversion and it terminated sucessfully
(so i suppose there is some kind of memory leak)

The only thing which bothers me now, ist the du -s .hg shows 188 MB 
and the guy whoi converted to git has repo size of 80 MB....
Is it expected?
msg5054 (view) Author: vadiml Date: 2008-01-31.12:41:23
I've tried the conversion on another machine, with much better (but not yet
correct) result:
hg convert
file://localhost/home/vadim/.svk/local/openwengo.svkmirror/wengophone-ng/branches/wengophone-2.2
tmp/wengophone-2.2
432 Merged revisions 11355 via svnmerge from
431 Merged revisions 11362 via svnmerge from
430 Merged revisions 11365 via svnmerge from
429 Merged revisions 11441 via svnmerge from
428 Merged revisions 11442 via svnmerge from
transaction abort!
rollback completed
** unknown exception encountered, details follow
** report bug details to http://www.selenic.com/mercurial/bts
** or mercurial@selenic.com
** Mercurial Distributed SCM (version 03ce5a919ae3)
Traceback (most recent call last):
  File "/usr/bin/hg", line 14, in <module>
    mercurial.dispatch.run()
  File "/usr/lib/python2.5/site-packages/mercurial/dispatch.py", line 20, in run
    sys.exit(dispatch(sys.argv[1:]))
  File "/usr/lib/python2.5/site-packages/mercurial/dispatch.py", line 29, in
dispatch
    return _runcatch(u, args)
  File "/usr/lib/python2.5/site-packages/mercurial/dispatch.py", line 45, in
_runcatch
    return _dispatch(ui, args)
  File "/usr/lib/python2.5/site-packages/mercurial/dispatch.py", line 348, in
_dispatch
    ret = _runcommand(ui, options, cmd, d)
  File "/usr/lib/python2.5/site-packages/mercurial/dispatch.py", line 401, in
_runcommand
    return checkargs()
  File "/usr/lib/python2.5/site-packages/mercurial/dispatch.py", line 357, in
checkargs
    return cmdfunc()
  File "/usr/lib/python2.5/site-packages/mercurial/dispatch.py", line 342, in
<lambda>
    d = lambda: func(ui, *args, **cmdoptions)
  File "/usr/lib/python2.5/site-packages/hgext/convert/__init__.py", line 68, in
convert
    return convcmd.convert(ui, src, dest, revmapfile, **opts)
  File "/usr/lib/python2.5/site-packages/hgext/convert/convcmd.py", line 329, in
convert
    c.convert()
  File "/usr/lib/python2.5/site-packages/hgext/convert/convcmd.py", line 272, in
convert
    self.copy(c)
  File "/usr/lib/python2.5/site-packages/hgext/convert/convcmd.py", line 249, in
copy
    newnode = self.dest.putcommit(filenames, parents, commit)
  File "/usr/lib/python2.5/site-packages/hgext/convert/hg.py", line 132, in
putcommit
    bin(p1), bin(p2), extra=extra)
  File "/usr/lib/python2.5/site-packages/mercurial/localrepo.py", line 658, in
rawcommit
    p1=p1, p2=p2, extra=extra, empty_ok=True)
  File "/usr/lib/python2.5/site-packages/hgext/mq.py", line 2081, in commit
    return super(mqrepo, self).commit(*args, **opts)
  File "/usr/lib/python2.5/site-packages/mercurial/localrepo.py", line 737, in
commit
    new[f] = self.filecommit(f, m1, m2, linkrev, trp, changed)
  File "/usr/lib/python2.5/site-packages/mercurial/localrepo.py", line 648, in
filecommit
    if fp2 == nullid and not fl.cmp(fp1, t) and not meta:
  File "/usr/lib/python2.5/site-packages/mercurial/filelog.py", line 79, in cmp
    if self.renamed(node):
  File "/usr/lib/python2.5/site-packages/mercurial/filelog.py", line 60, in renamed
    m = self._readmeta(node)
  File "/usr/lib/python2.5/site-packages/mercurial/filelog.py", line 38, in
_readmeta
    t = self.revision(node)
  File "/usr/lib/python2.5/site-packages/mercurial/revlog.py", line 945, in revision
    text = self.chunk(base, df=df)
  File "/usr/lib/python2.5/site-packages/mercurial/revlog.py", line 900, in chunk
    return decompress(c)
  File "/usr/lib/python2.5/site-packages/mercurial/revlog.py", line 97, in
decompress
    return _decompress(bin)
MemoryError
msg5039 (view) Author: mpm Date: 2008-01-30.22:51:32
The typical way to record memory usage is to run:

vmstat 1

..which will give a history of system VM usage in 1 second increments.
msg5037 (view) Author: pmezard Date: 2008-01-30.21:59:33
Just converted it on MacOSX without any problems. 2GB, svn bindings 1.4.5,
python 2.5. I am at loss for ideas right now.
msg5029 (view) Author: pmezard Date: 2008-01-30.12:55:56
I have no idea right now, maybe try more recent svn bindings if possible. 
On windows I have svn 1.4.4 bindings with python25.

Also, measuring the memory consumption spike might give a better idea of what's
happening. I don't know how to record that under linux though.
msg5024 (view) Author: vadiml Date: 2008-01-29.20:24:16
Great,

My problem occurs on ubuntu feisty with 4G of RAM....
Where should i look?
msg5005 (view) Author: pmezard Date: 2008-01-28.21:42:54
Splitting from issue 823.

vadiml:
I've tried the conversion on openwengo using hg-crew version and following command:
hg convert
file://localhost/home/vadim/.svk/local/openwengo.svkmirror/wengophone-ng/
openwengo.hgmirror/wengophone-ng

It procesed 2/3 of the repo and the failed

2032 * (clean) Replaced external owbuild with its content
2031 * (clean) Replaced external wifo with its content
2030 * (clean) Removed svn:externals property
2029 * (clean) Replaced external 3rdparty with its content
Killed


pmezard:
Likely an out of memory condition caused by

------------------------------------------------------------------------
r13240 | agateau | 2007-11-29 18:48:37 +0100 (Jeu, 29 nov 2007) | 1 line
Changed paths:
   A /wengophone-ng/branches/wengophone-2.2/libs/3rdparty (from
/owbuild/branches/wengophone-2.2/libs-3rdparty-cmakelists:13239)

* (clean) Replaced external 3rdparty with its content
------------------------------------------------------------------------
History
Date User Action Args
2008-06-29 11:40:31kirrsetnosy: + kirr
2008-01-31 13:21:12pmezardsetmessages: + msg5057
2008-01-31 13:05:26vadimlsetmessages: + msg5056
2008-01-31 12:41:23vadimlsetmessages: + msg5054
2008-01-30 23:02:41vadimlsetmessages: - msg5042
2008-01-30 23:00:55vadimlsetmessages: + msg5042
2008-01-30 22:51:32mpmsetnosy: + mpm
messages: + msg5039
2008-01-30 21:59:34pmezardsetmessages: + msg5037
2008-01-30 12:55:56pmezardsetmessages: + msg5029
2008-01-29 20:24:16vadimlsetmessages: + msg5024
2008-01-29 20:21:29vadimlsetmessages: - msg5006
2008-01-28 21:56:46pmezardsetstatus: unread -> chatting
messages: + msg5006
2008-01-28 21:42:56pmezardcreate