git 1.6 compared to hg 1.1 ? - correct test of commit speed

[Arne Babenhauserheide]

Am Montag 29 Dezember 2008 15:42:42 schrieb Alpár Jüttner:
> > Results:
> >
> > init:
> > - git: 0.217 ± 0.445
>
> Hmm... Then git is _very_ fast in some cases, indeed. :)

Which shows that the assumption that the speed is gaussian distributed doesn't 
fit when the minimum speed is 0 :) 
(I use the gaussian assumption to be able to use the simple formula for the 
standard deviation)

I assume it hit some load spike on my machine which added a very long time in 
between. You can see clearly from that how "predictable" single performance 
tests are, and how vitally important proper deviation calculation is to get 
useful data. 


I repeated this test since I found a flaw in the script. It always called "hg 
add *" explicitely before doing the commit. But while this is the preferred 
workflow for git, the preferred workflow for Mercurial is to commit 
every changed file with a single command. 


These are the results for a repository with 100 files, each test done 10 times 
(except the "only 2 files changed" one - that was done 100 times), using a 
Mercurial without activated extensions: 

init:
- git: 0.042 ± 0.007
- hg: 0.132 ± 0.010
- bzr: 0.616 ± 0.049

initial_commit:
- git: 0.210 ± 0.047
- hg: 0.885 ± 0.128
- bzr: 2.126 ± 0.117

commit_after_append_small:
- git: 0.140 ± 0.023
- hg: 0.420 ± 0.016
- bzr: 1.226 ± 0.081

commit_after_append_of_one_long_line:
- git: 0.452 ± 0.013
- hg: 0.504 ± 0.023
- bzr: 1.292 ± 0.026

commit_after_append_of_many_lines:
- git: 0.461 ± 0.012
- hg: 0.513 ± 0.021
- bzr: 1.346 ± 0.071

multiple_commits_with_only_two_files_changed_each:
- git: 0.030 ± 0.006
- hg: 0.145 ± 0.019
- bzr: 0.504 ± 0.081

What you see here is that for changes to only two files Mercurial is well 
below the noticeable 0.3s point and quite close to the no longer noticeable 
0.1s point. 

For this last test the fowllowing data is interesting: 

$ time python -c "print 'hello world'" # just startup python
...
real    0m0.051s

$ time hg version # load just the mercurial core
...
real    0m0.118s

So the time difference of about 0.115s between git and Mercurial can 
completely be attributed to having to load the Mercurial core before doing the 
commit (which is necessary to check if any plugin overwrites the commit 
command). 


For other commands I have the following results, but without deviation since 
the tests take too long for me to do 10 repetitions in a reasonable amount of 
time (mostly because of the slow bzr clones...). I plan to integrate these 
tests into my general test suite, though, once I have a final collection of 
tests to do. But first the current test results: 

I use the Python repositories created for the Python DVCS PEP: 
- bzr: http://code.python.org/python/trunk
- hg: http://code.python.org/hg/trunk/
- git: git://code.python.org/python/trunk

initial clone of the Python repository: 
- bzr: 10m24.372s
- hg: 2m8.449s
- git: 1m47.909s

comparing repository sizes
du -hs python-bzr-trunk/.bzr python-hg-trunk/.hg python-git-trunk/.git
215M    python-bzr-trunk/.bzr
111M    python-hg-trunk/.hg
74M     python-git-trunk/.git


status in a clean tree (Mercurial _without_ inotify extension; with inotify it 
goes down to 0.2s): 
- bzr: 0m3.993s
- hg: 0m0.655s
- git: 0m0.150s

a full log: 
- bzr: 0m55.191s
- hg: 0m22.042s
- git: 0m1.434s

a log of Lib/: 
- bzr: 0m27.412s
- hg: 0m13.407s
- git: 0m2.982s

see the last 20 changes: 
- bzr: 0m9.896s
- hg: 0m0.560s
- git: 0m0.006s

Annotate Misc/NEWS: 
- bzr: 0m12.382s
- hg: 0m30.561s
- git: 0m4.068s

Integrity check of the repository: 
- bzr: skipped because it takes far too long. 
- hg: 0m40.945s
- git: 4m20.604s

local clone: 
- bzr: 4m13.606s
- hg: 0m39.925s
- git: 0m8.831s

Note: Using a shared repository in bzr massively speeds up the local cloning 
(down to 3.5s for a clone) since every "copy" is done using hardlinks. You 
then have to keep your clones in the shared repository, though. 


Best wishes, 
Arne
PS: Sorry that it took me so long to reply to your mail. I wanted to check 
some things before sending it, and it got stuck in my drafts folder. 
-- 
-- My stuff: http://draketo.de - stories, songs, poems, programs and stuff :)
-- Infinite Hands: http://infinite-hands.draketo.de - singing a part of the 
history of free software.
-- Ein Würfel System: http://1w6.org - einfach saubere (Rollenspiel-) Regeln.

-- PGP/GnuPG: http://draketo.de/inhalt/ich/pubkey.txt

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
Url : http://selenic.com/pipermail/mercurial/attachments/20090107/e201b3f3/attachment.pgp