In main, resolving.

I've pushed the patches to crew (cb2f7652ad1b and 41bb88cb913e).

Setting this to testing.

That strikes me as an excellent idea.

Does commands.revert really have to call hg.revert?  AFAICS it calculates 
everything that has to be done, and then calls hg.revert to create/remove
files (after having recalculated everything).

I'm attaching a patch that makes commands.revert do everything by itself.
Tests pass.

Times (real user sys):
- original:
   14.427 11.673 2.088

- with a modified version of pmezard's patch:
    9.745  8.381 1.244

- together with this patch:
    2.699  2.544 0.140

pmezard's patch has to be changed a bit to avoid some "a: No such file or
directory" in test-backout and test-revert.  Instead of using names.keys(),
I think using the list of files from the first walk when src != 'b' should
work.

$ echo a > o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
$ time hg revert o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java

Without :

real   0m16.272s
user   0m6.484s
sys    0m9.045s

With it:

real   0m9.630s
user   0m4.302s
sys    0m4.723s

So how much of an improvement is your patch?

(A first version went by mail but did not get there after 30mn, so I am
copy/pasting it).

Matt Mackall a écrit :
> > Matt Mackall <mpm@selenic.com> added the comment:
> > 
> > On Sat, Dec 08, 2007 at 09:39:26AM -0000, Patrick M??zard wrote:
>> >> Patrick M??zard <pmezard@gmail.com> added the comment:

[...]

>> >>
>> >> $ echo a > o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
>> >> $ time hg st
>> >> M o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
>> >>
>> >> real    0m12.507s
>> >> user    0m3.960s
>> >> sys     0m6.003s
> > 
> > I assume this is cache-hot too.

Yes

> > 
>> >> $ hg --debug --profile revert
>> >> o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java

[...]

>> >>
>> >>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>> >>         2   12.251    6.126   13.805    6.903 dirstate.py:434(findfiles)
>> >>    340333    0.764    0.000    0.764    0.000 posixpath.py:56(join)
>> >>     67301    0.491    0.000   15.239    0.000 dirstate.py:382(statwalk)
>> >>     78/40    0.420    0.005    0.501    0.013 demandimport.py:43(_load)
>> >>     36471    0.401    0.000    0.401    0.000 posixpath.py:374(normpath)
>> >>         1    0.386    0.386    0.490    0.490 dirstate.py:130(_read)
>> >>         2    0.329    0.165   14.632    7.316 dirstate.py:523(status)
> > 
> > We spend most of our time checking the repo status. Is this 0.9.5?

This is crew before this week-end updates:

changeset:   5593:2e76e5a23c2b
tag:         qparent
user:        Steve Borho <steve@borho.org>
date:        Mon Dec 03 17:28:26 2007 -0600
summary:     workaround for raw_input() on Windows

> > 
>> >>         1    0.013    0.013    8.138    8.138 hg.py:292(revert)
> > 
> > And something in here is taking a while too, but it's not clear what.

That's merge working context _status being computed.


I made more tests with crew tip. First I added some timing code in
commands.revert() and merge.update(). Most time is spent in revert() status()
call and in merge working context _status initialization. The first can be fixed
with:

---
diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -2154,7 +2154,7 @@ def revert(ui, repo, *pats, **opts):
             names[abs] = (rel, exact)
             target_only[abs] = True
 
-        changes = repo.status(match=names.has_key)[:5]
+        changes = repo.status(files=names.keys())[:5]
         modified, added, removed, deleted, unknown = map(dict.fromkeys, changes)
 
         # if f is a rename, also revert the source
---

I ran the following test against linux-2.6 mirror with and without the patch:
- Append a character to a random file, call status, then revert the file
explicitely.
- Append a character to all files, call status, revert a single file, revert all
files.

I was afraid that "files" handling would be more expensive with larger
repositories and larger lists but revert timings were consistently better with
this patch.

The merge part looks harder to fix.

On Sat, Dec 08, 2007 at 09:39:26AM -0000, Patrick M??zard wrote:
> 
> Patrick M??zard <pmezard@gmail.com> added the comment:
> 
> MacBook Pro (HFS+), crew:
> 
> On a cold clone:
> 
> $ time hg revert -r 0 o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
> 
> real    0m29.538s
> user    0m8.883s
> sys     0m11.914s
> 
> 
> 
> On a hot clone:
> 
> $ echo a > o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
> $ time hg revert o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
> 
> real    0m17.101s
> user    0m6.633s
> sys     0m9.110s
> 
> $ echo a > o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
> $ time hg st
> M o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
> 
> real    0m12.507s
> user    0m3.960s
> sys     0m6.003s

I assume this is cache-hot too.

> 
> $ hg --debug --profile revert
> o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
> saving current version of
> o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java as
> o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java.orig
> reverting o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
> resolving manifests
>  overwrite True partial True
>  ancestor 579f9883aaeb local 579f9883aaeb+ remote 579f9883aaeb
>  o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java: remote is newer -> g
> getting o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
>          1417174 function calls (1414490 primitive calls) in 17.451 CPU seconds
> 
>    Ordered by: internal time, call count
>    List reduced from 308 to 40 due to restriction <40>
> 
>    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>         2   12.251    6.126   13.805    6.903 dirstate.py:434(findfiles)
>    340333    0.764    0.000    0.764    0.000 posixpath.py:56(join)
>     67301    0.491    0.000   15.239    0.000 dirstate.py:382(statwalk)
>     78/40    0.420    0.005    0.501    0.013 demandimport.py:43(_load)
>     36471    0.401    0.000    0.401    0.000 posixpath.py:374(normpath)
>         1    0.386    0.386    0.490    0.490 dirstate.py:130(_read)
>         2    0.329    0.165   14.632    7.316 dirstate.py:523(status)

We spend most of our time checking the repo status. Is this 0.9.5?

>         1    0.013    0.013    8.138    8.138 hg.py:292(revert)

And something in here is taking a while too, but it's not clear what.

MacBook Pro (HFS+), crew:

On a cold clone:

$ time hg revert -r 0 o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java

real    0m29.538s
user    0m8.883s
sys     0m11.914s



On a hot clone:

$ echo a > o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
$ time hg revert o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java

real    0m17.101s
user    0m6.633s
sys     0m9.110s

$ echo a > o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
$ time hg st
M o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java

real    0m12.507s
user    0m3.960s
sys     0m6.003s

$ hg --debug --profile revert
o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
saving current version of
o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java as
o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java.orig
reverting o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
resolving manifests
 overwrite True partial True
 ancestor 579f9883aaeb local 579f9883aaeb+ remote 579f9883aaeb
 o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java: remote is newer -> g
getting o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java
         1417174 function calls (1414490 primitive calls) in 17.451 CPU seconds

   Ordered by: internal time, call count
   List reduced from 308 to 40 due to restriction <40>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        2   12.251    6.126   13.805    6.903 dirstate.py:434(findfiles)
   340333    0.764    0.000    0.764    0.000 posixpath.py:56(join)
    67301    0.491    0.000   15.239    0.000 dirstate.py:382(statwalk)
    78/40    0.420    0.005    0.501    0.013 demandimport.py:43(_load)
    36471    0.401    0.000    0.401    0.000 posixpath.py:374(normpath)
        1    0.386    0.386    0.490    0.490 dirstate.py:130(_read)
        2    0.329    0.165   14.632    7.316 dirstate.py:523(status)
        4    0.271    0.068    1.372    0.343 localrepo.py:830(walk)
        1    0.259    0.259    0.259    0.259 manifest.py:39(parse)
        3    0.212    0.071    0.212    0.071 revlog.py:872(loadcache)
   134592    0.179    0.000    0.198    0.000 dirstate.py:406(imatch)
        6    0.171    0.028    0.171    0.028 revlog.py:94(decompress)
    728/2    0.133    0.000    0.234    0.117 sre_parse.py:385(_parse)
    67294    0.128    0.000    0.158    0.000 util.py:550(<lambda>)
    67303    0.103    0.000    0.103    0.000 struct.py:77(unpack)
    67298    0.079    0.000    0.160    0.000 dirstate.py:363(_supported)
   108/81    0.074    0.001    0.107    0.001 demandimport.py:80(_demandimport)
        1    0.061    0.061    0.226    0.226 merge.py:333(manifestmerge)
        3    0.060    0.020    0.060    0.020 revlog.py:51(hash)
    67298    0.060    0.000    0.081    0.000 stat.py:54(S_ISREG)
    783/2    0.059    0.000    0.069    0.034 sre_compile.py:38(_compile)
      719    0.058    0.000    0.058    0.000 ignore.py:11(_parselines)
        3    0.050    0.017    0.053    0.018 revlog.py:441(__init__)
    30686    0.047    0.000    0.047    0.000 sre_parse.py:188(__next)
   833/52    0.037    0.000    0.037    0.001 sre_parse.py:146(getwidth)
   169802    0.037    0.000    0.037    0.000 util.py:1143(pconvert)
        1    0.036    0.036    0.060    0.060 manifest.py:30(copy)
   134590    0.036    0.000    0.036    0.000 util.py:253(always)
    29947    0.031    0.000    0.077    0.000 sre_parse.py:207(get)
        2    0.024    0.012    0.024    0.012 manifest.py:13(__init__)
        4    0.022    0.006    0.957    0.239 dirstate.py:27(__getattr__)
    67366    0.022    0.000    0.022    0.000 stat.py:29(S_IFMT)
    29896    0.019    0.000    0.019    0.000 sre_parse.py:144(append)
        5    0.019    0.004    0.726    0.145 manifest.py:54(read)
        7    0.018    0.003    0.495    0.071 revlog.py:921(revision)
        1    0.015    0.015    0.015    0.015 shutil.py:18(copyfileobj)
    67294    0.014    0.000    0.014    0.000 util.py:541(<lambda>)
        1    0.013    0.013    8.138    8.138 hg.py:292(revert)
    30891    0.010    0.000    0.010    0.000 sre_parse.py:136(__getitem__)
        1    0.007    0.007    0.065    0.065 ignore.py:24(_parsefiles)

Can you run it with --profile --debug, please?

Also, how long does hg st take?

BTW the repository has around 67k files its the NetBeans IDE source.

Tested this on a Mac mini 1.8 GHz x86, dual core, 2 Gig Ram and took 50 sec
using Hg 0.9.5

The other test was run on a Sony Vaio, x86, dual core 2.4 GHz and 2 Gig Ram.

If you call Revert modification on a single file, it takes a long time on a big
repository. To reproduce:
Running with hg 0.9.3 on OpenSoalris b78:

- take repository from
http://deadlock.netbeans.org/hudson/job/hgimport/lastSuccessfulBuild/artifact/apisupportx/hgimport/out/main.tar.gz
- untar with gtar
- run hg update to populate the working dir
- Edit and save
~/work/main/o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java

- run:
$ time hg revert --repository ~/work/main --rev 0
~/work/main/o.o.util/src/org/netbeans/modules/openide/util/AWTBridge.java

real    0m59.899s
user    0m27.126s
sys     0m3.109s