works fine the last time I tested by disabling demandimport in hg, hgwebdir.cgi
and hg-ssh.

I think we've taken care of most of the circular bits, moving to testing.

The circular import patch<->cmdutil is no problem, even without demandimport.
(don't ask me why, but importing one or the other works fine)

Therefore I pushed an updated version of 55860a45bbf2 to crew (not crew-stable)

For transplanting this to -stable the changesets 18a9fbb5cd78, 60acf1432ee0 and
da1658d63647 would need to be transplanted, too.

mpm, should this be done?

If not, a dummy dispatch module should be added to -stable to provide
dispatch.dispatch(), too.

Attached with commit message for easy importing

the following (ugly) patch can break circular imports for test-context.py and
test-oldcgi:

diff -r 56ee6438977d -r 41f4abe3bbfd mercurial/commands.py
--- a/mercurial/commands.py	Thu Jun 28 16:32:37 2007 +0200
+++ b/mercurial/commands.py	Thu Aug 02 20:30:49 2007 +0200
@@ -11,7 +11,7 @@ import ui, hg, util, revlog, bundlerepo,
 import ui, hg, util, revlog, bundlerepo, extensions
 import difflib, patch, time, help, mdiff, tempfile
 import errno, version, socket
-import archival, changegroup, cmdutil, hgweb.server, sshserver
+import archival, changegroup, cmdutil, sshserver
 
 # Commands start here, listed alphabetically
 
@@ -2478,6 +2478,7 @@ def serve(ui, repo, **opts):
         def init(self):
             util.set_signal_handler()
             try:
+                import hgweb.server #FIXME
                 self.httpd = hgweb.server.create_server(parentui, repo)
             except socket.error, inst:
                 raise util.Abort(_('cannot start server: ') + inst.args[1])
diff -r 56ee6438977d -r 41f4abe3bbfd mercurial/extensions.py
--- a/mercurial/extensions.py	Thu Jun 28 16:32:37 2007 +0200
+++ b/mercurial/extensions.py	Thu Aug 02 20:30:49 2007 +0200
@@ -6,7 +6,7 @@
 # of the GNU General Public License, incorporated herein by reference.
 
 import imp, os
-import commands, hg, util, sys
+import hg, util, sys
 from i18n import _
 
 _extensions = {}
@@ -22,6 +22,7 @@ def find(name):
         raise KeyError(name)
 
 def load(ui, name, path):
+    import commands # FIXME
     if name in _extensions:
         return
     if path:

still not enough.

You can check by importing my patch "demandimport-script" which is attached to
this issue and run test-context.py or test-oldcgi:
ImportError: cannot import name hg

But we should still also enable demand import in scripts rather than in core.

This set is fixed in 616a5adbf402, now in tip.

There are a couple others that are more challenging, in particular commands <->
commandutils.

mpm, I think you mentioned on IRC that you have a patch to fix the circular
references?

attached is my patch to move enabling demandimport to the scripts

Here's a patch I wrote some time ago that checks for the presence of the module
at import time. It incurs a small performance cost, but should preserve most of
the benefits of demandimport. I think it solves your particular bug, and would
also allow most if not all of the blacklist to be removed.

# HG changeset patch
# User Brendan Cully <brendan@kublai.com>
# Date 1183049664 25200
# Node ID 1a0a31d44b87c4ce2a36305b1c53af87238f51d7
# Parent  2ececafa58596042edcaeb93f0a35b9d3daa36ab
demandimport: raise ImportError immediately for missing modules

diff --git a/mercurial/demandimport.py b/mercurial/demandimport.py
--- a/mercurial/demandimport.py
+++ b/mercurial/demandimport.py
@@ -25,6 +25,9 @@ These imports will not be delayed:
 '''
 
 _origimport = __import__
+import pdb
+import imp
+import os.path
 
 class _demandmod(object):
     """module demand-loader and proxy"""
@@ -35,6 +38,12 @@ class _demandmod(object):
         else:
             head = name
             after = []
+        try:
+            f = globals.get('__file__')
+            path = f and os.path.dirname(f)
+            imp.find_module(head, [path])
+        except ImportError:
+            imp.find_module(head)
         object.__setattr__(self, "_data", (head, globals, locals, after))
         object.__setattr__(self, "_module", None)
     def _extend(self, name):
@@ -78,7 +87,7 @@ class _demandmod(object):
         setattr(self._module, attr, val)
 
 def _demandimport(name, globals=None, locals=None, fromlist=None):
-    if not locals or name in ignore or fromlist == ('*',):
+    if not locals or name in ignore or (fromlist and '*' in fromlist):
         # these cases we can't really delay
         return _origimport(name, globals, locals, fromlist)
     elif not fromlist:

Here's a fix I'm testing:

diff -r cf8b8f62688a mercurial/commands.py
--- a/mercurial/commands.py     Mon Jun 25 21:23:24 2007 -0500
+++ b/mercurial/commands.py     Thu Jun 28 10:14:22 2007 -0500
@@ -3094,6 +3094,8 @@ table = {
     "version": (version_, [], _('hg version')),
 }

+extensions.commandtable = table
+
 norepo = ("clone init version help debugancestor debugcomplete
 debugdata"
           " debugindex debugindexdot debugdate debuginstall")
 optionalrepo = ("paths serve showconfig")
diff -r cf8b8f62688a mercurial/extensions.py
--- a/mercurial/extensions.py   Mon Jun 25 21:23:24 2007 -0500
+++ b/mercurial/extensions.py   Thu Jun 28 10:14:23 2007 -0500
@@ -6,10 +6,12 @@
 # of the GNU General Public License, incorporated herein by
 reference.

 import imp, os
-import commands, hg, util, sys
+import util, sys
 from i18n import _

 _extensions = {}
+commandtable = {}
+setuphooks = []

 def find(name):
     '''return module with given extension name'''
@@ -54,13 +56,13 @@ def load(ui, name, path):
         uisetup(ui)
     reposetup = getattr(mod, 'reposetup', None)
     if reposetup:
-        hg.repo_setup_hooks.append(reposetup)
+        setuphooks.append(reposetup)
     cmdtable = getattr(mod, 'cmdtable', {})
-    overrides = [cmd for cmd in cmdtable if cmd in commands.table]
+    overrides = [cmd for cmd in cmdtable if cmd in commandtable]
     if overrides:
         ui.warn(_("extension '%s' overrides commands: %s\n")
                 % (name, " ".join(overrides)))
-    commands.table.update(cmdtable)
+    commandtable.update(cmdtable)

 def loadall(ui):
     result = ui.configitems("extensions")
diff -r cf8b8f62688a mercurial/hg.py
--- a/mercurial/hg.py   Mon Jun 25 21:23:24 2007 -0500
+++ b/mercurial/hg.py   Thu Jun 28 10:14:23 2007 -0500
@@ -10,7 +10,7 @@ from repo import *
 from repo import *
 from i18n import _
 import localrepo, bundlerepo, httprepo, sshrepo, statichttprepo
-import errno, lock, os, shutil, util, cmdutil
+import errno, lock, os, shutil, util, cmdutil, extensions
 import merge as _merge
 import verify as _verify

@@ -50,13 +50,11 @@ def islocal(repo):
             return False
     return repo.local()

-repo_setup_hooks = []
-
 def repository(ui, path='', create=False):
     """return a repository object for the specified path"""
     repo = _lookup(path).instance(ui, path, create)
     ui = getattr(repo, "ui", ui)
-    for hook in repo_setup_hooks:
+    for hook in extensions.setuphooks:
         hook(ui, repo)
     return repo

And because of this hidden circular reference you can't import
mercurial.bundlerepo if the demandimport line is removed from mercurial.commands.

Ok, I can reproduce the issue.
Actually, in the "long(hexlify(urandom(4)), 16)" expression, the problem was
with "urandom", not with "hexlify" as I wrongly assumed!

With Python 2.3.5:
>>> from mercurial import hg
>>> from os import urandom
>>> urandom(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "C:\ProgramFiles\Python235\Lib\site-packages\mercurial\demandimport.py",
line 70, in __call__
    raise TypeError("'unloaded module' object is not callable")
TypeError: 'unloaded module' object is not callable


This comes from the fact that in Python 2.3.5, urandom is not defined, and from
the normal import, one should get:

>>> from os import urandom
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ImportError: cannot import name urandom

The original code that breaks (in AccountManager's plugin for Trac) is the
following:

# os.urandom was added in Python 2.4
# try to fall back on reading from /dev/urandom on older Python versions
try:
    from os import urandom
except ImportError:
    from random import randrange
    def urandom(n):
        return ''.join([chr(randrange(256)) for _ in xrange(n)])


With Python 2.3.5 and demandimport.enabled(), the ImportError is not raised, so
the compatibility function urandom is not installed.

Temporary workaround: import extensions first.

Ahh, I see, there's a circular dependency:

localrepo imports extensions imports commands imports bundlerepo imports localrepo.

Without demandload, by the time bundlerepo refers to localrepo, localrepo hasn't
finished filling out its namespace, but is still marked as imported.

With demandload, we don't bother with the other imports until later and
everything works out nicely.

I'll try to fix this, but I'm more interested in the case where demandload breaks.

On Wed, Jun 27, 2007 at 06:23:41PM -0000, Matt Mackall wrote:
> 
> Matt Mackall <mpm@selenic.com> added the comment:
> 
> Having trouble reproducing this. Here's one attempt:

Hello!

I have to admit that I can't generate a minimal example that fails,
either. Definitely I get the error in Trac if I update to 0.9.4,
however.

Trying the following example:
------------------------------------------------------------------------
from mercurial import demandimport
demandimport.enable = lambda: None

from mercurial import hg
from mercurial.ui import ui
from mercurial.repo import RepoError
from mercurial.node import hex, short, nullid
from mercurial.util import pathto, cachefunc
from mercurial.cmdutil import walkchangerevs
------------------------------------------------------------------------

results in a backtrace as reported earlier. Clearly that hack isn't
going to work, commands.py doesn't like it.

The backtrace we got from Trac seems to be cleared up by doing a
demandimport.disable() after the mercurial imports; perhaps there
should be a demandimport.disable() call at the bottom of commands.py?

Having trouble reproducing this. Here's one attempt:

from mercurial import demandimport
demandimport.enable()

import binascii

print binascii

from binascii import hexlify

print binascii

print hexlify

print binascii

$ python dl
<unloaded module 'binascii'>
<unloaded module 'binascii'>
<built-in function hexlify>
<unloaded module 'binascii'>

Right, as Gregory said.

From the backtrace he send me, it seems that the following happens:

from mercurial import hg # in Trac, with demandimport disabled at that point
import localrepo, ... # in mercurial/hg.py
import os, revlog, time, util, extensions, ... # in mercurial/localrepo.py
# ...
class bundlerepository(localrepo.localrepository): # in mercurial/bundlerepo.py
-> AttributeError: 'module' object has no attribute 'localrepository'

Any hint about a workaround I could apply in TracMercurial to make it compatible
with the released Mercurial 0.9.4?

On Wed, Jun 27, 2007 at 12:39:40PM -0000, Thomas Arendsen Hein wrote:
>
> New submission from Thomas Arendsen Hein <thomas@intevation.de>:
>
> When importing mercurial.commands or mercurial.hgweb.hgwebdir_mod demandimport
> is automatically activated, but not for many other modules.
> This is not consistent, e.g. for hgweb.cgi or other tools importing parts of
> mercurial.
>
> My suggestion:
>
> Add "import mercurial.demandimport; mercurial.demandimport.enable()" to the hg
> and cgi scripts and let other applications decide for themselves whether they
> want to use it or not.
>
> Currently even our own setup.py uses
> mercurial.demandimport.enable = lambda: None
> to disable it ...

In fact, this is preventing us from upgrading to 0.9.4 for our Trac
installation --- 0.9.3 works fine but after upgrading we got
demandimport errors from unrelated modules:

# in mercurial/node.py, with `demandimport` enabled:

import binascii

# in Trac's acct_mgr/pwhash.py:

from binascii import hexlify
...
    v = long(hexlify(urandom(4)), 16)
# raises => TypeError: 'unloaded module' object is not callable

Christian from the Trac project attempted the following fix:

try:
    from mercurial import demandimport
    demandimport.enable = lambda : None
except ImportError:
    pass

Unfortunately this causes modules imported from mercurial to fail on
instantiation. Looking at the revision history from 0.9.3's
demandimport.py, all of the changes seem innocuous to me.

When importing mercurial.commands or mercurial.hgweb.hgwebdir_mod demandimport
is automatically activated, but not for many other modules.
This is not consistent, e.g. for hgweb.cgi or other tools importing parts of
mercurial.

My suggestion:

Add "import mercurial.demandimport; mercurial.demandimport.enable()" to the hg
and cgi scripts and let other applications decide for themselves whether they
want to use it or not.

Currently even our own setup.py uses
mercurial.demandimport.enable = lambda: None
to disable it ...