In main, resolving.

patch pushed by mpm to main as 7946503ec76e

See also http://www.selenic.com/mercurial/wiki/index.cgi/fncacheRepoFormat

updated patch:
http://selenic.com/pipermail/mercurial-devel/2008-September/007902.html

New patch series:
intro:   http://selenic.com/pipermail/mercurial-devel/2008-August/007774.html
patch 1: http://selenic.com/pipermail/mercurial-devel/2008-August/007775.html
patch 2: http://selenic.com/pipermail/mercurial-devel/2008-August/007776.html

See also http://selenic.com/pipermail/mercurial-devel/2008-August/007494.html
(latest patch series does not work any more)

Yet another patch series ["introduce fncache repo layout (rev 2)"] which would
solve Issue839 and Issue793:

intro: http://selenic.com/pipermail/mercurial-devel/2008-July/007333.html
patch 1: http://selenic.com/pipermail/mercurial-devel/2008-July/007335.html
patch 2: http://selenic.com/pipermail/mercurial-devel/2008-July/007334.html
patch 3: http://selenic.com/pipermail/mercurial-devel/2008-July/007336.html
patch 4: http://selenic.com/pipermail/mercurial-devel/2008-July/007337.html
patch 5: http://selenic.com/pipermail/mercurial-devel/2008-July/007338.html
patch 6: http://selenic.com/pipermail/mercurial-devel/2008-July/007339.html

Yet another patch series:

http://selenic.com/pipermail/mercurial-devel/2008-July/007206.html
http://selenic.com/pipermail/mercurial-devel/2008-July/007205.html
http://selenic.com/pipermail/mercurial-devel/2008-July/007207.html

would solve this issue and works fine.

Yet another patch was posted here:
http://selenic.com/pipermail/mercurial-devel/2008-July/007155.html
("filenamelog repository layout").

I've made a proof of concept patch (auxencode.patch), which uses the same
strategy as pmezards patch, but introduces a new repository requires entry
('auxencode').

I think pmezard's patch can't be done like that, because we need a new
requires entry as the decoding function must now operate on the full
range of encoded chars ('au~78' -> 'aux'), which existing releases of
Mercurial can't read (e.g. 1.0.1).
Also the old path decoding function util.decodefilename should remain
unchanged in the code so that old repositories are treated as before
(for old repositories 'au~78' in the store is an illegal path).

Updates for this patch may be available at
http://www.cadifra.com/cgi-bin/repos/hg-auxencode/file/tip/auxencode.patch
for some time.

mpm has said that he will only accept a repo layout change if the
long path problem (Issue839) is solved in the same layout change.

See also Issue839 ("Hg local store creates paths too long for Windows")

Some work has been done on a patch that can clone long and reserved paths
to windows NTFS and FAT32. This work has been abandoned, because explorer
can't delete nor copy directories containing reserved or long paths (>256).
Some extra tools would be needed for this. Also other tools (like winzip) have
serious issues with long paths (winzip silently ignores long path files when
archiving).

See: http://selenic.com/pipermail/mercurial-devel/2008-June/006884.html

Re "clock$": I just found out by consulting an old MSDN library DVD that
clock$ *was* a reserved name on Windows NT. It is not a reserved name
on newer Windows platforms.

It seems Microsoft has removed this information from the current
online version of http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx
("Naming a File")

Think the encoding done by pmezard's patches is very complicated. As I understand
it, he scans for the reserved names explicitly and encodes them.
I think this is brittle approach (is clock$ really reserved?).
Prepending a period to every component of the path is very simple.
It would also be very easy to read the old format (which doesn't have
periods at the beginning).

@mpm: where do you have that "clock$" from? It seems I can create files
and dirs with that name on Windows XP SP3 just fine.

File and directory names of the type "aux" and "aux.*" (this includes e.g.
"aux.foo.bar") are illegal on Windows, but ".aux" is ok. Also ".aux.foo.bar"
is ok.

So, a simple file/directory encoding strategy (for a new Mercurial repo
layout) would be to prepend a period to every directoryname/filename
inside the .hg dir.

If we could go the route of introducing a new repo format (for the longer term
resolution), we could use even more _ chars to encode revlog paths. We could
end every directory name with a _ and insert a _ before every period in the store.
I think this would cover the Windows reserved names.

That would be costly with regards to precious Windows max path length, though.
(as a variant, maybe the trailing _ could be omitted if there is an uppercase
char in the unencoded directory/filename)

See also http://mail.python.org/pipermail/python-dev/2007-October/075009.html
(link provided by pmezard)

This means, os.stat does *not* raise an exception for some reserved files on
some Python versions.

So relying on os.stat raising an exception is not an option.

Trying to open an inexistent file "aux.i" does not raise a Python exception, as
it normally would with an inexistent file.
After a subsequent read, the Python shell hangs forever (see transcript below).

'''
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on
win32
Type "copyright", "credits" or "license()" for more information.
    
IDLE 1.2.1      ==== No Subprocess ====
>>> import os
>>> os.stat('aux.i')
Traceback (most recent call last):
  File "<pyshell#3>", line 1, in <module>
    os.stat('aux.i')
WindowsError: [Error 87] The parameter is incorrect: 'aux.i'
>>> os.stat('nonexistent')
Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    os.stat('nonexistent')
WindowsError: [Error 2] The system cannot find the file specified: 'nonexistent'
>>> f = open('aux.i','r')
>>> f
<open file 'aux.i', mode 'r' at 0x013B2D58>
>>> s = f.read(10)
'''

BTW, open('aux.i', 'w') does neither create that file nor raise an exception.

See also
http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx ("Naming a File"),
which states:
'''
Do not use the following reserved device names for the name of a file:
CON, PRN, AUX, NUL,
COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9,
LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9
Also avoid these names followed immediately by an extension; for example, NUL.txt
is not recommended.
'''

If I do on FreeBSD:
% hg init; echo xxx > aux; hg add; hg ci -m1
and then clone that repo using Mercurial 034f444902d9 on Windows XP SP3:

> hg clone -U --debug http://www.cadifra.com/cgi-bin/repos/auxtest
using http://www.cadifra.com/cgi-bin/repos/auxtest
destination directory: auxtest
sending heads command
requesting all changes
sending changegroup command
adding changesets
add changeset 911089afebcb
adding manifests
adding file changes
adding aux revisions
added 1 changesets with 1 changes to 1 files
updating the branch cache

it looks like the cloning would have succeeded, but it didn't:
the directory .hg\store\data is empty (it should contain a file
aux.i). A subsequent

> hg -R auxtest verify --debug
repository uses revlog format 1
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files

hangs forever.

1_of_3-encode-win-paths.diff:
teach decodefilename how to decode all tilde escaped sequences. It may be useful
for any future path rewriting patch, like the following one.

2_of_3-encode-win-paths.diff:
encode the first character of forbidden Windows paths components. I cannot find
a version using a single sub() call because of overlapping matches. Suggestions
are welcomed.

3_of_3-encode-win-paths.diff:
Test encodefilename and decodefilename.

> We should probably hack this into the encodefilename system (via escaping the
first character) even if it'll break compatibility.

Above patches would break OpenSolaris repository upon upgrade. Did you have
something else in mind ? Do we have a way to at least warn about that (store
format version) ? If we are breaking things shouldn't we rethink the store
format with the "Too long filenames" issue in mind ?

adding nosy from issue805

As a first step, we should add some code that detects the error on clone.

This is similar to the case collision problem in that someone could do it by
accident on a project intended to be shared with Windows and have a very hard
time recovering. We should probably hack this into the encodefilename system
(via escaping the first character) even if it'll break compatibility.

It's unfortunate _buildencodefun() decoding map does not contain all bytes in
the "~%02x" form.

I hacked a patch encoding path components starting with one of these files and
ending with a '\..*' or nothing. I can make a complete roundtrip between windows
and macosx with a repository containing directories and files mixing these
special files. Still have to clean it up.

Does it sound stupid ?

Ahh, Windows. IIRC, other "reserved words" are nul, con, prn, comx, lptx, and
clock$. Working around this is hard, and the result will tend to be a mess.

Making the title less mysterious.

What's happening is the onnv-gate repository contains a "aux" directory. "AUX"
files/directories are forbidden under Windows and we don't handle them. At least
we know what's going wrong now.

Following thread
<http://mail.python.org/pipermail/python-dev/2007-November/075113.html> has more
details about it.

> Mercurial cannot do a final "update" of the working directory (because of the
aforementioned case-sensitivity), but the repository is still there, I can do
for example "hg view" on it. This behaviour should stay the same for every repo
out there - that's what "crossplatform" means in this case.

Sorry, I did not mean the clone error is acceptable, I will investigate as soon
as I have time. Just that without further knowledge of what were your intents
with onnv, it's usefulness looked compromised on case-preserving fs, that's all.
But it should be fine for review. I am also surprised these kind of repositories
raise case-collisions issues at all. It seems bad practice to me to play with
these kind of subtleties but I am no unix hacker.

> BTW, it is not possible to update on case-preserving file systems because:
> [...]
OK, but still I am able to, for example, clone full Linux kernel sources from
http://www2.kernel.org/hg/linux-2.6 under WinXP. Mercurial cannot do a final
"update" of the working directory (because of the aforementioned
case-sensitivity), but the repository is still there, I can do for example "hg
view" on it. This behaviour should stay the same for every repo out there -
that's what "crossplatform" means in this case. Either that, or at least some
meaningful error message with explanation would be nice.

I can reproduce it under WinXP on localhost with hg serve and crew tip. The
clone is stuck when applying file changes.

Clone succeeds under MacOSX.

BTW, it is not possible to update on case-preserving file systems because:
"""
abort: case-folding collision between
usr/src/cmd/lp/filter/postscript/font/devpost/HB and
usr/src/cmd/lp/filter/postscript/font/devpost/Hb
""".

Whenever I try:

hg clone -r 0 http://hg.kublai.com/opensolaris/onnv-gate

with Mercurial 0.9.4 or 0.9.5 on Windows XP SP2 (Polish), it either hangs
forever after downloading ~11MB or exits with message "transaction abort" or
with some message about denied access. I've tested it independently on two
different machines.

This is a Windows-only issue: with current Ubuntu Feisty and standard package
mercurial-0.9.4 everything is fine. After creating a local repo that way and
executing "hg serve" the same things occur after I try to clone from that repo
over a LAN with Windows XP.

Of course removing the "-r 0" parameter makes things even worse - it downloads
more (~23MB) but then hangs anyway.

It might be, that the repository of OpenSolaris is somehow incompatible with
Windows XP, but still Mercurial should give at least some explanation about what
is going on, and it should NOT hang itself instead, so I consider this a bug
regardless of the cause.