perforce gateway

[Steve Borho]

On Tue, Dec 15, 2009 at 8:47 PM, Frank A. Kingswood
<frank at kingswood-consulting.co.uk> wrote:
> Steve Borho wrote:
>
>> I'm contemplating writing a set of scripts to serve as a gateway
>> between Mercurial and Perforce, and was hoping for some early feedback
>> to tell me if I'm heading towards a dead end.
>
> Hmm, well Perforce feels like a dead end to me, always.

I know what you mean.

>> To start with, we assume that you have a machine that is running a
>> conversion script (perhaps hg convert), converting perforce changesets
>> into mercurial changesets and that this is automated so you can pull
>> from this repository to get the latest perforce commits.  p4
>> changesets are added into a named branch, say 'p4trunk'.
>
> Why not 'default'?

Just to make my example more clear.  It could be default just as easily.

>>  We'll call
>> this the gateway repo.
>>
>> In your personal Mercurial repository, you use a named branch to track
>> the changes you are making.  Let's call this 'dev'.
>
> Is there a real need for the user to make a named branch? Matt recently
> showed a clever way to get the 'status' result for the difference between
> two clones. It would be nice if the user could just pull from the gateway
> repository.

I think it makes the work-flow more clear to innexperienced Mercurial
users (aka: recent perforce converts).  Named branches in the revision
history help illustrate the various lines of development.  They're not
strictly necessary though.

>> % hg p4merge dev
>
> hg p4push ? Because we're going to be pushing to the p4 repository.

essentially.  There's probably a better name for it.. I don't spend a
lot of time thinking of command names.

>> It would perform the following steps:
>> # validate dev has one head, and it is an ancestor of most recent p4
>> commit
>> # run 'hg status p4trunk:dev' to get accumulative list of modified,
>> added, removed files on dev branch
>> # run 'p4 edit' on modified file list
>> # run 'p4 rm' on the R (removed) files
>> # use 'hg revert -r dev' to get the modifed and added files from the dev
>> branch
>> # run 'p4 add' on the added files
>
> In which P4 context ("client spec") is this being done? The user only has a
> hg repository, right? And the gateway repository created by 'hg convert' is
> also not a P4 workarea.

Ah, this is a good question.  I've been using a p4 conversion script
for a few years now that works right out of a p4 client.  It
incrementally checks out each p4 changelist and checks it into
Mercurial.  Brute-force but it works.  I have not used the p4 support
in the convert extension yet.

> Now we could create a P4 workarea on the fly. At this point we have a list
> of affected files and directories, so we could create a p4 client spec with
> just these files and directories, do a p4 sync, *then* do the p4 edit/add/rm
> and finally delete the temporary files and the client spec. This would waste
> time downloading files from the P4 server which we already know will be
> overwritten by our changes. And it adds another race condition because the
> files we download from P4 could be a different version from what the most
> recent P4 change in mercurial is.
>
> Alternatively, we could have a "master" P4 workarea, but then all users'
> changes would be submitted as the same $P4USER. That might save quite a lot
> of money on P4 licenses... but would not be able to cope with multiple users
> attempting p4 submissions simultaneously. It does not avoid p4 syncing to
> ToT before marking the changes for submission either.

Yes, this would limit the usefulness of multiple users sharing a
single p4 gateway.

>> # submit this change with a comment like:
>>
>> merge from branch dev
>> Parent changeset ##############
>
> This "merge from branch dev" would not be very useful as a P4 log message.
> Remeber the P4 users don't see the intervening commits in hg. It would be
> better if the log messages of the changes were concatenated, for example
> like hg qfold does.

Certainly an option.  We could trivially allow either approach.

> I'd also prefer some markup in the P4 changelist comment that we're adding
> to reduce the likelyhood of misinterpreting some real user comments.
> [mercurial ids 398723895732454] is what I use in my scripts based on MQ, and
> the cvsps logic looks for {{...}} lines, which is probably safer.
>
> I don't think that you can assume that you are always allowed to submit.
> Imagine a hypothetical system where submitting changes to Perforce is
> strictly forbidden unless the P4 changelist has first been run through some
> semi-automatic regression script. So you must leave the changelist
> un-submitted for the user to do this regression check and submission
> manually.
> This would also allow the log message to be beautified by the user.

This gateway would not work for every p4 setup, that's for sure.  I
thought about doing this entirely on the client side.  Instead of
doing the merge on the gateway box, an extension on the user machine
would allow you to "push" to a local p4 client.  Something like this:

[p4push]
client = myclient
server = server:port

[paths]
default = http://p4mirror/hg/repo
default-push = p4://path/to/p4client/

The extension would monkey-patch 'push' and 'outgoing' and look for
p4:// urls, isolate the outgoing changesets, and then apply those
changes to the specified client as above and simply leave them there
for the user to commit (using whatever message they like).

However, this doesn't record the 'push' commit in p4 as a merge of any
kind on the mercurial side, so the user would have dangling hg heads
after each push.   In the end, the gateway approach seemed cleaner.

> (Of course such a hypothetical use case would have shocking race conditions,
> especially if the hypothetical regression test could take, say, 4 hours. All
> of this is completely hypothetical, you understand.)

Not so out of the ordinary, for large p4 shops.

>> Now we need some logic in the conversion script.  When it converts
>> this perforce commit, it needs to set the second parent to the node
>> hash in the commit message.   Now when the user pulls from the
>> gateway, they get a Mercurial changeset that actually merges their dev
>> branch with the p4trunk branch and they are free to continue working.
>
> OK, as the author of the P4 conversion I'll be happy to help implement this.
> It should be easier to use than MQ.

That was my thought as well.  If done right, this could be a wedge for
moving groups from p4 to hg entirely once they get used to it.

--
Steve Borho