overlay: command for overlaying a repo into a subdirectory of another (bug 1331697); r?glob draft
authorGregory Szorc <gps@mozilla.com>
Thu, 19 Jan 2017 15:26:53 -0800
changeset 10353 a1f5b0167ae86324fc64152c78b6743dcd39b246
parent 10350 f5b58a19bb3c97b4104babc19a7ef8006715e568
child 10354 81a4944e6295ba1a168ffe8ddf95cc4d8bade084
push id1519
push userbmo:gps@mozilla.com
push dateThu, 09 Feb 2017 02:56:29 +0000
reviewersglob
bugs1331697
overlay: command for overlaying a repo into a subdirectory of another (bug 1331697); r?glob This commit introduces a Mercurial extension for applying changesets from 1 Mercurial repository into the sub-directory of another. The primary goal of this extension is to support ongoing vendoring of "external" projects/repos into a monorepo in a way that preserves history of the source repo while not convoluting the history of the destination repo. The mechanisms used cater strongly to Mozilla's repository management philosophy and the initial feature set is somewhat limited because more advanced features are not yet required by Mozilla. The command works by cloning a "source" repository inside the .hg directory of the destination. Requested changesets from that source repository are then applied one at a time into the destination repository. The files between the two changesets are unioned. The extension currently assumes that the destination directory being written to is read-only outside of the overlay process and will abort if there is file content mismatch. This property is important because it prevents divergence between the source repository and the directory in the destination. Divergence is bad because, well, recovering from it is hard. When there is no divergence, there is no potential for merge conflicts (which may require human resolution) and there is no situation where the diff of changeset X in the destination contains changes that weren't there in the source repo. In the future, we may provide a mechanism to "reconcile" the state of the destination directory with the last overlayed changeset. Until then, read-only prevails. The command keeps track of which changesets from the source have been overlayed by storing metadata in the hidden "extras" dictionary in changesets. Stored are the repository URL and revision the changeset was overlayed from. This allows subsequent overlay operations to find the last-overlayed changeset and quickly perform an incremental overlay. In theory, this metadata can be spoofed and an actor could confuse the overlay mechanism. Mitigation for this is deferred to server operators at this time. I'm not too concerned it will be an issue at Mozilla in the short to medium term. I don't think there are any security implications, since the metadata is only advisory to aid subsequent overlay operations. MozReview-Commit-ID: 7rKBb7xnU71
hgext/overlay/__init__.py
hgext/overlay/tests/helpers.sh
hgext/overlay/tests/test-overlay-basic.t
hgext/overlay/tests/test-overlay-copies.t
hgext/overlay/tests/test-overlay-dest-state.t
hgext/overlay/tests/test-overlay-errors.t
hgext/overlay/tests/test-overlay-incremental.t
new file mode 100644
--- /dev/null
+++ b/hgext/overlay/__init__.py
@@ -0,0 +1,343 @@
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+"""Synchronize a foreign repository into a sub-directory of another.
+
+``hg overlay`` is used to "overlay" the changesets of a remote,
+unrelated repository into a sub-directory of another.
+"""
+
+from __future__ import absolute_import
+
+import os
+
+from mercurial.i18n import _
+from mercurial.node import bin, hex, short
+from mercurial import (
+    cmdutil,
+    context,
+    error,
+    exchange,
+    filelog,
+    hg,
+    scmutil,
+    store,
+    util,
+)
+
+
+testedwith = '4.0'
+
+cmdtable = {}
+command = cmdutil.command(cmdtable)
+
+
+REVISION_KEY = 'subtree_revision'
+SOURCE_KEY = 'subtree_source'
+
+
+def _verifymanifestsequal(sourcerepo, sourcectx, destrepo, destctx, prefix):
+    assert prefix.endswith('/')
+
+    sourceman = sourcectx.manifest()
+    destman = destctx.manifest()
+
+    sourcefiles = set(sourceman.iterkeys())
+    destfiles = set(p[len(prefix):] for p in destman if p.startswith(prefix))
+
+    if sourcefiles ^ destfiles:
+        raise error.Abort(_('files mismatch between source and destiation: %s')
+                          % _(', ').join(sorted(destfiles ^ sourcefiles)),
+                          hint=_('destination must match previously imported '
+                                 'changeset (%s) exactly') %
+                               short(sourcectx.node()))
+
+    # The set of paths is the same. Now verify the contents are identical.
+    for sourcepath, sourcenode, sourceflags in sourceman.iterentries():
+        destpath = '%s%s' % (prefix, sourcepath)
+        destnode, destflags = destman.find(destpath)
+
+        if sourceflags != destflags:
+            raise error.Abort(_('file flags mismatch between source and '
+                                'destination for %s: %s != %s') %
+                              (sourcepath,
+                               sourceflags or _('(none)'),
+                               destflags or _('(none)')))
+
+        # We can't just compare the nodes because they are derived from
+        # content that may contain file paths in metadata, causing divergence
+        # between the two repos. So we compare all the content in the
+        # revisions.
+        sourcefl = sourcerepo.file(sourcepath)
+        destfl = destrepo.file(destpath)
+
+        if sourcefl.read(sourcenode) != destfl.read(destnode):
+            raise error.Abort(_('content mismatch between source (%s) '
+                                'and destination (%s) in %s') % (
+                short(sourcectx.node()), short(destctx.node()), destpath))
+
+        sourcetext = sourcefl.revision(sourcenode)
+        desttext = destfl.revision(destnode)
+        sourcemeta = filelog.parsemeta(sourcetext)[0]
+        destmeta = filelog.parsemeta(desttext)[0]
+
+        # Copy path needs to be normalized before comparison.
+        if destmeta is not None and destmeta.get('copy', '').startswith(prefix):
+            destmeta['copy'] = destmeta['copy'][len(prefix):]
+
+        # Copy revision may not be consistent across repositories because it
+        # can be influenced by the path in a parent revision's copy metadata.
+        # So ignore it.
+        if sourcemeta and 'copyrev' in sourcemeta:
+            del sourcemeta['copyrev']
+        if destmeta and 'copyrev' in destmeta:
+            del destmeta['copyrev']
+
+        if sourcemeta != destmeta:
+            raise error.Abort(_('metadata mismatch for file %s between source '
+                                'and dest: %s != %s') % (
+                                destpath, sourcemeta, destmeta))
+
+
+def _overlayrev(sourcerepo, sourceurl, sourcectx, destrepo, destctx,
+                prefix):
+    """Overlay a single commit into another repo."""
+    assert prefix.endswith('/')
+    assert len(sourcectx.parents()) < 2
+
+    sourceman = sourcectx.manifest()
+
+    def filectxfn(repo, memctx, path):
+        sourcepath = path[len(prefix):]
+        if sourcepath not in sourceman:
+            return None
+
+        node, flags = sourceman.find(sourcepath)
+        sourcefl = sourcerepo.file(sourcepath)
+        data = sourcefl.read(node)
+
+        copied = None
+        renamed = sourcefl.renamed(node)
+        if renamed:
+            copied = '%s%s' % (prefix, renamed[0])
+
+        return context.memfilectx(repo, path, data, islink='l' in flags,
+                                  isexec='x' in flags, copied=copied,
+                                  memctx=memctx)
+
+    parents = [destctx.node(), None]
+    files = ['%s%s' % (prefix, f) for f in sourcectx.files()]
+    extra = dict(sourcectx.extra())
+    extra[REVISION_KEY] = sourcectx.hex()
+    extra[SOURCE_KEY] = sourceurl
+
+    memctx = context.memctx(destrepo, parents, sourcectx.description(),
+                            files, filectxfn, user=sourcectx.user(),
+                            date=sourcectx.date(), extra=extra)
+
+    return memctx.commit()
+
+
+def _dooverlay(sourcerepo, sourceurl, sourcerevs, destrepo, destctx, prefix):
+    """Overlay changesets from one repository into another.
+
+    ``sourcerevs`` (iterable of revs) from ``sourcerepo`` will effectively
+    be replayed into ``destrepo`` on top of ``destctx``. File paths will be
+    added to the directory ``prefix``.
+
+    ``sourcerevs`` may include revisions that have already been overlayed.
+    If so, overlay will resume at the first revision not yet processed.
+    """
+    assert prefix
+    prefix = prefix.rstrip('/') + '/'
+
+    ui = destrepo.ui
+
+    sourcerevs.sort()
+
+    # Source revisions must be a contiguous, single DAG range.
+    left = set(sourcerevs)
+    left.remove(sourcerevs.last())
+    for ctx in sourcerepo[sourcerevs.last()].ancestors():
+        if not left:
+            break
+
+        try:
+            left.remove(ctx.rev())
+        except KeyError:
+            raise error.Abort(_('source revisions must be part of contiguous '
+                                'DAG range'))
+
+    if left:
+        raise error.Abort(_('source revisions must be part of same DAG head'))
+
+    sourcerevs = list(sourcerevs)
+
+    sourcecl = sourcerepo.changelog
+    allsourcehexes = set(hex(sourcecl.node(rev)) for rev in
+                         sourcecl.ancestors([sourcerevs[-1]], inclusive=True))
+
+    # Attempt to find an incoming changeset in dest and prune already processed
+    # source revisions.
+    lastsourcectx = None
+    for rev in destrepo.changelog.ancestors([destctx.rev()], inclusive=True):
+        ctx = destrepo[rev]
+        overlayed = ctx.extra().get(REVISION_KEY)
+
+        # Changesets that weren't imported or that didn't come from the source
+        # aren't important to us.
+        if not overlayed or overlayed not in allsourcehexes:
+            continue
+
+        lastsourcectx = sourcerepo[overlayed]
+
+        # If this imported changeset is in the set scheduled for import,
+        # we can prune it and all ancestors from the source set. Since
+        # sourcerevs is sorted and is a single DAG head, we can simply find
+        # the offset of the first seen rev and assume everything before
+        # has been imported.
+        try:
+            idx = sourcerevs.index(lastsourcectx.rev()) + 1
+            ui.write(_('%s already processed as %s; '
+                       'skipping %d/%d revisions\n' %
+                       (short(lastsourcectx.node()), short(ctx.node()),
+                        idx, len(sourcerevs))))
+            sourcerevs = sourcerevs[idx:]
+            break
+        except ValueError:
+            # Else the changeset in the destination isn't in the incoming set.
+            # This is OK iff the destination changeset is a conversion of
+            # the parent of the first incoming changeset.
+            firstsourcectx = sourcerepo[sourcerevs[0]]
+            if firstsourcectx.p1().hex() == overlayed:
+                break
+
+            raise error.Abort(_('first source changeset (%s) is not a child '
+                                'of last overlayed changeset (%s)') % (
+                short(firstsourcectx.node()), short(bin(overlayed))))
+
+    if not sourcerevs:
+        ui.write(_('no source revisions left to process\n'))
+        return
+
+    # We don't (yet) support overlaying merge commits.
+    for rev in sourcerevs:
+        ctx = sourcerepo[rev]
+        if len(ctx.parents()) > 1:
+            raise error.Abort(_('do not support overlaying merges: %s') %
+                              short(ctx.node()))
+
+    # If we previously performed an overlay, verify that changeset
+    # continuity is uninterrupted. We ensure the parent of the first source
+    # changeset matches the last imported changeset and that the state of
+    # files in the last imported changeset matches exactly the state of files
+    # in the destination changeset. If these conditions don't hold, the repos
+    # got out of sync. If we continued, the first overlayed changeset would
+    # have a diff that didn't match the source repository. In other words,
+    # the history wouldn't be accurate. So prevent that from happening.
+    if lastsourcectx:
+        if sourcerepo[sourcerevs[0]].p1() != lastsourcectx:
+            raise error.Abort(_('parent of initial source changeset does not '
+                                'match last overlayed changeset (%s)') %
+                              short(lastsourcectx.node()))
+
+        _verifymanifestsequal(sourcerepo, lastsourcectx, destrepo, destctx,
+                              prefix)
+
+    # All the validation is done. Proceed with the data conversion.
+    with destrepo.lock():
+        with destrepo.transaction('overlay'):
+            for i, rev in enumerate(sourcerevs):
+                ui.progress(_('revisions'), i + 1, total=len(sourcerevs))
+                sourcectx = sourcerepo[rev]
+                node = _overlayrev(sourcerepo, sourceurl, sourcectx,
+                                   destrepo, destctx, prefix)
+                summary = sourcectx.description().splitlines()[0]
+                ui.write('%s -> %s: %s\n' % (short(sourcectx.node()),
+                                             short(node), summary))
+                destctx = destrepo[node]
+
+            ui.progress(_('revisions'), None)
+
+
+def _mirrorrepo(ui, repo, url):
+    """Mirror a source repository into the .hg directory of another."""
+    u = util.url(url)
+    if u.islocal():
+        raise error.Abort(_('source repo cannot be local'))
+
+    # Remove scheme from path and normalize reserved characters.
+    path = url.replace('%s://' % u.scheme, '').replace('/', '_')
+    mirrorpath = repo.join(store.encodefilename(path))
+
+    peer = hg.peer(ui, {}, url)
+    mirrorrepo = hg.repository(ui, mirrorpath,
+                               create=not os.path.exists(mirrorpath))
+
+    missingheads = [head for head in peer.heads() if head not in mirrorrepo]
+    if missingheads:
+        ui.write(_('pulling %s into %s\n' % (url, mirrorpath)))
+        exchange.pull(mirrorrepo, peer)
+
+    return mirrorrepo
+
+
+@command('overlay', [
+    ('d', 'dest', '', _('destination changeset on top of which to overlay '
+                        'changesets')),
+    ('', 'into', '', _('directory in destination in which to add files')),
+], _('[-d REV] SOURCEURL [REVS]'))
+def overlay(ui, repo, sourceurl, revs=None, dest=None, into=None):
+    """Integrate contents of another repository.
+
+    This command essentially replays changesets from another repository into
+    this one. Unlike a simple pull + rebase, the files from the remote
+    repository are "overlayed" or unioned with the contents of the destination
+    repository.
+
+    The functionality of this command is nearly identical to what ``hg
+    transplant`` provides. However, the internal mechanism varies
+    substantially.
+
+    There are currently several restrictions to what can be imported:
+
+    * The imported changesets must be in a single DAG head
+    * The imported changesets (as evaluated by ``REVS``) must be a contiguous
+      DAG range.
+    * Importing merges is not supported.
+    * The state of the files in the destination directory/changeset must
+      exactly match the last imported changeset.
+
+    That last point is important: it means that this command can effectively
+    only be used for unidirectional syncing. In other words, the source
+    repository must be the single source of all changes to the destination
+    directory.
+
+    The restriction of states being identical is to ensure that changesets
+    in the source and destination are as similar as possible. For example,
+    if the file content in the destination did not match the source, then
+    the ``hg diff`` output for the next overlayed changeset would differ from
+    the source.
+    """
+    # We could potentially support this later.
+    if not into:
+        raise error.Abort(_('--into must be specified'))
+
+    if not revs:
+        revs = 'all()'
+
+    sourcerepo = _mirrorrepo(ui, repo, sourceurl)
+    sourcerevs = scmutil.revrange(sourcerepo, [revs])
+
+    if not sourcerevs:
+        raise error.Abort(_('unable to determine source revisions'))
+
+    if dest:
+        destctx = repo[dest]
+    else:
+        destctx = repo['tip']
+
+    # Backdoor for testing to force static URL.
+    sourceurl = ui.config('overlay', 'sourceurl', sourceurl)
+
+    _dooverlay(sourcerepo, sourceurl, sourcerevs, repo, destctx, into)
new file mode 100644
--- /dev/null
+++ b/hgext/overlay/tests/helpers.sh
@@ -0,0 +1,17 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+export HGUSER='Test User <someone@example.com>'
+
+cat >> $HGRCPATH << EOF
+[extensions]
+strip =
+overlay = $TESTDIR/hgext/overlay
+
+[diff]
+git = true
+
+[overlay]
+sourceurl = https://example.com/repo
+EOF
new file mode 100644
--- /dev/null
+++ b/hgext/overlay/tests/test-overlay-basic.t
@@ -0,0 +1,109 @@
+  $ . $TESTDIR/hgext/overlay/tests/helpers.sh
+
+  $ hg init source
+  $ cd source
+  $ mkdir dir0
+  $ echo dir0/file0 > dir0/file0
+  $ echo dir0/file1 > dir0/file1
+  $ hg -q commit -A -m 'add dir0/file0 and dir0/file1'
+  $ mkdir dir1
+  $ echo dir1/file0 > dir1/file0
+  $ hg -q commit -A -m 'add dir1/file0'
+  $ hg serve -d --pid-file hg.pid -p $HGPORT
+  $ cat hg.pid >> $DAEMON_PIDS
+
+  $ cd ..
+
+  $ hg init dest
+  $ cd dest
+  $ echo foo > foo
+  $ hg -q commit -A -m initial
+  $ hg -q up null
+
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  pulling http://localhost:$HGPORT into $TESTTMP/dest/.hg/localhost~3a* (glob)
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 2 changesets with 3 changes to 3 files
+  44791c369f4c -> c129882b47be: add dir0/file0 and dir0/file1
+  afdf9d98d53c -> 3c931698b680: add dir1/file0
+
+  $ hg log -p --debug
+  changeset:   2:3c931698b680b225f15c9a27fc0aee486afc11cb
+  tag:         tip
+  phase:       draft
+  parent:      1:c129882b47be0de27c3e32ab643aa36d193ccea7
+  parent:      -1:0000000000000000000000000000000000000000
+  manifest:    2:e6f7ff8522567635c175d22989b27eef6cb8df60
+  user:        Test User <someone@example.com>
+  date:        Thu Jan 01 00:00:00 1970 +0000
+  files+:      subdir/dir1/file0
+  extra:       branch=default
+  extra:       subtree_revision=afdf9d98d53cb160d4a61267450bf32a8d1aa534
+  extra:       subtree_source=https://example.com/repo
+  description:
+  add dir1/file0
+  
+  
+  diff --git a/subdir/dir1/file0 b/subdir/dir1/file0
+  new file mode 100644
+  --- /dev/null
+  +++ b/subdir/dir1/file0
+  @@ -0,0 +1,1 @@
+  +dir1/file0
+  
+  changeset:   1:c129882b47be0de27c3e32ab643aa36d193ccea7
+  phase:       draft
+  parent:      0:21e2edf037c2267b7c1d7a038d64bca58d5caa59
+  parent:      -1:0000000000000000000000000000000000000000
+  manifest:    1:29cd433f4d664a8e65770ff64d91830206f30e9c
+  user:        Test User <someone@example.com>
+  date:        Thu Jan 01 00:00:00 1970 +0000
+  files+:      subdir/dir0/file0 subdir/dir0/file1
+  extra:       branch=default
+  extra:       subtree_revision=44791c369f4cd3098f627ec7ef4a014946f5a5ae
+  extra:       subtree_source=https://example.com/repo
+  description:
+  add dir0/file0 and dir0/file1
+  
+  
+  diff --git a/subdir/dir0/file0 b/subdir/dir0/file0
+  new file mode 100644
+  --- /dev/null
+  +++ b/subdir/dir0/file0
+  @@ -0,0 +1,1 @@
+  +dir0/file0
+  diff --git a/subdir/dir0/file1 b/subdir/dir0/file1
+  new file mode 100644
+  --- /dev/null
+  +++ b/subdir/dir0/file1
+  @@ -0,0 +1,1 @@
+  +dir0/file1
+  
+  changeset:   0:21e2edf037c2267b7c1d7a038d64bca58d5caa59
+  phase:       draft
+  parent:      -1:0000000000000000000000000000000000000000
+  parent:      -1:0000000000000000000000000000000000000000
+  manifest:    0:9091aa5df980aea60860a2e39c95182e68d1ddec
+  user:        Test User <someone@example.com>
+  date:        Thu Jan 01 00:00:00 1970 +0000
+  files+:      foo
+  extra:       branch=default
+  description:
+  initial
+  
+  
+  diff --git a/foo b/foo
+  new file mode 100644
+  --- /dev/null
+  +++ b/foo
+  @@ -0,0 +1,1 @@
+  +foo
+  
+  $ hg files -r tip
+  foo
+  subdir/dir0/file0
+  subdir/dir0/file1
+  subdir/dir1/file0
new file mode 100644
--- /dev/null
+++ b/hgext/overlay/tests/test-overlay-copies.t
@@ -0,0 +1,76 @@
+  $ . $TESTDIR/hgext/overlay/tests/helpers.sh
+
+  $ hg init source
+  $ cd source
+  $ echo content > file-original
+  $ hg -q commit -A -m 'add file-original'
+  $ hg cp file-original file-copy
+  $ hg commit -A -m 'copy to file-copy'
+  $ hg mv file-original file-renamed
+  $ hg commit -m 'rename file-original to file-renamed'
+  $ hg serve -d --pid-file hg.pid -p $HGPORT
+  $ cat hg.pid >> $DAEMON_PIDS
+
+  $ cd ..
+
+  $ hg init dest
+  $ cd dest
+  $ echo foo > foo
+  $ hg -q commit -A -m initial
+  $ hg -q up null
+
+  $ hg overlay http://localhost:$HGPORT --into overlayed
+  pulling http://localhost:$HGPORT into $TESTTMP/dest/.hg/localhost~3a* (glob)
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 3 changesets with 3 changes to 3 files
+  6e554f89d70b -> b7adb4318010: add file-original
+  abfabb8b7304 -> fb3553af8eea: copy to file-copy
+  120fa44d9d88 -> 5159eec60fc8: rename file-original to file-renamed
+
+  $ hg log -p
+  changeset:   3:5159eec60fc8
+  tag:         tip
+  user:        Test User <someone@example.com>
+  date:        Thu Jan 01 00:00:00 1970 +0000
+  summary:     rename file-original to file-renamed
+  
+  diff --git a/overlayed/file-original b/overlayed/file-renamed
+  rename from overlayed/file-original
+  rename to overlayed/file-renamed
+  
+  changeset:   2:fb3553af8eea
+  user:        Test User <someone@example.com>
+  date:        Thu Jan 01 00:00:00 1970 +0000
+  summary:     copy to file-copy
+  
+  diff --git a/overlayed/file-original b/overlayed/file-copy
+  copy from overlayed/file-original
+  copy to overlayed/file-copy
+  
+  changeset:   1:b7adb4318010
+  user:        Test User <someone@example.com>
+  date:        Thu Jan 01 00:00:00 1970 +0000
+  summary:     add file-original
+  
+  diff --git a/overlayed/file-original b/overlayed/file-original
+  new file mode 100644
+  --- /dev/null
+  +++ b/overlayed/file-original
+  @@ -0,0 +1,1 @@
+  +content
+  
+  changeset:   0:21e2edf037c2
+  user:        Test User <someone@example.com>
+  date:        Thu Jan 01 00:00:00 1970 +0000
+  summary:     initial
+  
+  diff --git a/foo b/foo
+  new file mode 100644
+  --- /dev/null
+  +++ b/foo
+  @@ -0,0 +1,1 @@
+  +foo
+  
new file mode 100644
--- /dev/null
+++ b/hgext/overlay/tests/test-overlay-dest-state.t
@@ -0,0 +1,122 @@
+  $ . $TESTDIR/hgext/overlay/tests/helpers.sh
+
+  $ hg init source
+  $ cd source
+  $ echo 0 > foo
+  $ hg -q commit -A -m 'add foo'
+  $ echo 1 > bar
+  $ hg -q commit -A -m 'add bar'
+  $ hg cp foo foo-copy
+  $ hg commit -m 'copy foo to foo-copy'
+Chain copies so copyrev differs
+  $ hg cp foo-copy foo-copy2
+  $ hg commit -m 'copy foo-copy to foo-copy2'
+  $ hg serve -d --pid-file hg.pid -p $HGPORT
+  $ cat hg.pid >> $DAEMON_PIDS
+  $ cd ..
+
+  $ hg init dest
+  $ cd dest
+  $ echo root > root
+  $ hg -q commit -A -m initial
+
+First overlay works fine
+
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  pulling http://localhost:$HGPORT into $TESTTMP/dest/.hg/localhost~3a* (glob)
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 4 changesets with 4 changes to 4 files
+  bd685f66c1fc -> 81fcbcf78f0a: add foo
+  13a44cc39ddc -> 0aac34b31cb4: add bar
+  2bb8fd7676d0 -> 645c9fffdee6: copy foo to foo-copy
+  0f7e081c425c -> 4930b59d9987: copy foo-copy to foo-copy2
+
+Create a new changeset to import
+
+  $ cd ../source
+  $ echo 2 > baz
+  $ hg -q commit -A -m 'add baz'
+  $ cd ../dest
+
+Addition of file in destination fails precondition testing
+
+  $ hg -q up tip
+  $ echo extra > subdir/extra-file
+  $ hg -q commit -A -m 'add extra file'
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  pulling http://localhost:$HGPORT into $TESTTMP/dest/.hg/localhost~3a* (glob)
+  searching for changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 2 changes to 2 files
+  0f7e081c425c already processed as 4930b59d9987; skipping 4/5 revisions
+  abort: files mismatch between source and destiation: extra-file
+  (destination must match previously imported changeset (0f7e081c425c) exactly)
+  [255]
+
+  $ hg -q strip -r .
+
+Removal of file in destination fails precondition testing
+
+  $ hg rm subdir/bar
+  $ hg commit -m 'remove bar'
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  0f7e081c425c already processed as 4930b59d9987; skipping 4/5 revisions
+  abort: files mismatch between source and destiation: bar
+  (destination must match previously imported changeset (0f7e081c425c) exactly)
+  [255]
+
+  $ hg -q strip -r .
+
+File mode difference in destination fails precondition testing
+
+  $ chmod +x subdir/foo
+  $ hg commit -m 'make foo executable'
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  0f7e081c425c already processed as 4930b59d9987; skipping 4/5 revisions
+  abort: file flags mismatch between source and destination for foo: (none) != x
+  [255]
+
+  $ hg -q strip -r .
+
+File content difference in destination fails precondition testing
+
+  $ echo rewritten > subdir/bar
+  $ hg commit -m 'change bar'
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  0f7e081c425c already processed as 4930b59d9987; skipping 4/5 revisions
+  abort: content mismatch between source (0f7e081c425c) and destination (7874b1d840a6) in subdir/bar
+  [255]
+
+  $ hg -q strip -r .
+
+No copy metadata in dest fails precondition testing
+
+  $ hg rm subdir/foo-copy2
+  $ hg commit -m 'remove foo-copy2'
+  $ echo 0 > subdir/foo-copy2
+  $ hg -q commit -A -m 'create foop-copy2 without copy metadata'
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  0f7e081c425c already processed as 4930b59d9987; skipping 4/5 revisions
+  abort: metadata mismatch for file subdir/foo-copy2 between source and dest: {'copy': 'foo-copy'} != None
+  [255]
+
+  $ hg -q strip -r .
+  $ hg -q strip -r .
+
+Metadata mismatch between source and dest fails precondition testing
+
+  $ hg rm subdir/foo-copy2
+  $ hg commit -m 'remove foo-copy2'
+  $ hg cp root subdir/foo-copy2
+  $ echo 0 > subdir/foo-copy2
+  $ hg commit -m 'create foo-copy2 from different source'
+
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  0f7e081c425c already processed as 4930b59d9987; skipping 4/5 revisions
+  abort: metadata mismatch for file subdir/foo-copy2 between source and dest: {'copy': 'foo-copy'} != {'copy': 'root'}
+  [255]
new file mode 100644
--- /dev/null
+++ b/hgext/overlay/tests/test-overlay-errors.t
@@ -0,0 +1,96 @@
+  $ . $TESTDIR/hgext/overlay/tests/helpers.sh
+
+  $ hg init empty
+  $ hg -R empty serve -d --pid-file hg.pid -p $HGPORT
+  $ cat hg.pid >> $DAEMON_PIDS
+
+  $ hg init repo0
+  $ cd repo0
+  $ echo 0 > foo
+  $ hg -q commit -A -m initial
+  $ echo 1 > foo
+  $ hg commit -m 'head 1 commit 1'
+  $ echo 2 > foo
+  $ hg commit -m 'head 1 commit 2'
+  $ echo 3 > foo
+  $ hg commit -m 'head 1 commit 3'
+  $ hg -q up -r 0
+  $ echo 4 > foo
+  $ hg commit -m 'head 2 commit 1'
+  created new head
+  $ echo 5 > foo
+  $ hg commit -m 'head 2 commit 2'
+  $ hg merge -t :local 3
+  0 files updated, 1 files merged, 0 files removed, 0 files unresolved
+  (branch merge, don't forget to commit)
+  $ hg commit -m 'merge 3 into 5'
+  $ hg log -G -T '{node|short} {desc}'
+  @    775588bbd687 merge 3 into 5
+  |\
+  | o  ac6bba5999bc head 2 commit 2
+  | |
+  | o  09ef50e3bf32 head 2 commit 1
+  | |
+  o |  5272c3c4ef03 head 1 commit 3
+  | |
+  o |  38627e51950d head 1 commit 2
+  | |
+  o |  eb87a779cc67 head 1 commit 1
+  |/
+  o  af1e0a150cd4 initial
+  
+
+  $ hg serve -d --pid-file hg.pid -p $HGPORT1
+  $ cat hg.pid >> $DAEMON_PIDS
+  $ cd ..
+
+  $ hg init dest
+
+--into required
+
+  $ cd dest
+  $ hg overlay http://localhost:$HGPORT
+  abort: --into must be specified
+  [255]
+
+Local repos not accepted
+
+  $ hg overlay ../empty --into prefix
+  abort: source repo cannot be local
+  [255]
+
+No revisions is an error
+
+  $ hg overlay http://localhost:$HGPORT --into prefix
+  abort: unable to determine source revisions
+  [255]
+
+Non-contiguous revision range is an error
+
+  $ hg overlay http://localhost:$HGPORT1 'af1e0a150cd4 + ac6bba5999bc' --into prefix
+  pulling http://localhost:$HGPORT1 into $TESTTMP/dest/.hg/localhost~3a* (glob)
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 7 changesets with 7 changes to 1 files
+  abort: source revisions must be part of contiguous DAG range
+  [255]
+
+Multiple heads is an error
+
+  $ hg overlay http://localhost:$HGPORT1 '::5272c3c4ef03 + ::ac6bba5999bc' --into prefix
+  abort: source revisions must be part of same DAG head
+  [255]
+
+Cannot overlay merges
+
+  $ hg overlay http://localhost:$HGPORT1 --into prefix
+  abort: do not support overlaying merges: 775588bbd687
+  [255]
+
+Dest revision is invalid
+
+  $ hg overlay --dest foo http://localhost:$HGPORT1 af1e0a150cd4::tip --into prefix
+  abort: unknown revision 'foo'!
+  [255]
new file mode 100644
--- /dev/null
+++ b/hgext/overlay/tests/test-overlay-incremental.t
@@ -0,0 +1,174 @@
+  $ . $TESTDIR/hgext/overlay/tests/helpers.sh
+
+  $ hg init source
+  $ cd source
+  $ echo 0 > foo
+  $ hg -q commit -A -m 'source commit 0'
+  $ echo 1 > foo
+  $ hg commit -m 'source commit 1'
+  $ hg serve -d --pid-file hg.pid -p $HGPORT
+  $ cat hg.pid >> $DAEMON_PIDS
+  $ cd ..
+
+  $ hg init dest
+  $ cd dest
+  $ touch root
+  $ hg -q commit -A -m 'dest commit 0'
+
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  pulling http://localhost:$HGPORT into $TESTTMP/dest/.hg/localhost~3a* (glob)
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 2 changesets with 2 changes to 1 files
+  00f6e41c0e85 -> 680a5f65e0c3: source commit 0
+  c71ec8379b05 -> 81f80944e32d: source commit 1
+
+Incremental overlay will no-op since no new changesets
+
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  c71ec8379b05 already processed as 81f80944e32d; skipping 2/2 revisions
+  no source revisions left to process
+
+New changeset in source should get applied as expected
+
+  $ cd ../source
+  $ echo 2 > foo
+  $ hg commit -m 'source commit 2'
+  $ cd ../dest
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  pulling http://localhost:$HGPORT into $TESTTMP/dest/.hg/localhost~3a* (glob)
+  searching for changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  c71ec8379b05 already processed as 81f80944e32d; skipping 2/3 revisions
+  60f2998d907d -> 50fab12f8664: source commit 2
+
+  $ hg log -G -T '{node|short} {desc}'
+  o  50fab12f8664 source commit 2
+  |
+  o  81f80944e32d source commit 1
+  |
+  o  680a5f65e0c3 source commit 0
+  |
+  @  cb699e5348c1 dest commit 0
+  
+
+New changeset in source and dest results in being applied on latest in dest
+
+  $ cd ../source
+  $ echo 3 > foo
+  $ hg commit -m 'source commit 3'
+  $ cd ../dest
+
+  $ hg -q up tip
+  $ echo 'source 2' > root
+  $ hg commit -m 'dest commit 1'
+
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  pulling http://localhost:$HGPORT into $TESTTMP/dest/.hg/localhost~3a* (glob)
+  searching for changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  60f2998d907d already processed as 50fab12f8664; skipping 3/4 revisions
+  2d54a6016dfe -> 3b62843da7a4: source commit 3
+
+  $ hg log -G -T '{node|short} {desc}'
+  o  3b62843da7a4 source commit 3
+  |
+  @  504ce2b98c14 dest commit 1
+  |
+  o  50fab12f8664 source commit 2
+  |
+  o  81f80944e32d source commit 1
+  |
+  o  680a5f65e0c3 source commit 0
+  |
+  o  cb699e5348c1 dest commit 0
+  
+
+Overlaying onto a head without all changesets will pick up where it left off
+
+  $ hg -q up 81f80944e32d
+  $ echo 'head 1' > root
+  $ hg commit -m 'head 1'
+  created new head
+  $ hg overlay http://localhost:$HGPORT --into subdir
+  c71ec8379b05 already processed as 81f80944e32d; skipping 2/4 revisions
+  60f2998d907d -> 13ddb87af500: source commit 2
+  2d54a6016dfe -> b06ac9515e0a: source commit 3
+
+  $ hg log -G -T '{node|short} {desc}'
+  o  b06ac9515e0a source commit 3
+  |
+  o  13ddb87af500 source commit 2
+  |
+  @  9af62c37d9de head 1
+  |
+  | o  3b62843da7a4 source commit 3
+  | |
+  | o  504ce2b98c14 dest commit 1
+  | |
+  | o  50fab12f8664 source commit 2
+  |/
+  o  81f80944e32d source commit 1
+  |
+  o  680a5f65e0c3 source commit 0
+  |
+  o  cb699e5348c1 dest commit 0
+  
+
+Source rev that has already been overlayed will fail
+
+  $ hg overlay http://localhost:$HGPORT 'c71ec8379b05::' --into subdir
+  2d54a6016dfe already processed as b06ac9515e0a; skipping 3/3 revisions
+  no source revisions left to process
+
+Source rev starting at next changeset will work
+
+  $ echo 'head 1 commit 2' > root
+  $ hg commit -m 'head 1 commit 2'
+  created new head
+  $ hg overlay http://localhost:$HGPORT '60f2998d907d::' --into subdir
+  60f2998d907d -> 0a78f301953e: source commit 2
+  2d54a6016dfe -> 4c9b5c9fec78: source commit 3
+
+  $ hg log -G -T '{node|short} {desc}'
+  o  4c9b5c9fec78 source commit 3
+  |
+  o  0a78f301953e source commit 2
+  |
+  @  c99f42f18be8 head 1 commit 2
+  |
+  | o  b06ac9515e0a source commit 3
+  | |
+  | o  13ddb87af500 source commit 2
+  |/
+  o  9af62c37d9de head 1
+  |
+  | o  3b62843da7a4 source commit 3
+  | |
+  | o  504ce2b98c14 dest commit 1
+  | |
+  | o  50fab12f8664 source commit 2
+  |/
+  o  81f80944e32d source commit 1
+  |
+  o  680a5f65e0c3 source commit 0
+  |
+  o  cb699e5348c1 dest commit 0
+  
+
+Selecting a source changeset that is missing parents in dest will fail
+
+  $ echo 'head 1 commit 3' > root
+  $ hg commit -m 'head 1 commit 3'
+  created new head
+  $ hg overlay http://localhost:$HGPORT '2d54a6016dfe::' --into subdir
+  abort: first source changeset (2d54a6016dfe) is not a child of last overlayed changeset (c71ec8379b05)
+  [255]