pushlog: add tests for pulling entries with obsolete changesets (bug 1286426); r?glandium draft
authorGregory Szorc <gps@mozilla.com>
Thu, 14 Jul 2016 16:06:49 -0700
changeset 8970 b7d763081bf2873710a9aaa73c28483603c5dc28
parent 8969 6900f22f81a21153f82389478368b19a0d86de64
child 8971 e07fa19f4f70672710978271f594181b824b4b8e
push id1034
push userbmo:gps@mozilla.com
push dateTue, 19 Jul 2016 18:10:24 +0000
reviewersglandium
bugs1286426
pushlog: add tests for pulling entries with obsolete changesets (bug 1286426); r?glandium The test demonstrates sub-optimal behavior when pulling normally. This is because the remote server operates against the "visible" filter, which doesn't expose obsoleted changesets. We have a hack in the replication extension used on hg.mozilla.org to allow replication access to the unfiltered repository on the server, thus ensuring obsoleted changesets are transferred to clients during regular `hg pull` operations. Tests for this will be added in a different commit. We also took the opportunity to update the docs about what pushlog pulling does and when it can fail. MozReview-Commit-ID: ERInCX0yLRY
hgext/pushlog/__init__.py
hgext/pushlog/tests/test-pull-obsolete.t
--- a/hgext/pushlog/__init__.py
+++ b/hgext/pushlog/__init__.py
@@ -82,17 +82,23 @@ def pushlogwireproto(repo, proto, firstp
         for pushid, who, when, nodes in repo.pushlog.pushes(startid=firstpush):
             lines.append('%d %s %d %s' % (pushid, who, when, ' '.join(nodes)))
 
         return '\n'.join(lines)
     except Exception as e:
         return '\n'.join(['0', str(e)])
 
 def exchangepullpushlog(orig, pullop):
-    """This is called during pull to fetch pushlog data."""
+    """This is called during pull to fetch pushlog data.
+
+    The goal of this function is to replicate the entire pushlog. This is
+    in contrast to replicating only the pushlog data for changesets the
+    client has pulled. Put another way, this attempts complete replication
+    as opposed to partial, hole-y replication.
+    """
     # check stepsdone for future compatibility with bundle2 pushlog exchange.
     res = orig(pullop)
 
     if 'pushlog' in pullop.stepsdone or not pullop.remote.capable('pushlog'):
         return res
 
     repo = pullop.repo
     fetchfrom = repo.pushlog.lastpushid() + 1
@@ -106,19 +112,30 @@ def exchangepullpushlog(orig, pullop):
         raise Abort('error fetching pushlog: unexpected response: %s\n' %
             statusline)
 
     pushes = []
     for line in lines:
         pushid, who, when, nodes = line.split(' ', 3)
         nodes = [bin(n) for n in nodes.split()]
 
-        # Verify incoming changesets are known and stop processing when we see
-        # an unknown changeset. This can happen when we're pulling a former
-        # head instead of all changesets.
+        # We stop processing if there is a reference to an unknown changeset.
+        # This can happen in a few scenarios.
+        #
+        # Since the server streams *all* pushlog entries (from a starting
+        # number), it could send pushlog entries for changesets the client
+        # didn't request or were pushed since the client started pulling.
+        #
+        # If the remote repo contains obsolete changesets, we may see a
+        # reference to a hidden changeset.
+        #
+        # This is arguably not the desirable behavior: pushlog replication
+        # should be robust. However, doing things this way helps defend
+        # against pushlog "corruption" since inserting references to unknown
+        # changesets into the database is dangerous.
         try:
             [repo[n] for n in nodes]
         except error.RepoLookupError:
             repo.ui.warn('received pushlog entry for unknown changeset; ignoring\n')
             break
 
         pushes.append((int(pushid), who, int(when), nodes))
 
new file mode 100644
--- /dev/null
+++ b/hgext/pushlog/tests/test-pull-obsolete.t
@@ -0,0 +1,139 @@
+  $ . $TESTDIR/hghooks/tests/common.sh
+
+  $ cat >> $HGRCPATH << EOF
+  > [ui]
+  > ssh = python "$TESTDIR/pylib/mercurial-support/dummyssh"
+  > [experimental]
+  > evolution = all
+  > [extensions]
+  > rebase =
+  > EOF
+
+  $ export USER=hguser
+  $ hg init server
+  $ cd server
+  $ cat >> .hg/hgrc << EOF
+  > [extensions]
+  > pushlog = $TESTDIR/hgext/pushlog
+  > [phases]
+  > publish = false
+  > EOF
+  $ cd ..
+
+  $ hg -q clone ssh://user@dummy/$TESTTMP/server client
+  $ cd client
+  $ touch foo
+  $ hg -q commit -A -m initial
+  $ hg push
+  pushing to ssh://user@dummy/$TESTTMP/server
+  searching for changes
+  remote: adding changesets
+  remote: adding manifests
+  remote: adding file changes
+  remote: added 1 changesets with 1 changes to 1 files
+  remote: recorded push in pushlog
+
+  $ touch file0
+  $ hg -q commit -A -m file0
+  $ hg -q push
+  $ hg -q up -r 0
+  $ touch file1
+  $ hg -q commit -A -m file1
+  $ hg -q push -f
+  $ hg rebase -s . -d 1
+  rebasing 2:80c2c663cb83 "file1" (tip)
+  $ hg push
+  pushing to ssh://user@dummy/$TESTTMP/server
+  searching for changes
+  remote: adding changesets
+  remote: adding manifests
+  remote: adding file changes
+  remote: added 1 changesets with 0 changes to 1 files
+  remote: recorded push in pushlog
+  remote: 1 new obsolescence markers
+
+  $ hg --hidden log -G
+  @  changeset:   3:a129f82339bb
+  |  tag:         tip
+  |  parent:      1:ae13d9da6966
+  |  user:        test
+  |  date:        Thu Jan 01 00:00:00 1970 +0000
+  |  summary:     file1
+  |
+  | x  changeset:   2:80c2c663cb83
+  | |  parent:      0:96ee1d7354c4
+  | |  user:        test
+  | |  date:        Thu Jan 01 00:00:00 1970 +0000
+  | |  summary:     file1
+  | |
+  o |  changeset:   1:ae13d9da6966
+  |/   user:        test
+  |    date:        Thu Jan 01 00:00:00 1970 +0000
+  |    summary:     file0
+  |
+  o  changeset:   0:96ee1d7354c4
+     user:        test
+     date:        Thu Jan 01 00:00:00 1970 +0000
+     summary:     initial
+  
+
+  $ cd ..
+
+Server pushlog should have 4 pushes and push from hidden changeset (80c2c663cb83)
+
+  $ dumppushlog server
+  ID: 1; user: hguser; Date: \d+; Rev: 0; Node: 96ee1d7354c4ad7372047672c36a1f561e3a6a4c (re)
+  ID: 2; user: hguser; Date: \d+; Rev: 1; Node: ae13d9da6966307c98b60987fb4fedc2e2f29736 (re)
+  ID: 3; user: hguser; Date: \d+; Rev: 2; Node: 80c2c663cb8364f6898662a8379cb25df3ebe719 (re)
+  ID: 4; user: hguser; Date: \d+; Rev: 3; Node: a129f82339bb933c4d72353c44bb29eb685f3d1e (re)
+
+Cloning normally will receive obsolete data
+
+  $ hg clone -U ssh://user@dummy/$TESTTMP/server clone-obsolete1
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 3 changesets with 3 changes to 3 files
+  1 new obsolescence markers
+
+Default behavior of pushlog is to stop applying incoming push data when it sees
+an unknown changeset. Since hidden changesets aren't transferred normally,
+pushlog will stop replicating when it encounters a hidden changeset.
+
+  $ hg -R clone-obsolete1 --config extensions.pushlog=$TESTDIR/hgext/pushlog pull
+  pulling from ssh://user@dummy/$TESTTMP/server
+  searching for changes
+  no changes found
+  received pushlog entry for unknown changeset; ignoring
+  added 2 pushes
+
+Pushlog stops at 80c2c663cb83 because it is hidden
+
+  $ dumppushlog clone-obsolete1
+  ID: 1; user: hguser; Date: \d+; Rev: 0; Node: 96ee1d7354c4ad7372047672c36a1f561e3a6a4c (re)
+  ID: 2; user: hguser; Date: \d+; Rev: 1; Node: ae13d9da6966307c98b60987fb4fedc2e2f29736 (re)
+
+An uncompressed clone transfers obsolete changesets and markers
+
+  $ hg clone -U --uncompressed ssh://user@dummy/$TESTTMP/server clone-obsolete2
+  streaming all changes
+  5 files to transfer, * KB of data (glob)
+  transferred 1.19 KB in 0.0 seconds (*) (glob)
+  searching for changes
+  no changes found
+  1 new obsolescence markers
+
+The pushlog should pull cleanly because hidden changesets are present locally
+
+  $ hg -R clone-obsolete2 --config extensions.pushlog=$TESTDIR/hgext/pushlog pull
+  pulling from ssh://user@dummy/$TESTTMP/server
+  searching for changes
+  no changes found
+  added 4 pushes
+
+  $ dumppushlog clone-obsolete2
+  ID: 1; user: hguser; Date: \d+; Rev: 0; Node: 96ee1d7354c4ad7372047672c36a1f561e3a6a4c (re)
+  ID: 2; user: hguser; Date: \d+; Rev: 1; Node: ae13d9da6966307c98b60987fb4fedc2e2f29736 (re)
+  ID: 3; user: hguser; Date: \d+; Rev: 2; Node: 80c2c663cb8364f6898662a8379cb25df3ebe719 (re)
+  ID: 4; user: hguser; Date: \d+; Rev: 3; Node: a129f82339bb933c4d72353c44bb29eb685f3d1e (re)