vcsreplicator: use remote value for incoming bookmarks (bug 1268963); r?glob draft
authorGregory Szorc <gps@mozilla.com>
Wed, 24 May 2017 17:10:16 -0700
changeset 11103 8cf6ea042fcee698427b8922a483ad4298d8ddfe
parent 11102 0a8b10d64a89294306724bb51f0cf702f17f8fb2
push id1685
push userbmo:gps@mozilla.com
push dateThu, 25 May 2017 00:37:14 +0000
reviewersglob
bugs1268963
vcsreplicator: use remote value for incoming bookmarks (bug 1268963); r?glob Previously, `hg pull` relied on Mercurial's built-in behavior for bookmark processing. The built-in behavior was to look at differences between local and remote bookmarks and reconcile the differences. Since the point of vcsreplicator is to run a mirror, the remote state should be the final local state. So Mercurial's default behavior isn't appropriate. This means we need to monkeypatch Mercurial. We introduce a minimal consumer-only extension to be installed on mirrors. The extension monkeypatches the bookmarks function that is called when local bookmarks are to be updated from a remote. The way bookmarks exchange works in Mercurial is that the full set of bookmarks from the server is fetched and applied. In other words, we never only receive bookmarks for just the changesets that are being changed. This means it is sufficient to blow away all local bookmarks and just let Mercurial's default behavior kick in to re-set them. There may be some corner cases when obsolescence is involved that this behavior breaks down. But obsolescence is experimental. So I'd rather get bookmark replication working for normal Mercurial and break obsolescence than spend a lot of time writing tests that may not uncover a problem. Writing this commit reminded me that I want to add an `hg pull --mirror` feature to upstream Mercurial. I /think/ I have buy-in on the idea. I just need to produce code. MozReview-Commit-ID: Kkx9vcXW1fW
pylib/vcsreplicator/tests/test-pushkey-bookmarks.t
pylib/vcsreplicator/vcsreplicator/consumer.py
pylib/vcsreplicator/vcsreplicator/consumerext.py
--- a/pylib/vcsreplicator/tests/test-pushkey-bookmarks.t
+++ b/pylib/vcsreplicator/tests/test-pushkey-bookmarks.t
@@ -284,24 +284,20 @@ Now test bookmark divergence
   vcsreplicator.consumer processing heartbeat-1 from partition 2 offset 19
   $ consumer --onetime
   vcsreplicator.consumer processing heartbeat-1 from partition 2 offset 20
   $ consumer --onetime
   vcsreplicator.consumer processing hg-changegroup-2 from partition 2 offset 21
   vcsreplicator.consumer pulling 1 heads (e84fdf206e79496713b3a56eae2e16c490475cc8) and 1 nodes from ssh://$DOCKER_HOSTNAME:$HGPORT/mozilla-central into $TESTTMP/repos/mozilla-central
   vcsreplicator.consumer pulled 1 changesets into $TESTTMP/repos/mozilla-central
 
-TODO bug 1268963 divergence should not occur
-
   $ hg -R $TESTTMP/repos/mozilla-central bookmarks
-     divergent                 3:e20ecd72ffa9
-     divergent@1               4:e84fdf206e79
+     divergent                 4:e84fdf206e79
      my-bookmark               3:e20ecd72ffa9
 
   $ hgmo exec hgweb0 /var/hg/venv_replication/bin/vcsreplicator-consumer --wait-for-no-lag /etc/mercurial/vcsreplicator.ini
   $ hgmo exec hgweb0 /var/hg/venv_replication/bin/hg -R /repo/hg/mozilla/mozilla-central bookmarks
-     divergent                 3:e20ecd72ffa9
-     divergent@1               4:e84fdf206e79
+     divergent                 4:e84fdf206e79
      my-bookmark               3:e20ecd72ffa9
 
 Cleanup
 
   $ hgmo clean
--- a/pylib/vcsreplicator/vcsreplicator/consumer.py
+++ b/pylib/vcsreplicator/vcsreplicator/consumer.py
@@ -15,16 +15,21 @@ import hglib
 from kafka.consumer import SimpleConsumer
 
 from .config import Config
 from .util import (
     consumer_offsets,
     wait_for_topic,
 )
 
+
+HERE = os.path.abspath(os.path.dirname(__file__))
+CONSUMER_EXT = os.path.join(HERE, 'consumerext.py')
+
+
 logger = logging.getLogger('vcsreplicator.consumer')
 
 MAX_BUFFER_SIZE = 104857600 # 100 MB
 
 MESSAGE_HEADER_V1 = b'1\n'
 
 
 class Consumer(SimpleConsumer):
@@ -254,17 +259,21 @@ def process_hg_sync(config, path, requir
 
         newtip = int(c.log('tip')[0].rev)
 
         logger.warn('pulled %d changesets into %s' % (newtip - oldtip,
                                                       local_path))
 
 
 def get_hg_client(path):
-    return hglib.open(path, encoding='UTF-8')
+    # This engages some client-specific functionality to mirror changes instead
+    # of using default Mercurial semantics.
+    configs = ['extensions.vcsreplicatorconsumer=%s' % CONSUMER_EXT]
+
+    return hglib.open(path, encoding='UTF-8', configs=configs)
 
 
 def update_hgrc(repo_path, content):
     """Update the .hg/hgrc file for a repo with content.
 
     If ``content`` is None, the file will be removed.
     """
     p = os.path.join(repo_path, '.hg', 'hgrc')
new file mode 100644
--- /dev/null
+++ b/pylib/vcsreplicator/vcsreplicator/consumerext.py
@@ -0,0 +1,42 @@
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+"""Mercurial extension for vcsreplicator consumers."""
+
+from __future__ import absolute_import
+
+from mercurial import (
+    bookmarks as bookmod,
+    extensions,
+)
+
+testedwith = '4.1 4.2'
+minimumhgversion = '4.1'
+
+
+def bookmarks_updatefromremote(orig, ui, repo, remotemarks, *args, **kwargs):
+    """Wraps bookmarks.updatefromremote() to force use remote values.
+
+    We change the arguments to make it appear that all remote bookmarks were
+    explicitly requested. That simulates a force pull and allows divergence
+    without the poor naming.
+    """
+    # By default, the remote bookmarks are compared to the local bookmarks
+    # and actions are taken. The action for diverged bookmarks is to create a
+    # new local bookmark of the name ``foo@N``. This obviously isn't wanted
+    # on a mirror.
+    #
+    # There are various ways we could trick the original function to do what we
+    # want. Our solution is to just remove all local bookmarks. Mercurial
+    # fetches the full set of bookmarks during pull operations. So this
+    # effectively simulates all incoming bookmarks as being new/canonical.
+    local_bookmarks = repo._bookmarks
+    for book in list(local_bookmarks):
+        del local_bookmarks[book]
+
+    return orig(ui, repo, remotemarks, *args, **kwargs)
+
+
+def extsetup(ui):
+    extensions.wrapfunction(bookmod, 'updatefromremote',
+                            bookmarks_updatefromremote)