vcsreplicator: use remote value for incoming bookmarks (
bug 1268963); r?glob
Previously, `hg pull` relied on Mercurial's built-in behavior for
bookmark processing. The built-in behavior was to look at differences
between local and remote bookmarks and reconcile the differences.
Since the point of vcsreplicator is to run a mirror, the remote state
should be the final local state. So Mercurial's default behavior isn't
appropriate. This means we need to monkeypatch Mercurial.
We introduce a minimal consumer-only extension to be installed on
mirrors. The extension monkeypatches the bookmarks function that
is called when local bookmarks are to be updated from a remote.
The way bookmarks exchange works in Mercurial is that the full
set of bookmarks from the server is fetched and applied. In other
words, we never only receive bookmarks for just the changesets
that are being changed. This means it is sufficient to blow away
all local bookmarks and just let Mercurial's default behavior
kick in to re-set them. There may be some corner cases when
obsolescence is involved that this behavior breaks down. But
obsolescence is experimental. So I'd rather get bookmark replication
working for normal Mercurial and break obsolescence than spend a
lot of time writing tests that may not uncover a problem.
Writing this commit reminded me that I want to add an
`hg pull --mirror` feature to upstream Mercurial. I /think/ I have
buy-in on the idea. I just need to produce code.
MozReview-Commit-ID: Kkx9vcXW1fW
--- a/pylib/vcsreplicator/tests/test-pushkey-bookmarks.t
+++ b/pylib/vcsreplicator/tests/test-pushkey-bookmarks.t
@@ -284,24 +284,20 @@ Now test bookmark divergence
vcsreplicator.consumer processing heartbeat-1 from partition 2 offset 19
$ consumer --onetime
vcsreplicator.consumer processing heartbeat-1 from partition 2 offset 20
$ consumer --onetime
vcsreplicator.consumer processing hg-changegroup-2 from partition 2 offset 21
vcsreplicator.consumer pulling 1 heads (e84fdf206e79496713b3a56eae2e16c490475cc8) and 1 nodes from ssh://$DOCKER_HOSTNAME:$HGPORT/mozilla-central into $TESTTMP/repos/mozilla-central
vcsreplicator.consumer pulled 1 changesets into $TESTTMP/repos/mozilla-central
-TODO bug 1268963 divergence should not occur
-
$ hg -R $TESTTMP/repos/mozilla-central bookmarks
- divergent 3:e20ecd72ffa9
- divergent@1 4:e84fdf206e79
+ divergent 4:e84fdf206e79
my-bookmark 3:e20ecd72ffa9
$ hgmo exec hgweb0 /var/hg/venv_replication/bin/vcsreplicator-consumer --wait-for-no-lag /etc/mercurial/vcsreplicator.ini
$ hgmo exec hgweb0 /var/hg/venv_replication/bin/hg -R /repo/hg/mozilla/mozilla-central bookmarks
- divergent 3:e20ecd72ffa9
- divergent@1 4:e84fdf206e79
+ divergent 4:e84fdf206e79
my-bookmark 3:e20ecd72ffa9
Cleanup
$ hgmo clean
--- a/pylib/vcsreplicator/vcsreplicator/consumer.py
+++ b/pylib/vcsreplicator/vcsreplicator/consumer.py
@@ -15,16 +15,21 @@ import hglib
from kafka.consumer import SimpleConsumer
from .config import Config
from .util import (
consumer_offsets,
wait_for_topic,
)
+
+HERE = os.path.abspath(os.path.dirname(__file__))
+CONSUMER_EXT = os.path.join(HERE, 'consumerext.py')
+
+
logger = logging.getLogger('vcsreplicator.consumer')
MAX_BUFFER_SIZE = 104857600 # 100 MB
MESSAGE_HEADER_V1 = b'1\n'
class Consumer(SimpleConsumer):
@@ -254,17 +259,21 @@ def process_hg_sync(config, path, requir
newtip = int(c.log('tip')[0].rev)
logger.warn('pulled %d changesets into %s' % (newtip - oldtip,
local_path))
def get_hg_client(path):
- return hglib.open(path, encoding='UTF-8')
+ # This engages some client-specific functionality to mirror changes instead
+ # of using default Mercurial semantics.
+ configs = ['extensions.vcsreplicatorconsumer=%s' % CONSUMER_EXT]
+
+ return hglib.open(path, encoding='UTF-8', configs=configs)
def update_hgrc(repo_path, content):
"""Update the .hg/hgrc file for a repo with content.
If ``content`` is None, the file will be removed.
"""
p = os.path.join(repo_path, '.hg', 'hgrc')
new file mode 100644
--- /dev/null
+++ b/pylib/vcsreplicator/vcsreplicator/consumerext.py
@@ -0,0 +1,42 @@
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+"""Mercurial extension for vcsreplicator consumers."""
+
+from __future__ import absolute_import
+
+from mercurial import (
+ bookmarks as bookmod,
+ extensions,
+)
+
+testedwith = '4.1 4.2'
+minimumhgversion = '4.1'
+
+
+def bookmarks_updatefromremote(orig, ui, repo, remotemarks, *args, **kwargs):
+ """Wraps bookmarks.updatefromremote() to force use remote values.
+
+ We change the arguments to make it appear that all remote bookmarks were
+ explicitly requested. That simulates a force pull and allows divergence
+ without the poor naming.
+ """
+ # By default, the remote bookmarks are compared to the local bookmarks
+ # and actions are taken. The action for diverged bookmarks is to create a
+ # new local bookmark of the name ``foo@N``. This obviously isn't wanted
+ # on a mirror.
+ #
+ # There are various ways we could trick the original function to do what we
+ # want. Our solution is to just remove all local bookmarks. Mercurial
+ # fetches the full set of bookmarks during pull operations. So this
+ # effectively simulates all incoming bookmarks as being new/canonical.
+ local_bookmarks = repo._bookmarks
+ for book in list(local_bookmarks):
+ del local_bookmarks[book]
+
+ return orig(ui, repo, remotemarks, *args, **kwargs)
+
+
+def extsetup(ui):
+ extensions.wrapfunction(bookmod, 'updatefromremote',
+ bookmarks_updatefromremote)