hgmo: serve AWS clients bundles from local S3 region (bug 1249197); r?fubar draft
authorGregory Szorc <gps@mozilla.com>
Thu, 18 Feb 2016 17:17:54 -0800
changeset 7282 c9643e6ea3aef306c2b7bf978d10092f514d0af3
parent 7281 dba06bd375b44115443f53668e6f3f414d45d2c9
push id647
push usergszorc@mozilla.com
push dateFri, 19 Feb 2016 01:18:31 +0000
reviewersfubar
bugs1249197
hgmo: serve AWS clients bundles from local S3 region (bug 1249197); r?fubar This commit modifies the hgmo extension to inspect the source IP address of HTTP requests for bundle clone and clone bundles manifests and compare against known AWS IP ranges. If the request is coming from an AWS region that we have S3 URLs for (identified by the presence of "ec2region=" in the manifest metadata), we filter the manifest to only advertise URLs in the same region. This guarantees an intra-region transfer without any client configuration. This is fast and free since intra-region S3 transfers don't cost anything! In addition, we also reorder the manifest to advertise the stream clone bundles first. We recommend clients in automation apply stream clone bundles (over the default gzip bundles) because they are the fastest mechanism to clone repositories (a stream clone bundle can apply at >80 MB/s). Tests for the new features have been added to the hgserver tests because we have existing tests that run the code for generating manifests and we want to use realistic values for testing. Documentation for the bundle hosting has been updated to reflect the change in behavior. MozReview-Commit-ID: 2ZaoJpaiqBU
ansible/roles/hg-web/files/hgrc
ansible/roles/hg-web/files/requirements-hgweb.txt
docs/hgmo/bundleclone.rst
hgext/hgmo/__init__.py
hgserver/tests/test-clonebundles.t
--- a/ansible/roles/hg-web/files/hgrc
+++ b/ansible/roles/hg-web/files/hgrc
@@ -52,8 +52,9 @@ bugzilla = s|((?:bug[\s#]*|b=#?|#)(\d{4,
 #[serverlog]
 #reporoot = /repo_local/mozilla/
 
 [bundleclone]
 pullmanifest=True
 
 [hgmo]
 mozbuildinfowrapper = /usr/bin/sudo /usr/local/bin/mozbuild-eval %repo%
+awsippath = /etc/mercurial/aws-ip-ranges.json
--- a/ansible/roles/hg-web/files/requirements-hgweb.txt
+++ b/ansible/roles/hg-web/files/requirements-hgweb.txt
@@ -1,7 +1,8 @@
+# sha256: k1cSgAzkdgcB2JrWd2Zs1SaR_S9vCzQMi0I5o8F5iKU
 ipaddress==1.0.16
 
 # sha256: CcVnBJw-MPeR2wz1k3NGx_81aN6t9OsdTi98gAAcs9Y
 Mercurial==3.6.2
 
 # sha256: fdlUDbl5fypbSR6v9jRGQwUFU-vLti93V2LL46A0PCU
 mod-wsgi==4.4.11
--- a/docs/hgmo/bundleclone.rst
+++ b/docs/hgmo/bundleclone.rst
@@ -1,72 +1,81 @@
 .. _hgmo_bundleclone:
 
 ==================================
 Cloning from Pre-Generated Bundles
 ==================================
 
 ``hg.mozilla.org`` supports offloading clone requests to pre-generated
-bundle files stored in Amazon S3. **This results in drastically reduced
-server load (which helps prevent outages due to accidental, excessive
-load) and frequently results in faster clone times.**
+bundle files stored in a CDN and Amazon S3. **This results in drastically
+reduced server load (which helps prevent outages due to accidental,
+excessive load) and frequently results in faster clone times.**
 
 How It Works
 ============
 
 When a Mercurial client clones a repository, it looks to see if the
 server is advertising a list of available, pre-generated bundle files.
 If it is, it looks at the list, finds the most appropriate entry,
 downloads and applies that bundle, then does the equivalent of an ``hg
 pull`` against the original Mercurial server to fetch new data since the
 time the bundle file was produced. The end result is a faster clone with
 drastically reduced load on the Mercurial server.
 
 Enabling
 ========
 
-If you are running Mercurial 3.6.1 or newer, support for cloning from
-pre-generated bundles is built-in to Mercurial itself. However, it
-requires enabling a config option::
+If you are running Mercurial 3.7 or newer, support for cloning from
+pre-generated bundles is built-in to Mercurial itself and enabled
+by default.
+
+If you are running Mercurial 3.6, support is built-in but requires
+enabling a config option::
 
    [experimental]
    clonebundles = true
 
 If you are running a Mercurial older than 3.6, first please consider
-upgrading to 3.6.1 or newer, as 3.6 contains a number of performance
-enhancements to cloning. If you absolutely must run a Mercurial older
-than 3.6, you can install the
+upgrading to 3.7 or newer, as newer versions contain a number of
+performance enhancements to cloning. If you absolutely must run a
+Mercurial older than 3.6, you can install the
 `bundleclone extension <https://hg.mozilla.org/hgcustom/version-control-tools/file/default/hgext/bundleclone/__init__.py>`_.
 Simply `download
 <https://hg.mozilla.org/hgcustom/version-control-tools/raw-file/default/hgext/bundleclone/__init__.py>`_
 that file then add the following to your global hgrc file (likely
 ``/etc/mercurial/hgrc``)::
 
    [extensions]
    bundleclone = /path/to/bundleclone.py
 
 .. tip::
 
    You can rename the ``__init__.py`` file as you see fit.
 
 Configuring
 ===========
 
-By default, the first entry in the bundles file list will be used. The
-server is configured so the first entry is the best choice for the most
-people. However, various audiences will want to prioritize certain
-bundles over others.
+hg.mozilla.org will advertise multiple bundles/URLs for each repository.
+Each listing varies by:
+
+* Bundle type
+* Server location
+
+By default, Mercurial uses the first entry in the server-advertised
+bundles list.
 
 Both the built-in *clone bundles* feature and *bundleclone* allow the
 client to define preferences of which bundles to fetch. The way this
 works is the client defines some key-value pairs in its config and
 bundles having these attributes will be upweighted.
 
-On ``hg.mozilla.org``, we define the following attributes are defined in
-the manifest:
+Bundle Attributes on hg.mozilla.org
+-----------------------------------
+
+On ``hg.mozilla.org``, following attributes are defined in the manifest:
 
 BUNDLESPEC (clonebundles only)
    This defines the type of bundle.
 
    We currently generate bundles with the following specifications:
    ``gzip-v1``, ``bzip2-v1``, ``none-packed1``.
 
 REQUIRESNI (clonebundles only)
@@ -99,16 +108,19 @@ stream (bundleclone only)
 
    This is captured by the ``BUNDLESPEC`` attribute in *clone bundles*.
 
 cdn (clonebundles and bundleclone)
    Indicates whether the URL is on a CDN. Value is ``true`` to indicate
    the URL is a CDN. All other values or undefined values are to be
    interpretted as not a CDN.
 
+Example Manifests
+-----------------
+
 Here is an example *clone bundles* manifest::
 
    https://hg.cdn.mozilla.net/mozilla-central/4a7526d26bd47ce2e01f938702b91c95424026ed.gzip.hg BUNDLESPEC=gzip-v1 REQUIRESNI=true cdn=true
    https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/4a7526d26bd47ce2e01f938702b91c95424026ed.gzip.hg BUNDLESPEC=gzip-v1 ec2region=us-west-2
    https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-central/4a7526d26bd47ce2e01f938702b91c95424026ed.gzip.hg BUNDLESPEC=gzip-v1 ec2region=us-east-1
    https://hg.cdn.mozilla.net/mozilla-central/4a7526d26bd47ce2e01f938702b91c95424026ed.bzip2.hg BUNDLESPEC=bzip2-v1 REQUIRESNI=true cdn=true
    https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/4a7526d26bd47ce2e01f938702b91c95424026ed.bzip2.hg BUNDLESPEC=bzip2-v1 ec2region=us-west-2
    https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-central/4a7526d26bd47ce2e01f938702b91c95424026ed.bzip2.hg BUNDLESPEC=bzip2-v1 ec2region=us-east-1
@@ -137,16 +149,19 @@ 2. A bzip2 bundle (smaller, but slower)
 3. A *streaming* bundle file (larger but faster)
 
 For each of these bundles, we upload them to 3 locations:
 
 1. CloudFront CDN
 2. S3 in us-west-2 region
 3. S3 in us-east-1 region
 
+Which Bundles to Prefer
+-----------------------
+
 The gzipped bundle hosted on CloudFront is the first entry and is thus
 preferred by clients by default. **This is optimized for developers on
 high speed network connections.**
 
 If you have a slower internet connection, you may want to prefer bzip2
 bundles. While they take several more minutes of CPU time to apply, this
 could be cancelled out from the shorter time required to download them.
 To prefer bzip2 bundles::
@@ -166,40 +181,42 @@ S3 or CloudFront at 1 Gbps speeds, you s
 in under 60s.::
 
    [experimental]
    clonebundleprefers = VERSION=packed1
 
    [bundleclone]
    prefers = stream=revlogv1
 
-If you are in EC2, you should **always** pin your EC2 region as the
-first entry. You should also prefer *stream bundle* mode, as network
-bandwidth is plentiful and clones will be faster. e.g.::
+Manifest Advertisement to AWS Clients
+-------------------------------------
 
-   [experimental]
-   clonebundleprefers = ec2region=us-west-2, VERSION=packed1
+If a client in Amazon Web Services (e.g. EC2) is requesting a bundle
+manifest and that client is in an AWS region where bundles are hosted
+in S3, the advertised manifest will only show S3 URLs for the same AWS
+region. In addition, stream clone bundles are the highest priority bundle.
 
-   [bundleclone]
-   prefers = ec2region=us-west-2, stream=revlogv1
+This behavior ensures that AWS transfer are intra-region (which means
+they are fast and don't result in a billable AWS event) and that ``hg
+clone`` completes as fast as possible (stream clone bundles are faster
+than gzip bundles).
 
 .. important::
 
-   If you have machinery in an EC2 region where we don't host bundles,
+   If you have machinery in an AWS region where we don't host bundles,
    please let us know. There's a good chance that establishing bundles
    in your region is cheaper than paying the cross-region transfer costs
    (intra-region transfer is free).
 
 Which Repositories Have Bundles Available
 =========================================
 
 Bundles are automatically generated for repositories that are high
 volume (in terms of repository size and clone frequency) or have a need
 for bundles.
 
-If you have the ``bundleclone`` extension installed and Mercurial doesn't
-print information about downloading a bundle file when you ``hg clone``
-from ``hg.mozilla.org``, bundles probably aren't being generated for
-that repository.
+The list of repositories with bundles enabled can be found at
+https://hg.cdn.mozilla.net/.
 
-If you think bundles should be made available, let a server operator
-know by filing a ``Developer Services :: hg.mozilla.org`` bug or by
-asking in #vcs on irc.mozilla.org.
+If you think bundles should be made available for a particular
+repository, let a server operator know by filing a
+``Developer Services :: hg.mozilla.org`` bug or by asking in #vcs
+on irc.mozilla.org.
--- a/hgext/hgmo/__init__.py
+++ b/hgext/hgmo/__init__.py
@@ -78,22 +78,28 @@ from mercurial import (
     cmdutil,
     commands,
     encoding,
     error,
     extensions,
     hg,
     revset,
     util,
+    wireproto,
 )
 from mercurial.hgweb import (
     webcommands,
     webutil,
 )
-from mercurial.hgweb.common import HTTP_OK
+from mercurial.hgweb.common import (
+    HTTP_OK,
+)
+from mercurial.hgweb.protocol import (
+    webproto,
+)
 
 
 OUR_DIR = os.path.dirname(__file__)
 ROOT = os.path.normpath(os.path.join(OUR_DIR, '..', '..'))
 execfile(os.path.join(OUR_DIR, '..', 'bootstrap.py'))
 
 import mozautomation.commitparser as commitparser
 import mozhg.mozbuildinfo as mozbuildinfo
@@ -419,17 +425,17 @@ def revset_automationrelevant(repo, subs
     revision and any ancestors that are part of the same push unioned with
     non-public ancestors.
     """
     s = revset.getset(repo, revset.fullreposet(repo), x)
     if len(s) > 1:
         raise util.Abort('can only evaluate single changeset')
 
     ctx = repo[s.first()]
-    revs = {ctx.rev()}
+    revs = set([ctx.rev()])
 
     # The pushlog is used to get revisions part of the same push as
     # the requested revision.
     pushlog = getattr(repo, 'pushlog', None)
     if pushlog:
         pushinfo = repo.pushlog.pushfromchangeset(ctx)
         for n in pushinfo[3]:
             pctx = repo[n]
@@ -517,16 +523,107 @@ def mozbuildinfocommand(ui, repo, *paths
         d = {'error': 'no moz.build info available'}
 
     # TODO send data to templater.
     # Use stable output and indentation to make testing easier.
     ui.write(json.dumps(d, indent=2, sort_keys=True))
     ui.write('\n')
     return
 
+
+def clonebundleswireproto(orig, repo, proto):
+    """Wraps wireproto.clonebundles."""
+    return processbundlesmanifest(repo, proto, orig(repo, proto))
+
+
+def bundleclonewireproto(orig, repo, proto):
+    """Wraps wireproto.bundles."""
+    return processbundlesmanifest(repo, proto, orig(repo, proto))
+
+
+def processbundlesmanifest(repo, proto, manifest):
+    """Processes a bundleclone/clonebundles manifest.
+
+    We examine source IP addresses and advertise URLs for the same
+    AWS region if the source is in AWS.
+    """
+    # Delay import because this extension can be run on local
+    # developer machines.
+    import ipaddress
+
+    if not isinstance(proto, webproto):
+        return manifest
+
+    awspath = repo.ui.config('hgmo', 'awsippath')
+    if not awspath:
+        return manifest
+
+    # Mozilla's load balancers add a X-Cluster-Client-IP header to identify the
+    # actual source IP, so prefer it.
+    sourceip = proto.req.env.get('HTTP_X_CLUSTER_CLIENT_IP',
+                              proto.req.env.get('REMOTE_ADDR'))
+    if not sourceip:
+        return manifest
+
+    origlines = [l for l in manifest.splitlines()]
+    # ec2 region not listed in any manifest entries. This is weird but it means
+    # there is nothing for us to do.
+    if not any('ec2region=' in l for l in origlines):
+        return manifest
+
+    try:
+        # constructor insists on unicode instances.
+        sourceip = ipaddress.IPv4Address(sourceip.decode('ascii'))
+
+        with open(awspath, 'rb') as fh:
+            awsdata = json.load(fh)
+
+        for ipentry in awsdata['prefixes']:
+            network = ipaddress.IPv4Network(ipentry['ip_prefix'])
+
+            if sourceip not in network:
+                continue
+
+            region = ipentry['region']
+
+            filtered = [l for l in origlines if 'ec2region=%s' % region in l]
+            # No manifest entries for this region. Ignore match and try others.
+            if not filtered:
+                continue
+
+            # We prioritize stream clone bundles to AWS clients because they are
+            # the fastest way to clone and we want our automation to be fast.
+            def mancmp(a, b):
+                packed = 'BUNDLESPEC=none-packed1'
+                stream = 'stream='
+
+                if packed in a and packed not in b:
+                    return -1
+                if packed in b and packed not in a:
+                    return 1
+
+                if stream in a and stream not in b:
+                    return -1
+                if stream in b and stream not in a:
+                    return 1
+
+                return 0
+
+            filtered = sorted(filtered, cmp=mancmp)
+
+            # We got a match. Write out the filtered manifest (with a trailing newline).
+            filtered.append('')
+            return '\n'.join(filtered)
+
+        return manifest
+
+    except Exception:
+        return manifest
+
+
 def filelog(orig, web, req, tmpl):
     """Wraps webcommands.filelog to provide pushlog metadata to template."""
 
     if hasattr(web.repo, 'pushlog'):
 
         class _tmpl(object):
 
             def __init__(self):
@@ -567,16 +664,26 @@ def extsetup(ui):
     extensions.wrapfunction(webcommands, 'filelog', filelog)
 
     revset.symbols['reviewer'] = revset_reviewer
     revset.safesymbols.add('reviewer')
 
     revset.symbols['automationrelevant'] = revset_automationrelevant
     revset.safesymbols.add('automationrelevant')
 
+    # Install IP filtering for bundle URLs.
+
+    # Build-in command from core Mercurial.
+    extensions.wrapcommand(wireproto.commands, 'clonebundles', clonebundleswireproto)
+
+    # Legacy bundleclone command. Need to support until all clients run
+    # 3.6+.
+    if 'bundles' in wireproto.commands:
+        extensions.wrapcommand(wireproto.commands, 'bundles', bundleclonewireproto)
+
     entry = extensions.wrapcommand(commands.table, 'serve', servehgmo)
     entry[1].append(('', 'hgmo', False,
                      'Run a server configured like hg.mozilla.org'))
 
     wrapper = ui.config('hgmo', 'mozbuildinfowrapper')
     if wrapper:
         if '"' in wrapper or "'" in wrapper:
             raise util.Abort('quotes may not appear in hgmo.mozbuildinfowrapper')
--- a/hgserver/tests/test-clonebundles.t
+++ b/hgserver/tests/test-clonebundles.t
@@ -64,9 +64,69 @@ We shouldn't see the message if we attem
   $ hg --config extensions.bundleclone=$TESTDIR/hgext/bundleclone clone -U ${HGWEB_0_URL}mozilla-central bundleclone
   downloading bundle https://hg.cdn.mozilla.net/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg
   abort: HTTP error fetching bundle: HTTP Error 403: Forbidden
   (consider contacting the server operator if this error persists)
   [255]
 
 #endif
 
+The full manifest is fetched normally
+
+  $ http --no-headers ${HGWEB_0_URL}mozilla-central?cmd=clonebundles
+  200
+  
+  https://hg.cdn.mozilla.net/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg BUNDLESPEC=gzip-v1 REQUIRESNI=true cdn=true
+  https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg BUNDLESPEC=gzip-v1 ec2region=us-west-2
+  https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg BUNDLESPEC=gzip-v1 ec2region=us-east-1
+  https://hg.cdn.mozilla.net/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.packed1.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1 REQUIRESNI=true cdn=true
+  https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.packed1.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1 ec2region=us-west-2
+  https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.packed1.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1 ec2region=us-east-1
+
+  $ http --no-headers ${HGWEB_0_URL}mozilla-central?cmd=bundles
+  200
+  
+  https://hg.cdn.mozilla.net/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg compression=gzip cdn=true requiresni=true
+  https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg ec2region=us-west-2 compression=gzip
+  https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg ec2region=us-east-1 compression=gzip
+  https://hg.cdn.mozilla.net/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.stream-legacy.hg stream=revlogv1 cdn=true requiresni=true
+  https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.stream-legacy.hg ec2region=us-west-2 stream=revlogv1
+  https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.stream-legacy.hg ec2region=us-east-1 stream=revlogv1
+
+Fetching with an AWS us-west-2 IP will limit to same region URLs
+
+  $ http --no-headers --request-header "X-Cluster-Client-IP: 54.245.168.15" ${HGWEB_0_URL}mozilla-central?cmd=clonebundles
+  200
+  
+  https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.packed1.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1 ec2region=us-west-2
+  https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg BUNDLESPEC=gzip-v1 ec2region=us-west-2
+  
+
+  $ http --no-headers --request-header "X-Cluster-Client-IP: 54.245.168.15" ${HGWEB_0_URL}mozilla-central?cmd=bundles
+  200
+  
+  https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.stream-legacy.hg ec2region=us-west-2 stream=revlogv1
+  https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg ec2region=us-west-2 compression=gzip
+  
+
+Fetching with an AWS IP from "other" region returns full list
+
+  $ http --no-headers --request-header "X-Cluster-Client-IP: 54.248.220.10" ${HGWEB_0_URL}mozilla-central?cmd=clonebundles
+  200
+  
+  https://hg.cdn.mozilla.net/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg BUNDLESPEC=gzip-v1 REQUIRESNI=true cdn=true
+  https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg BUNDLESPEC=gzip-v1 ec2region=us-west-2
+  https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg BUNDLESPEC=gzip-v1 ec2region=us-east-1
+  https://hg.cdn.mozilla.net/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.packed1.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1 REQUIRESNI=true cdn=true
+  https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.packed1.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1 ec2region=us-west-2
+  https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.packed1.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1 ec2region=us-east-1
+
+  $ http --no-headers --request-header "X-Cluster-Client-IP: 54.248.220.10" ${HGWEB_0_URL}mozilla-central?cmd=bundles
+  200
+  
+  https://hg.cdn.mozilla.net/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg compression=gzip cdn=true requiresni=true
+  https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg ec2region=us-west-2 compression=gzip
+  https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.gzip.hg ec2region=us-east-1 compression=gzip
+  https://hg.cdn.mozilla.net/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.stream-legacy.hg stream=revlogv1 cdn=true requiresni=true
+  https://s3-us-west-2.amazonaws.com/moz-hg-bundles-us-west-2/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.stream-legacy.hg ec2region=us-west-2 stream=revlogv1
+  https://s3-external-1.amazonaws.com/moz-hg-bundles-us-east-1/mozilla-central/77538e1ce4bec5f7aac58a7ceca2da0e38e90a72.stream-legacy.hg ec2region=us-east-1 stream=revlogv1
+
   $ hgmo clean