Bug 1232442 - Seed images with a stream clone of the unified Firefox repo; r?catlee draft
authorGregory Szorc <gps@mozilla.com>
Thu, 09 Jun 2016 17:12:04 -0700
changeset 3806 7636388a4c8aed7aadebc6b132b608fda9d6304c
parent 3805 f8780f7ccecead2c6984454ce17d22db7f31fb47
push id2471
push userbmo:gps@mozilla.com
push dateWed, 06 Jul 2016 22:07:36 +0000
reviewerscatlee
bugs1232442, 1270317, 1249197, 1270951
Bug 1232442 - Seed images with a stream clone of the unified Firefox repo; r?catlee The https://hg.mozilla.org/mozilla-unified repo is a single Mercurial repository with relevant heads from all important repos (mozilla-central, mozilla-inbound, mozilla-aurora, mozilla-release, esrs, etc). The repository is encoded as generaldelta, which means it is smaller than mozilla-central (even though it contains 30,000+ more commits!) Recent work in automation (namely bug 1270317) changed automation to always use shared, pooled storage for Mercurial repos. This meant that we only need a single store for Firefox repos. When this change was made, we didn't change AMI seeding. This means that a worker would clone the Firefox repo on first job that needed it. This is obviously inefficient. This commit changes the shared repo seeding so the pooled/shared repo now populated in automation is seeded at AMI generation time. So on first job run, most commits will be present and we'll only do an incremental pull. This restores the behavior from before bug 1270317 landed. There are multiple benefits: 1) Shared repo population will complete quicker (because we're only populating 1 repo) 2) We'll use less disk space for local repos (because we will only populate 1) 3) Jobs will start faster since most commits from most Firefox repos will already be present in the pre-populated shared repo. The previous version of this file had code to map the instance's current availability zone to an S3 bucket. As of bug 1249197, hg.mozilla.org advertises bundle URLs to the appropriate S3 endpoint based on the requesting IP. This favors same-AZ serving and means there should be 0 cost for data downloads from S3. Since this mapping is now done server side as part of clone bundles, we remove this feature. The previous version of this file downloaded a tar file of the .hg directory for various repos and uncompressed it. The new version just does an `hg clone` preferring "streaming clones." "Streaming clones" are effectively `tar | nc` and are extremely fast. IMO the tar file provides little value so it has been removed from the equation. A downside of not using a tar file is that seeding now talks to hg.mozilla.org instead of only S3. This could potentially drive a lot of load to hg.mozilla.org if multiple machines perform this seeding at the same time. However, 99+% of clone load will be offloaded to S3 via the clone bundles and hg.mozilla.org will only need to serve commits since the bundle was created. This should not be more than a few hundred commits and should not require much effort on behalf of the server. But if this does overwhelm the server, we can restore tar files (or provide a clone mode that only fetches the bundle). This commit assumes that all machines have Mercurial 3.7 as `hg` in PATH. If an older version of Mercurial is present, the clone will take several minutes longer than it should or it will fail due to the client not having bundle2 support (the firefox repo requires bundle2). A downside of this commit is that jobs not having the new shared/pooled storage code deployed will need to perform a full clone on first job because the old paths (e.g. /builds/hg-shared/mozilla-central) are no longer present. This only impacts legacy commits/jobs and the number of jobs should diminish over time. Furthermore, once hgtool is updated to use shared/pooled storage, this won't be an issue (that is tracked in bug 1270951). MozReview-Commit-ID: 584WIULj5nQ
modules/runner/templates/tasks/populate_shared_repos.erb
--- a/modules/runner/templates/tasks/populate_shared_repos.erb
+++ b/modules/runner/templates/tasks/populate_shared_repos.erb
@@ -1,120 +1,56 @@
 #!/usr/bin/env python
 # vim: ft=python
 
-try:
-    import requests
-    import boto
-except ImportError:
-    print "No boto or requests installed, skipping"
-    exit(0)
-
 import argparse
 import logging
 import grp
 import os
 import pwd
-import re
 import socket
-import tarfile
+import subprocess
+
 
 SUPPORTED_HOSTNAME_PREFIXES = ["bld-linux64", "try-linux64"]
 
-DIRS = {
-    "/builds/hg-shared/integration/mozilla-inbound/.hg": {
-        "regex": "mozilla-inbound",
-        "deploy_to_prod_slaves": True,
-        "deploy_to_try_slaves": False,
-        "buckets": {
-            "us-east-1": "mozilla-releng-tarballs-use1",
-            "us-west-1": "mozilla-releng-tarballs-usw1",
-            "us-west-2": "mozilla-releng-tarballs-usw2",
-        },
-        "key": "mozilla-inbound.tar"
-    },
-    "/builds/hg-shared/mozilla-central/.hg": {
-        "regex": "mozilla-central",
-        "deploy_to_prod_slaves": True,
-        "deploy_to_try_slaves": False,
-        "buckets": {
-            "us-east-1": "mozilla-releng-tarballs-use1",
-            "us-west-1": "mozilla-releng-tarballs-usw1",
-            "us-west-2": "mozilla-releng-tarballs-usw2",
-        },
-        "key": "mozilla-central.tar"
-    },
-    "/builds/hg-shared/try/.hg": {
-        "deploy_to_prod_slaves": False,
-        "deploy_to_try_slaves": True,
-        "buckets": {
-            "us-east-1": "mozilla-releng-tarballs-use1",
-            "us-west-1": "mozilla-releng-tarballs-usw1",
-            "us-west-2": "mozilla-releng-tarballs-usw2",
-        },
-        "key": "try.tar"
-    },
-}
+SHARE_BASE_DIR = '/builds/hg-shared'
+FIREFOX_REPO = 'https://hg.mozilla.org/mozilla-unified'
+FIREFOX_SHA1 = '8ba995b74e18334ab3707f27e9eb8f4e37ba3d29'
+
 
 log = logging.getLogger(__name__)
 
 
 def is_suported_slave(hostname):
     return any(hostname.startswith(prefix) for prefix in
                SUPPORTED_HOSTNAME_PREFIXES)
 
 
-def is_try_slave(hostname):
-    return hostname.startswith("try-")
-
-
-def get_availability_zone():
-    url = "http://169.254.169.254/latest/meta-data/placement/availability-zone"
-    log.info("Fetching region data")
-    az = requests.get(url, timeout=3).content
-    return az
-
-
-def get_prepopulated_dirs(is_try=False):
-    dirs = {}
-    for target_dir, info in DIRS.iteritems():
-        if (info["deploy_to_prod_slaves"] and not is_try) or \
-                (info["deploy_to_try_slaves"] and is_try):
-            log.info("To deploy: %s", target_dir)
-            dirs[target_dir] = info
+def clone_firefox():
+    """Clone the Firefox repo to the hg-shared directory."""
+    dest_dir = os.path.join(SHARE_BASE_DIR, FIREFOX_SHA1)
+    if os.path.exists(dest_dir):
+        log.info('%s already exists; skipping' % dest_dir)
+        return
 
-    az = get_availability_zone()
-    for target_dir, info in dirs.iteritems():
-        bucket_map = info["buckets"]
-        for region, bucket in bucket_map.iteritems():
-            if az.startswith(region):
-                # change in-place
-                info["bucket"] = bucket
-    return dirs
-
-
-def deploy(dirs):
-    for target_dir, info in dirs.iteritems():
-        if os.path.exists(target_dir):
-            log.info("%s already exists, skipping.", target_dir)
-            continue
-        unpack_tarball(info["bucket"], info["key"], target_dir)
-
-
-def unpack_tarball(bucket, key, target_dir):
-    log.info("Creatging %s", target_dir)
-    os.makedirs(target_dir, 0775)
-    log.info("Fetching s3://%s/%s to %s...", bucket, key, target_dir)
-    conn = boto.connect_s3()
-    bucket = conn.get_bucket(bucket)
-    key = bucket.get_key(key)
-    tar = tarfile.open(mode="r|*", fileobj=key)
-    log.info("Extracting...")
-    tar.extractall(path=target_dir)
-    log.info("Extracted")
+    # hg.mozilla.org automatically serves a clone bundle URL appropriate
+    # for the client's IP. So just `hg clone` to get the data.
+    log.info('creating %s' % dest_dir)
+    args = [
+        'hg',
+        # Prefer a streaming clone bundle because they are the fastest
+        # to download and preserve optimal encoding from server.
+        '--config', 'ui.clonebundleprefers=VERSION=packed1',
+        'clone',
+        '--noupdate',
+        FIREFOX_REPO,
+        dest_dir,
+    ]
+    return subprocess.call(args)
 
 
 def run_as(user, group):
     current_id = os.getuid()
     current_gid = os.getgid()
     target_uid = pwd.getpwnam(user).pw_uid
     target_gid = grp.getgrnam(group).gr_gid
     # Set GID before dropping to UID
@@ -144,23 +80,19 @@ def main():
 
     hostname = socket.gethostname().split(".")[0]
     log.info("Working on %s", hostname)
 
     if not is_suported_slave(hostname):
         log.warn("%s is not supported", hostname)
         exit(0)
 
-    if is_try_slave(hostname):
-        log.info("Try slave detected")
-        dirs = get_prepopulated_dirs(is_try=True)
-        deploy(dirs)
-    else:
-        log.info("Prod slave detected")
-        dirs = get_prepopulated_dirs(builders=None, is_try=False)
-        deploy(dirs)
+    # The Firefox repo is the only one large enough to warrant
+    # seeding.
+    clone_firefox()
+
 
 if __name__ == "__main__":
     try:
         main()
     except Exception:
-        log.exception("Failed to fetch tarballs, gracefully exiting...")
+        log.exception("Failed to seed repo, gracefully exiting...")
         exit(0)