Bug 1245969 - TaskCluster tasks for performing system bootstrap; r?dustin draft
authorGregory Szorc <gps@mozilla.com>
Fri, 19 Feb 2016 12:55:13 -0800
changeset 441535 ecda3d4fd7b29ed1b35a284d8929bfde8195e54f
parent 441534 1dcd43dd2a7b04e2bb714349033a456ea5158f3e
child 537571 dc13d8839f9e647717e7a007dd367f3e29388196
push id36442
push userbmo:gps@mozilla.com
push dateSat, 19 Nov 2016 01:00:57 +0000
reviewersdustin
bugs1245969
milestone53.0a1
Bug 1245969 - TaskCluster tasks for performing system bootstrap; r?dustin (THIS PATCH IS FEEDBACK QUALITY AND ISN'T READY FOR FULL REVIEW) We've had `mach bootstrap` and the one-line wget + `python bootstrap` bootstrap tool for years. Unfortunately, it's never been formally tested. TaskCluster enables us to finally do that. This commit introduces support for testing the system bootstrap tool on various Linux distros. The bootstrap tasks are implemented as their own kind because they are somewhat special. Each OS is defined once along with its base Docker images, a command to pre-bootstrap the Docker environment (typically to install Python and wget), and the name of the Python binary to use. A transform "splits" each job 4 separate jobs, each one testing a different bootstrap application: firefox and fennec with normal and artifact variations. The task itself is a giant pile of shell commands that essentially: 1) Perform (hopefully minimal) pre-bootstrap setup 2) Download bootstrap.py from the revision running in CI 3) Invoke the bootstrapper 4) Manually upgrade Mercurial because many distros have old versions that don't use clone bundles and we want clones to hit S3 so we don't DoS hg.mozilla.org 5) Clone the Mercurial repo 6) Run `mach build` TODO * Arch is failing across the board * CentOS6 is failing across the board * Gentoo is having a lot of problems * Fennec bootstrap is failing due to interactive prompt accepting Android License foo * Figure out how to schedule every ~24 hours * Configure to run when python/mozboot changes MozReview-Commit-ID: 1ncrnBT1Sti
taskcluster/ci/bootstrap/kind.yml
taskcluster/taskgraph/transforms/job/bootstrap.py
taskcluster/taskgraph/transforms/task.py
taskcluster/taskgraph/try_option_syntax.py
new file mode 100644
--- /dev/null
+++ b/taskcluster/ci/bootstrap/kind.yml
@@ -0,0 +1,117 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+implementation: taskgraph.task.transform:TransformTask
+
+transforms:
+   - taskgraph.transforms.job.bootstrap:transforms
+   - taskgraph.transforms.job:transforms
+   - taskgraph.transforms.task:transforms
+
+job-defaults:
+    attributes:
+        build_platform: bootstrap
+        build_type: opt
+    treeherder:
+        kind: build
+        tier: 2
+        platform: bootstrap/opt
+    worker-type: aws-provisioner-v1/gecko-{level}-b-linux
+    worker:
+        implementation: docker-worker
+        max-run-time: 1800
+    run:
+        using: bootstrap
+
+jobs:
+    bootstrap/arch:
+        description: "Arch Linux Bootstrap Support"
+        treeherder:
+            symbol: "Arch"
+        worker:
+            docker-image: "base/archlinux:latest"
+        run:
+            system-setup: >
+                pacman -S --refresh
+                && pacman -S --noconfirm archlinux-keyring
+                && pacman -S -u --noconfirm
+                && pacman-db-upgrade
+                && pacman -S --noconfirm ca-certificates-mozilla
+                && pacman -S --noconfirm python2 wget
+            python: python2.7
+
+    bootstrap/centos6:
+        description: "CentOS 6 Bootstrap Support"
+        treeherder:
+            symbol: "CentOS6"
+        worker:
+            docker-image: "centos:centos6"
+        run:
+            system-setup: "yum update -y && yum install -y python-devel wget which"
+            python: python
+
+    bootstrap/centos7:
+        description: "CentOS 7 Bootstrap Support"
+        treeherder:
+            symbol: "CentOS7"
+        worker:
+            docker-image: "centos:centos7"
+        run:
+            system-setup: "yum update -y && yum install -y python-devel wget which"
+            python: python
+
+    bootstrap/debian8:
+        description: "Debian 8 Bootstrap Support"
+        treeherder:
+            symbol: "Debian8"
+        worker:
+            docker-image: "debian:jessie"
+        run:
+            system-setup: "apt-get update && apt-get -y install python2.7 wget"
+            python: python2.7
+
+    bootstrap/fedora22:
+        description: "Fedora 22 Bootstrap Support"
+        treeherder:
+            symbol: "Fedora22"
+        worker:
+            docker-image: "fedora:22"
+        run:
+            system-setup: "dnf update -y && dnf install -y unzip wget zip"
+            python: python
+
+    bootstrap/fedora23:
+        description: "Fedora 23 Bootstrap Support"
+        treeherder:
+            symbol: "Fedora23"
+        worker:
+            docker-image: "fedora:23"
+        run:
+            system-setup: "dnf update -y && dnf install -y python2 redhat-rpm-config wget"
+            python: python2.7
+
+    bootstrap/gentoo:
+        description: "Gentoo Bootstrap Support"
+        treeherder:
+            symbol: "Gentoo"
+        worker:
+            docker-image: "gentoo/stage3-amd64:latest"
+        run:
+            system-setup: >
+                emerge --sync --quiet
+                && echo ">=media-libs/libpng-1.6.19 apng" >> /etc/portage/package.use/firefox
+                && echo ">=dev-lang/python-2.7.10-r1:2.7 sqlite" >> /etc/portage/package.use/firefox
+                && echo ">=x11-libs/cairo-1.14.2 X" >> /etc/portage/package.use/firefox
+                && emerge -UD world
+            python: python2.7
+
+    bootstrap/ubuntu1404:
+        description: "Ubuntu 16.04 Bootstrap Support"
+        treeherder:
+            symbol: "Ubuntu16.04"
+        worker:
+            docker-image: "ubuntu:16.04"
+        run:
+            system-setup: "apt-get update && apt-get -y install python2.7 wget"
+            python: python2.7
new file mode 100644
--- /dev/null
+++ b/taskcluster/taskgraph/transforms/job/bootstrap.py
@@ -0,0 +1,107 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+"""Support for bootstrap tasks."""
+
+from __future__ import absolute_import, unicode_literals
+
+import copy
+
+from voluptuous import (
+    Required,
+    Schema,
+)
+
+from taskgraph.transforms.base import (
+    TransformSequence,
+)
+from taskgraph.transforms.job import (
+    run_job_using,
+)
+
+
+transforms = TransformSequence()
+
+
+bootstrap_schema = Schema({
+    Required('using'): 'bootstrap',
+
+    # Command to configure system install. This is basically to get system
+    # packages updated and to get pre-bootstrap requirements on the system.
+    # As part of this, wget should be installed.
+    Required('system-setup'): basestring,
+    Required('python'): basestring,
+    Required('application'): basestring,
+})
+
+
+@transforms.add
+def clone_applications(config, jobs):
+    """Clones the job to bootstrap multiple applications."""
+    APPS = {
+        b'browser': b'fx',
+        b'browser_artifact_mode': b'fx-a',
+        b'mobile_android': b'an',
+        b'mobile_android_artifact_mode': b'an-a',
+    }
+
+    for job in jobs:
+        for app, symbol in APPS.items():
+            j = copy.deepcopy(job)
+            j['treeherder']['symbol'] = b'bs-%s(%s)' % (
+                symbol, j['treeherder']['symbol'])
+            j['name'] = '%s-%s' % (j['name'], symbol)
+            j['run']['application'] = app
+            yield j
+
+@run_job_using('docker-worker', 'bootstrap', schema=bootstrap_schema)
+def docker_worker_bootstrap(config, job, taskdesc):
+    run = job['run']
+    worker = taskdesc['worker']
+    worker['env'].update({
+        'SHELL': '/bin/bash',
+    })
+
+    # We use a cache for the checkout because Mercurial overwhelms AUFS.
+    worker.setdefault('caches', []).append({
+        'type': 'persistent',
+        'name': 'level-%s-bootstrap-checkouts' % config.params['level'],
+        'mount-point': '/checkouts',
+    })
+
+    bsurl = '%s/raw-file/%s/python/mozboot/bin/bootstrap.py' % (
+        config.params['head_repository'].rstrip('/'),
+        config.params['head_rev'])
+
+    app = run['application']
+
+    bs = '%s bootstrap.py --no-interactive --application-choice=%s --repo-url=%s --repo-rev=%s' % (
+        run['python'], app, config.params['head_repository'],
+        config.params['head_rev'])
+
+    script_args = [
+        run['system-setup'].strip(),
+        'wget %s' % bsurl,
+        bs,
+        # Many distros run an ancient Mercurial that doesn't do cloen bundles.
+        # So install a modern Mercurial manually to ensure the clone below
+        # runs optimally.
+        'wget https://bootstrap.pypa.io/get-pip.py',
+        '%s get-pip.py' % run['python'],
+        'pip install --upgrade Mercurial==4.0',
+        'rm -rf /checkouts/*',
+        'hg clone -U %s /checkouts/repo' % config.params['base_repository'],
+        'cd /checkouts/repo',
+        'hg pull -r %s %s' % (config.params['head_rev'], config.params['head_repository']),
+        'hg up -r %s' % config.params['head_rev'],
+    ]
+
+    if 'artifact' in app:
+        # This is needed to prevent `mach` from hanging on first invocation.
+        script_args.append('mkdir -p ~/.mozbuild')
+        script_args.append('./mach configure --enable-artifact-builds')
+
+    script_args.append('./mach build')
+
+    worker['command'] = ['bash', '-cx', ' && '.join(script_args)]
--- a/taskcluster/taskgraph/transforms/task.py
+++ b/taskcluster/taskgraph/transforms/task.py
@@ -302,16 +302,20 @@ GROUP_NAMES = {
     'tc-W': 'Web platform tests executed by TaskCluster',
     'tc-W-e10s': 'Web platform tests executed by TaskCluster with e10s',
     'tc-X': 'Xpcshell tests executed by TaskCluster',
     'tc-X-e10s': 'Xpcshell tests executed by TaskCluster with e10s',
     'Aries': 'Aries Device Image',
     'Nexus 5-L': 'Nexus 5-L Device Image',
     'Cc': 'Toolchain builds',
     'SM-tc': 'Spidermonkey builds',
+    'bs-fx': 'Firefox full build bootstrap',
+    'bs-fx-a': 'Firefox artifact build bootstrap',
+    'bs-an': 'Fennec full build bootstrap',
+    'bs-an-a': 'Fennec artifact build bootstrap',
 }
 UNKNOWN_GROUP_NAME = "Treeherder group {} has no name; add it to " + __file__
 
 BUILDBOT_ROUTE_TEMPLATES = [
     "index.buildbot.branches.{project}.{job-name-buildbot}",
     "index.buildbot.revisions.{head_rev}.{project}.{job-name-buildbot}",
 ]
 
--- a/taskcluster/taskgraph/try_option_syntax.py
+++ b/taskcluster/taskgraph/try_option_syntax.py
@@ -30,16 +30,17 @@ BUILD_KINDS = set([
     'upload-symbols',
     'valgrind',
     'static-analysis',
     'spidermonkey',
 ])
 
 # anything in this list is governed by -j
 JOB_KINDS = set([
+    'bootstrap',
     'source-check',
     'toolchain',
     'marionette-harness',
     'android-stuff',
 ])
 
 
 # mapping from shortcut name (usable with -u) to a boolean function identifying