Bug 1245969 - TaskCluster tasks for performing system bootstrap; r?dustin
(THIS PATCH IS FEEDBACK QUALITY AND ISN'T READY FOR FULL REVIEW)
We've had `mach bootstrap` and the one-line wget + `python bootstrap`
bootstrap tool for years. Unfortunately, it's never been formally
tested. TaskCluster enables us to finally do that.
This commit introduces support for testing the system bootstrap tool
on various Linux distros.
The bootstrap tasks are implemented as their own kind because they
are somewhat special.
Each OS is defined once along with its base Docker images, a command
to pre-bootstrap the Docker environment (typically to install Python
and wget), and the name of the Python binary to use.
A transform "splits" each job 4 separate jobs, each one testing a
different bootstrap application: firefox and fennec with normal and
artifact variations.
The task itself is a giant pile of shell commands that essentially:
1) Perform (hopefully minimal) pre-bootstrap setup
2) Download bootstrap.py from the revision running in CI
3) Invoke the bootstrapper
4) Manually upgrade Mercurial because many distros have old versions
that don't use clone bundles and we want clones to hit S3 so we
don't DoS hg.mozilla.org
5) Clone the Mercurial repo
6) Run `mach build`
TODO
* Arch is failing across the board
* CentOS6 is failing across the board
* Gentoo is having a lot of problems
* Fennec bootstrap is failing due to interactive prompt accepting Android License foo
* Figure out how to schedule every ~24 hours
* Configure to run when python/mozboot changes
MozReview-Commit-ID: 1ncrnBT1Sti
new file mode 100644
--- /dev/null
+++ b/taskcluster/ci/bootstrap/kind.yml
@@ -0,0 +1,117 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+implementation: taskgraph.task.transform:TransformTask
+
+transforms:
+ - taskgraph.transforms.job.bootstrap:transforms
+ - taskgraph.transforms.job:transforms
+ - taskgraph.transforms.task:transforms
+
+job-defaults:
+ attributes:
+ build_platform: bootstrap
+ build_type: opt
+ treeherder:
+ kind: build
+ tier: 2
+ platform: bootstrap/opt
+ worker-type: aws-provisioner-v1/gecko-{level}-b-linux
+ worker:
+ implementation: docker-worker
+ max-run-time: 1800
+ run:
+ using: bootstrap
+
+jobs:
+ bootstrap/arch:
+ description: "Arch Linux Bootstrap Support"
+ treeherder:
+ symbol: "Arch"
+ worker:
+ docker-image: "base/archlinux:latest"
+ run:
+ system-setup: >
+ pacman -S --refresh
+ && pacman -S --noconfirm archlinux-keyring
+ && pacman -S -u --noconfirm
+ && pacman-db-upgrade
+ && pacman -S --noconfirm ca-certificates-mozilla
+ && pacman -S --noconfirm python2 wget
+ python: python2.7
+
+ bootstrap/centos6:
+ description: "CentOS 6 Bootstrap Support"
+ treeherder:
+ symbol: "CentOS6"
+ worker:
+ docker-image: "centos:centos6"
+ run:
+ system-setup: "yum update -y && yum install -y python-devel wget which"
+ python: python
+
+ bootstrap/centos7:
+ description: "CentOS 7 Bootstrap Support"
+ treeherder:
+ symbol: "CentOS7"
+ worker:
+ docker-image: "centos:centos7"
+ run:
+ system-setup: "yum update -y && yum install -y python-devel wget which"
+ python: python
+
+ bootstrap/debian8:
+ description: "Debian 8 Bootstrap Support"
+ treeherder:
+ symbol: "Debian8"
+ worker:
+ docker-image: "debian:jessie"
+ run:
+ system-setup: "apt-get update && apt-get -y install python2.7 wget"
+ python: python2.7
+
+ bootstrap/fedora22:
+ description: "Fedora 22 Bootstrap Support"
+ treeherder:
+ symbol: "Fedora22"
+ worker:
+ docker-image: "fedora:22"
+ run:
+ system-setup: "dnf update -y && dnf install -y unzip wget zip"
+ python: python
+
+ bootstrap/fedora23:
+ description: "Fedora 23 Bootstrap Support"
+ treeherder:
+ symbol: "Fedora23"
+ worker:
+ docker-image: "fedora:23"
+ run:
+ system-setup: "dnf update -y && dnf install -y python2 redhat-rpm-config wget"
+ python: python2.7
+
+ bootstrap/gentoo:
+ description: "Gentoo Bootstrap Support"
+ treeherder:
+ symbol: "Gentoo"
+ worker:
+ docker-image: "gentoo/stage3-amd64:latest"
+ run:
+ system-setup: >
+ emerge --sync --quiet
+ && echo ">=media-libs/libpng-1.6.19 apng" >> /etc/portage/package.use/firefox
+ && echo ">=dev-lang/python-2.7.10-r1:2.7 sqlite" >> /etc/portage/package.use/firefox
+ && echo ">=x11-libs/cairo-1.14.2 X" >> /etc/portage/package.use/firefox
+ && emerge -UD world
+ python: python2.7
+
+ bootstrap/ubuntu1404:
+ description: "Ubuntu 16.04 Bootstrap Support"
+ treeherder:
+ symbol: "Ubuntu16.04"
+ worker:
+ docker-image: "ubuntu:16.04"
+ run:
+ system-setup: "apt-get update && apt-get -y install python2.7 wget"
+ python: python2.7
new file mode 100644
--- /dev/null
+++ b/taskcluster/taskgraph/transforms/job/bootstrap.py
@@ -0,0 +1,107 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+"""Support for bootstrap tasks."""
+
+from __future__ import absolute_import, unicode_literals
+
+import copy
+
+from voluptuous import (
+ Required,
+ Schema,
+)
+
+from taskgraph.transforms.base import (
+ TransformSequence,
+)
+from taskgraph.transforms.job import (
+ run_job_using,
+)
+
+
+transforms = TransformSequence()
+
+
+bootstrap_schema = Schema({
+ Required('using'): 'bootstrap',
+
+ # Command to configure system install. This is basically to get system
+ # packages updated and to get pre-bootstrap requirements on the system.
+ # As part of this, wget should be installed.
+ Required('system-setup'): basestring,
+ Required('python'): basestring,
+ Required('application'): basestring,
+})
+
+
+@transforms.add
+def clone_applications(config, jobs):
+ """Clones the job to bootstrap multiple applications."""
+ APPS = {
+ b'browser': b'fx',
+ b'browser_artifact_mode': b'fx-a',
+ b'mobile_android': b'an',
+ b'mobile_android_artifact_mode': b'an-a',
+ }
+
+ for job in jobs:
+ for app, symbol in APPS.items():
+ j = copy.deepcopy(job)
+ j['treeherder']['symbol'] = b'bs-%s(%s)' % (
+ symbol, j['treeherder']['symbol'])
+ j['name'] = '%s-%s' % (j['name'], symbol)
+ j['run']['application'] = app
+ yield j
+
+@run_job_using('docker-worker', 'bootstrap', schema=bootstrap_schema)
+def docker_worker_bootstrap(config, job, taskdesc):
+ run = job['run']
+ worker = taskdesc['worker']
+ worker['env'].update({
+ 'SHELL': '/bin/bash',
+ })
+
+ # We use a cache for the checkout because Mercurial overwhelms AUFS.
+ worker.setdefault('caches', []).append({
+ 'type': 'persistent',
+ 'name': 'level-%s-bootstrap-checkouts' % config.params['level'],
+ 'mount-point': '/checkouts',
+ })
+
+ bsurl = '%s/raw-file/%s/python/mozboot/bin/bootstrap.py' % (
+ config.params['head_repository'].rstrip('/'),
+ config.params['head_rev'])
+
+ app = run['application']
+
+ bs = '%s bootstrap.py --no-interactive --application-choice=%s --repo-url=%s --repo-rev=%s' % (
+ run['python'], app, config.params['head_repository'],
+ config.params['head_rev'])
+
+ script_args = [
+ run['system-setup'].strip(),
+ 'wget %s' % bsurl,
+ bs,
+ # Many distros run an ancient Mercurial that doesn't do cloen bundles.
+ # So install a modern Mercurial manually to ensure the clone below
+ # runs optimally.
+ 'wget https://bootstrap.pypa.io/get-pip.py',
+ '%s get-pip.py' % run['python'],
+ 'pip install --upgrade Mercurial==4.0',
+ 'rm -rf /checkouts/*',
+ 'hg clone -U %s /checkouts/repo' % config.params['base_repository'],
+ 'cd /checkouts/repo',
+ 'hg pull -r %s %s' % (config.params['head_rev'], config.params['head_repository']),
+ 'hg up -r %s' % config.params['head_rev'],
+ ]
+
+ if 'artifact' in app:
+ # This is needed to prevent `mach` from hanging on first invocation.
+ script_args.append('mkdir -p ~/.mozbuild')
+ script_args.append('./mach configure --enable-artifact-builds')
+
+ script_args.append('./mach build')
+
+ worker['command'] = ['bash', '-cx', ' && '.join(script_args)]
--- a/taskcluster/taskgraph/transforms/task.py
+++ b/taskcluster/taskgraph/transforms/task.py
@@ -302,16 +302,20 @@ GROUP_NAMES = {
'tc-W': 'Web platform tests executed by TaskCluster',
'tc-W-e10s': 'Web platform tests executed by TaskCluster with e10s',
'tc-X': 'Xpcshell tests executed by TaskCluster',
'tc-X-e10s': 'Xpcshell tests executed by TaskCluster with e10s',
'Aries': 'Aries Device Image',
'Nexus 5-L': 'Nexus 5-L Device Image',
'Cc': 'Toolchain builds',
'SM-tc': 'Spidermonkey builds',
+ 'bs-fx': 'Firefox full build bootstrap',
+ 'bs-fx-a': 'Firefox artifact build bootstrap',
+ 'bs-an': 'Fennec full build bootstrap',
+ 'bs-an-a': 'Fennec artifact build bootstrap',
}
UNKNOWN_GROUP_NAME = "Treeherder group {} has no name; add it to " + __file__
BUILDBOT_ROUTE_TEMPLATES = [
"index.buildbot.branches.{project}.{job-name-buildbot}",
"index.buildbot.revisions.{head_rev}.{project}.{job-name-buildbot}",
]
--- a/taskcluster/taskgraph/try_option_syntax.py
+++ b/taskcluster/taskgraph/try_option_syntax.py
@@ -30,16 +30,17 @@ BUILD_KINDS = set([
'upload-symbols',
'valgrind',
'static-analysis',
'spidermonkey',
])
# anything in this list is governed by -j
JOB_KINDS = set([
+ 'bootstrap',
'source-check',
'toolchain',
'marionette-harness',
'android-stuff',
])
# mapping from shortcut name (usable with -u) to a boolean function identifying