Bug 1396154 - Support building Docker images without Dockerfile; r?dustin
Today, we use Dockerfile's for building Docker images. Here's a
*partial* list of problems with our current approach:
* We don't enforce that the base image is deterministic (e.g. we
could pull a different image build for e.g. "ubuntu:16.04" every
time we build)
* System package installs aren't deterministic. This is because we
point at a package repository that is currently changing.
* Common tasks need to be performed in every Docker image (e.g. install
Mercurial, run-task, create worker user, set environment variables,
etc)
* Sharing of common system setup snippets is not turn-key. You can
create standalone scripts (like install-mercurial.sh). But you need
to explicitly call these from some script specific to the image.
* The ``# %include`` syntax in Dockerfile is non-standard and a bit
wonky.
* File downloads aren't consistently handled. Some images download
tooltool from GitHub. Some images hit PyPI and other package
repositories. We want a way to force downloads through trusted
and supported channels. Allowing any RUN from Docker makes this
hard.
This leads to Dockerfile's that are large, inconsistent, contain a
lot of copy pasta, and hard to maintain.
This commit attempts to change that by doing away with Dockerfile's
as our primary image building mechanism.
We introdce a YAML-based mechanism for declaring images. The YAML
allows the image to declare common primitives, such as tooltool
artifacts to download and lists of *recipe* scripts to execute.
This mechanism also automagically adds common primitives to the
Docker image like run-task, robustcheckout, and Mercurial package
installation.
The image.yml is converted to an auto-generated Dockerfile along
with a set of well-defined files ADDed to /image-build.
The only RUN directive invokes a minimal shell script that downloads
a standalone Python 3.6.2 distribution from tooltool and then
executes a Python script that performs the bulk of the work. The
Python script handles tooltool downloads, recipe execution, etc.
To prove the new image building mechanism works, the "lint" image
is converted to use it. The image is now ~627MB instead of ~691MB.
(probably from nuking apt caches). More importantly, it is only
6 layers instead of 28. This is because 99% of the work is done
as an ADD+RUN instead of N of each. (The remaining 4 layers are a
bit harder to eliminate.)
There are still a few features I'd like to implement. These include:
* Pinning the system packaging tools to a deterministic endpoint
(this requires running a custom package server and is somewhat
tracked in
bug 1289812).
* Disabling or firewalling the network during recipe execution
to limit what hosts or services we can use (so we don't introduce
security issues or uptime concerns).
MozReview-Commit-ID: HdfBTh4KqRl
new file mode 100755
--- /dev/null
+++ b/taskcluster/docker/bootstrap/build-image-bootstrap
@@ -0,0 +1,70 @@
+#!/bin/sh
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+# Do not remove -e because we want failures to be fatal.
+set -ex
+
+cd /image-build
+
+# Create our worker user as early as possible to ensure it has
+# a consistent uid/gid.
+mkdir -p /builds
+groupadd -g 1000 worker
+
+if [ -e /builds/worker ]; then
+ echo '/builds/worker already exists; did a VOLUME sneak in?'
+ exit 1
+fi
+
+useradd -d /builds/worker -s /bin/bash -m -u 1000 -g worker worker
+
+# Download a Python to run our building script. We purposefully don't use
+# the Python from the system packager because the image may have
+# other plans for Python and we don't want to contaminate the system.
+
+# FUTURE: consider adding this Python to the Docker image context.
+# This is certainly possible. But it would add overhead to building
+# the context archive. And, the files may linger in the exported image
+# if all layers are exported. By downloading it here and removing it
+# as part of a single Docker build step, the files should never be
+# exported in the final Docker image.
+if [ -f /etc/lsb-release ]; then
+ . /etc/lsb-release
+
+ apt-get update && apt-get install -y wget
+
+ if [ "${DISTRIB_ID}" = "Ubuntu" -a "${DISTRIB_RELEASE}" = "16.04" ]; then
+ PYTHON_HASH=9078b397f3edbea1cdb09c4ff4474f50e1245e3bd13f1e295a00bd3caad001897bbd9e75c06104c0e53007adad1f30a7e2f3c348cfd98a9a3817ab2033c28855
+ else
+ echo "do not know how to install Python on this Debian distro version"
+ exit 1
+ fi
+else
+ echo "do not know how to install Python on this Linux distro"
+ exit 1
+fi
+
+TOOLTOOL_SERVER=${TOOLTOOL_SERVER:=https://tooltool.mozilla-releng.net/}
+URL=${TOOLTOOL_SERVER}sha512/${PYTHON_HASH}
+
+cat >python.hash << EOF
+${PYTHON_HASH} python.tar.gz
+EOF
+
+wget -O python.tar.gz --progress=dot:giga ${URL}
+sha512sum -c python.hash
+tar -xzf python.tar.gz
+
+if [ -n "${DISTRIB_ID}" ]; then
+ apt-get remove -y wget
+ apt-get autoremove -y
+fi
+
+# The archive contains its own copies of some libraries. Add it to
+# loader search path.
+# TODO produce standalone packages properly so this isn't needed.
+export LD_LIBRARY_PATH=/image-build/python-bootstrap/lib
+
+exec python-bootstrap/bin/python3.6 -u build-image-main
new file mode 100644
--- /dev/null
+++ b/taskcluster/docker/bootstrap/build-image-main
@@ -0,0 +1,261 @@
+#!/usr/bin/env python3.6 -u
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+import grp
+import hashlib
+import io
+import json
+import os
+import pathlib
+import pwd
+import shutil
+import stat
+import ssl
+import subprocess
+import sys
+import tarfile
+import urllib.request
+import zipfile
+
+
+Path = pathlib.Path
+
+
+ARCHIVE_SUFFIXES = {
+ '.tar.gz': 'tar-gz',
+ '.tar.bz2': 'tar-bz2',
+ '.tar.xz': 'tar-xz',
+ '.zip': 'zip',
+}
+
+
+def have_apt():
+ return any(os.path.exists(p) for p in (
+ '/usr/bin/apt',
+ '/usr/bin/apt-get',
+ ))
+
+
+def tooltool_download(dest_dir, e):
+ """Download a file from its tooltool manifest entry."""
+ base_url = os.environ.get('TOOLTOOL_SERVER',
+ 'https://tooltool.mozilla-releng.net/')
+ url = '%s/sha512/%s' % (base_url.rstrip('/'), e['digest'])
+
+ print('downloading %s from %s' % (e['filename'], url))
+
+ # Because we do hash validation, disable security checking because it
+ # can only fail (due to e.g. missing certs) and doesn't add any extra
+ # protection for read-only operations.
+ ctx = ssl.SSLContext()
+ ctx.verify_mode = ssl.CERT_NONE
+
+ res = urllib.request.urlopen(url, context=ctx)
+ if res.status != 200:
+ raise Exception('HTTP %s from %s' % (res.status, url))
+
+ buf = io.BytesIO()
+ hasher = hashlib.sha512()
+ while True:
+ chunk = res.read(1000000)
+ if not chunk:
+ break
+
+ buf.write(chunk)
+ hasher.update(chunk)
+ print('read %d bytes' % buf.tell())
+
+ if hasher.hexdigest() != e['digest']:
+ raise Exception('hash verification failed: got %s; expected %s' % (
+ hasher.hexdigest(), e['digest']))
+
+ print('validated digest of %s' % hasher.hexdigest())
+
+ dest_path = Path(dest_dir, e['filename'])
+ with dest_path.open('wb') as fh:
+ buf.seek(0)
+ while True:
+ data = buf.read(32768)
+ if not data:
+ break
+
+ fh.write(data)
+
+ print('wrote %s' % dest_path)
+
+ if e.get('unpack', False):
+ found = False
+
+ for suffix, flavor in ARCHIVE_SUFFIXES.items():
+ if e['filename'].endswith(suffix):
+ found = True
+ break
+
+ if not found:
+ raise Exception('do not know how to unpack file %s' % e['filename'])
+
+ dest = Path(dest_dir, e['filename'][:-len(suffix)])
+ print('extracting %s to %s' % (e['filename'], dest))
+
+ buf.seek(0)
+
+ if flavor == 'tar-gz':
+ a = tarfile.open(mode='r:gz', fileobj=buf)
+ elif flavor == 'tar-bz2':
+ a = tarfile.open(mode='r:bz2', fileobj=buf)
+ elif flavor == 'tar-xz':
+ a = tarfile.open(mode='r:xz', fileobj=buf)
+ elif flavor == 'zip':
+ a = zipfile.ZipFile(buf, mode='r')
+ else:
+ raise Exception('unhandled archive flavor %s' % flavor)
+
+ with a:
+ a.extractall(dest)
+
+
+def remove_packages(packages):
+ if not packages:
+ return
+
+ if have_apt():
+ print('uninstalling %s' % ' '.join(sorted(packages)))
+ subprocess.check_call(['apt-get', 'remove', '-y'] + sorted(packages))
+ subprocess.check_call(['apt-get', 'autoremove', '-y'])
+ else:
+ raise Exception('do not know how to remove packages')
+
+
+def get_recipes(path):
+ if not os.path.exists(path):
+ return []
+
+ recipes = []
+
+ for recipe in sorted(os.listdir(path)):
+ p = os.path.join(path, recipe)
+
+ st = os.stat(p)
+ if not st.st_mode & stat.S_IXUSR:
+ print('ERROR: recipe %s is not executable' % recipe)
+ sys.exit(1)
+
+ recipes.append(recipe)
+
+ return recipes
+
+
+def main(here):
+ if os.getuid() != 0:
+ print('ERROR: must run as root')
+ return 1
+
+ recipe_dir = os.path.join(here, 'recipes')
+ recipes = get_recipes(recipe_dir)
+
+ with open(os.path.join(here, 'tooltool-manifest.tt'), 'rb') as fh:
+ downloads = json.load(fh)
+
+ # Input validation is complete. Now do some work.
+
+ # Copy run-task.
+ os.makedirs('/builds/worker/bin', exist_ok=True)
+ shutil.copyfile(os.path.join(here, 'run-task'),
+ '/builds/worker/bin/run-task')
+
+ # Download requested files using the tooltool server. We don't use tooltool
+ # itself because it isn't necessary and brings in extra dependencies.
+ download_dir = os.path.join(here, 'downloads')
+ os.mkdir(download_dir)
+ for download in downloads:
+ tooltool_download(download_dir, download)
+
+ # Now run each recipe in isolation.
+
+ recipe_states = []
+
+ # TODO "sandbox" process to limit network accesses to well-defined
+ # and allowed endpoints.
+ with open(os.devnull, 'wb') as devnull:
+ for recipe in recipes:
+ print('executing recipe %s' % recipe)
+ state = {}
+ full = os.path.join(recipe_dir, recipe)
+ try:
+ subprocess.check_call(full, cwd=here, stdin=devnull)
+ except subprocess.CalledProcessError as e:
+ print('ERROR: recipe %s exited %d' % (recipe, e.returncode))
+ return 1
+
+ for s in ('persisted', 'temporary'):
+ path = Path(here, 'system-packages.%s' % s)
+ try:
+ with open(path, 'r') as fh:
+ packages = set(p.strip() for p in fh if p.strip())
+
+ state['system-packages-%s' % s] = packages
+ os.unlink(path)
+ except FileNotFoundError:
+ pass
+
+ recipe_states.append(state)
+
+ all_persisted_packages = set()
+ all_temporary_packages = set()
+ for state in recipe_states:
+ all_persisted_packages |= state.get('system-packages-persisted', set())
+ all_temporary_packages |= state.get('system-packages-temporary', set())
+
+ # Prune temporary packages.
+ remove_packages(all_temporary_packages - all_persisted_packages)
+
+ # Normalize some permissions.
+ u_worker = pwd.getpwnam('worker')
+ g_worker = grp.getgrnam('worker')
+ uid = u_worker.pw_uid
+ gid = g_worker.gr_gid
+
+ for root, dirs, files in os.walk('/builds/worker'):
+ os.chown(root, uid, gid)
+
+ for f in files:
+ os.chown(os.path.join(root, f), uid, gid)
+
+ for root, dirs, files in os.walk('/builds/worker/bin'):
+ for f in files:
+ path = os.path.join(root, f)
+
+ mode = os.stat(path).st_mode
+ mode |= stat.S_IXUSR | stat.S_IXGRP
+ os.chmod(path, mode)
+
+ if have_apt():
+ subprocess.check_call(['apt-get', 'clean'])
+ for f in sorted(os.listdir('/var/lib/apt/lists')):
+ p = Path('/var/lib/apt/lists', f)
+ if p.is_dir():
+ shutil.rmtree(p)
+ else:
+ p.unlink()
+
+ shutil.rmtree(here)
+ return 0
+
+
+if __name__ == '__main__':
+ here = os.path.abspath(os.path.dirname(sys.argv[0]))
+
+ if len(sys.argv) > 1:
+ action = sys.argv[1]
+
+ # `$0 tooltool` will process the tooltool manifest passed via stdin.
+ if action == 'tooltool':
+ manifest = json.load(sys.stdin)
+ for entry in manifest:
+ tooltool_download(os.path.join(here, 'downloads'), entry)
+
+ sys.exit(0)
+
+ sys.exit(main(here))
new file mode 100644
--- /dev/null
+++ b/taskcluster/docker/bootstrap/shell-helper.sh
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+# This script provides helper functions for shell-based recipes. Simply source
+# it from a script to get useful helper functions.
+
+export DEBIAN_FRONTEND=noninteractive
+
+# Install a system package using the system package manager.
+install_system_packages () {
+ for ent in "$@"; do
+ echo ${ent} >> system-packages.persisted
+ done
+
+ apt-get install -y ${@}
+}
+
+# Like ``install_system_packages`` but the added packages will be
+# removed once all recipes have executed. This is used to install
+# packages needed to build the image but that aren't needed by the
+# image itself.
+install_temporary_system_packages() {
+ for ent in "$@"; do
+ echo ${ent} >> system-packages.temporary
+ done
+
+ apt-get install -y ${@}
+}
+
+# Process a tooltool manifest sent via stdin.
+tooltool_download() {
+ python-bootstrap/bin/python3.6 -u build-image-main tooltool
+}
deleted file mode 100644
--- a/taskcluster/docker/lint/Dockerfile
+++ /dev/null
@@ -1,49 +0,0 @@
-FROM ubuntu:16.04
-MAINTAINER Andrew Halberstadt <ahalberstadt@mozilla.com>
-
-RUN mkdir /builds
-RUN useradd -d /builds/worker -s /bin/bash -m worker
-WORKDIR /builds/worker
-
-VOLUME /builds/worker/.cache
-VOLUME /builds/worker/checkouts
-
-RUN mkdir /build
-# %include python/mozbuild/mozbuild/action/tooltool.py
-ADD topsrcdir/python/mozbuild/mozbuild/action/tooltool.py /build/tooltool.py
-
-# %include testing/mozharness/external_tools/robustcheckout.py
-ADD topsrcdir/testing/mozharness/external_tools/robustcheckout.py /usr/local/mercurial/robustcheckout.py
-
-# %include taskcluster/docker/recipes/install-node.sh
-ADD topsrcdir/taskcluster/docker/recipes/install-node.sh /build/install-node.sh
-
-# %include taskcluster/docker/recipes/install-mercurial.sh
-ADD topsrcdir/taskcluster/docker/recipes/install-mercurial.sh /build/install-mercurial.sh
-ADD system-setup.sh /tmp/system-setup.sh
-# %include tools/lint/eslint/manifest.tt
-ADD topsrcdir/tools/lint/eslint/manifest.tt /tmp/eslint.tt
-# %include tools/lint/eslint/eslint-plugin-mozilla/manifest.tt
-ADD topsrcdir/tools/lint/eslint/eslint-plugin-mozilla/manifest.tt /tmp/eslint-plugin-mozilla.tt
-# %include tools/lint/python/flake8_requirements.txt
-ADD topsrcdir/tools/lint/python/flake8_requirements.txt /tmp/flake8_requirements.txt
-# %include tools/lint/tox/tox_requirements.txt
-ADD topsrcdir/tools/lint/tox/tox_requirements.txt /tmp/tox_requirements.txt
-RUN bash /tmp/system-setup.sh
-
-# %include taskcluster/docker/recipes/run-task
-ADD topsrcdir/taskcluster/docker/recipes/run-task /builds/worker/bin/run-task
-RUN chown -R worker:worker /builds/worker/bin && chmod 755 /builds/worker/bin/*
-
-# Set variable normally configured at login, by the shells parent process, these
-# are taken from GNU su manual
-ENV HOME /builds/worker
-ENV SHELL /bin/bash
-ENV USER worker
-ENV LOGNAME worker
-ENV HOSTNAME taskcluster-worker
-ENV LANG en_US.UTF-8
-ENV LC_ALL en_US.UTF-8
-
-# Set a default command useful for debugging
-CMD ["/bin/bash", "--login"]
new file mode 100644
--- /dev/null
+++ b/taskcluster/docker/lint/image.yml
@@ -0,0 +1,20 @@
+flavor: ubuntu1604
+
+volumes:
+ - /builds/worker/.cache
+ - /builds/worker/checkouts
+
+tooltool-manifests:
+ - tools/lint/eslint/manifest.tt
+ - tools/lint/eslint/eslint-plugin-mozilla/manifest.tt
+
+recipes:
+ - install-fzf.sh
+ - install-node.sh
+
+extra-files:
+ - tools/lint/python/flake8_requirements.txt
+ - tools/lint/tox/tox_requirements.txt
+
+local-recipes:
+ - system-setup.sh
old mode 100644
new mode 100755
--- a/taskcluster/docker/lint/system-setup.sh
+++ b/taskcluster/docker/lint/system-setup.sh
@@ -1,91 +1,27 @@
#!/usr/bin/env bash
-# This allows ubuntu-desktop to be installed without human interaction
-export DEBIAN_FRONTEND=noninteractive
set -ve
-test `whoami` == 'root'
-
-mkdir -p /setup
-cd /setup
+. shell-helper.sh
-apt_packages=()
-apt_packages+=('curl')
-apt_packages+=('locales')
-apt_packages+=('git')
-apt_packages+=('python')
-apt_packages+=('python-pip')
-apt_packages+=('python3')
-apt_packages+=('python3-pip')
-apt_packages+=('sudo')
-apt_packages+=('wget')
-apt_packages+=('xz-utils')
-
-apt-get update
-apt-get install -y ${apt_packages[@]}
-
+install_system_packages locales
# Without this we get spurious "LC_ALL: cannot change locale (en_US.UTF-8)" errors,
# and python scripts raise UnicodeEncodeError when trying to print unicode characters.
locale-gen en_US.UTF-8
dpkg-reconfigure locales
+install_system_packages git
su -c 'git config --global user.email "worker@mozilla.test"' worker
su -c 'git config --global user.name "worker"' worker
-tooltool_fetch() {
- cat >manifest.tt
- /build/tooltool.py fetch
- rm manifest.tt
-}
-
-cd /build
-. install-mercurial.sh
-
-###
-# ESLint Setup
-###
-
-# install node
-
-. install-node.sh
-
-/build/tooltool.py fetch -m /tmp/eslint.tt
-mv /build/node_modules /build/node_modules_eslint
-/build/tooltool.py fetch -m /tmp/eslint-plugin-mozilla.tt
-mv /build/node_modules /build/node_modules_eslint-plugin-mozilla
-
-###
-# fzf setup
-###
+mkdir -p /build
+mv /image-build/downloads/eslint/node_modules /build/node_modules_eslint
+mv /image-build/downloads/eslint-plugin-mozilla/node_modules /build/node_modules_eslint-plugin-mozilla
-tooltool_fetch <<EOF
-[
- {
- "size": 866160,
- "digest": "9f0ef6bf44b8622bd0e4e8b0b5b5c714c0a2ce4487e6f234e7d4caac458164c521949f4d84b8296274e8bd20966f835e26f6492ba499405d38b620181e82429e",
- "algorithm": "sha512",
- "filename": "fzf-0.16.11-linux_amd64.tgz",
- "unpack": true
- }
-]
-EOF
-mv fzf /usr/local/bin
+# Install Python packages required by various jobs.
+install_system_packages python-pip
+/usr/bin/pip2 install --require-hashes -r /image-build/flake8_requirements.txt
+/usr/bin/pip2 install --require-hashes -r /image-build/tox_requirements.txt
-###
-# Flake8 Setup
-###
-
-cd /setup
-
-pip install --require-hashes -r /tmp/flake8_requirements.txt
-
-###
-# tox Setup
-###
-
-cd /setup
-
-pip install --require-hashes -r /tmp/tox_requirements.txt
-
-cd /
-rm -rf /setup
+# Python 3 is required by various jobs.
+install_system_packages python3-pip
new file mode 100755
--- /dev/null
+++ b/taskcluster/docker/recipes/install-fzf.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+. shell-helper.sh
+
+tooltool_download <<EOF
+[
+ {
+ "size": 866160,
+ "digest": "9f0ef6bf44b8622bd0e4e8b0b5b5c714c0a2ce4487e6f234e7d4caac458164c521949f4d84b8296274e8bd20966f835e26f6492ba499405d38b620181e82429e",
+ "algorithm": "sha512",
+ "filename": "fzf-0.16.11-linux_amd64.tar.gz",
+ "unpack": true
+ }
+]
+EOF
+
+mv /image-build/downloads/fzf-0.16.11-linux_amd64/fzf /usr/local/bin
old mode 100644
new mode 100755
--- a/taskcluster/docker/recipes/install-mercurial.sh
+++ b/taskcluster/docker/recipes/install-mercurial.sh
@@ -2,16 +2,28 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
# This script installs and configures Mercurial.
set -e
+# If using our new image build tool, source its includes, set up a tooltool
+# helper, and install package dependencies.
+if [ -f shell-helper.sh ]; then
+ . shell-helper.sh
+
+ tooltool_fetch() {
+ tooltool_download $@
+ }
+
+ install_system_packages python
+fi
+
# Detect OS.
if [ -f /etc/lsb-release ]; then
. /etc/lsb-release
if [ "${DISTRIB_ID}" = "Ubuntu" -a "${DISTRIB_RELEASE}" = "16.04" ]; then
HG_DEB=1
HG_DIGEST=dd4dd7759fe73985b6a0424b34a3036d130c26defdd866a9fdd7302e40c7417433b93f020497ceb40593eaead8e86be55e48340887015645202b47ff7b0d7ac6
HG_SIZE=181722
@@ -57,48 +69,67 @@ tooltool_fetch <<EOF
"size": ${HG_COMMON_SIZE},
"digest": "${HG_COMMON_DIGEST}",
"algorithm": "sha512",
"filename": "${HG_COMMON_FILENAME}"
}
]
EOF
- dpkg -i ${HG_COMMON_FILENAME} ${HG_FILENAME}
+ if [ -f downloads/${HG_COMMON_FILENAME} ]; then
+ dpkg -i downloads/${HG_COMMON_FILENAME} downloads/${HG_FILENAME}
+ else
+ dpkg -i ${HG_COMMON_FILENAME} ${HG_FILENAME}
+ fi
elif [ -n "${HG_RPM}" ]; then
tooltool_fetch <<EOF
[
{
"size": ${HG_SIZE},
"digest": "${HG_DIGEST}",
"algorithm": "sha512",
"filename": "${HG_FILENAME}"
}
]
EOF
- rpm -i ${HG_FILENAME}
+ if [ -f downloads/${HG_FILENAME} ]; then
+ rpm -i downloads/${HG_FILENAME}
+ else
+ rpm -i ${HG_FILENAME}
+ fi
elif [ -n "${PIP_PATH}" ]; then
tooltool_fetch <<EOF
[
{
"size": 5475042,
"digest": "4c42d06b7f111a3e825dd927704a30f88f0b2225cf87ab8954bf53a7fbc0edf561374dd49b13d9c10140d98ff5853a64acb5a744349727abae81d32da401922b",
"algorithm": "sha512",
"filename": "mercurial-4.3.1.tar.gz"
}
]
EOF
- ${PIP_PATH} install mercurial-4.3.1.tar.gz
+ if [ -f downloads/mercurial-4.3.1.tar.gz ]; then
+ ${PIP_PATH} install downloads/mercurial-4.3.1.tar.gz
+ else
+ ${PIP_PATH} install mercurial-4.3.1.tar.gz
+ fi
else
echo "Do not know how to install Mercurial on this OS"
exit 1
fi
+# Robustcheckout is either in present directory for new bootstrap tool
+# or installed in final place via ADD in Dockerfile.
+if [ -f robustcheckout.py ]; then
+ mkdir -p /usr/local/mercurial
+ cp robustcheckout.py /usr/local/mercurial/robustcheckout.py
+fi
+
chmod 644 /usr/local/mercurial/robustcheckout.py
mkdir -p /etc/mercurial
cat >/etc/mercurial/hgrc <<EOF
# By default the progress bar starts after 3s and updates every 0.1s. We
# change this so it shows and updates every 1.0s.
# We also tell progress to assume a TTY is present so updates are printed
# even if there is no known TTY.
old mode 100644
new mode 100755
--- a/taskcluster/docker/recipes/install-node.sh
+++ b/taskcluster/docker/recipes/install-node.sh
@@ -1,12 +1,22 @@
#!/bin/bash
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
# This script installs Node v6.
+if [ -f shell-helper.sh ]; then
+ . shell-helper.sh
+
+ install_system_packages wget
+fi
+
+# TODO download via tooltool.
wget https://nodejs.org/dist/v6.9.1/node-v6.9.1-linux-x64.tar.gz
echo 'a9d9e6308931fa2a2b0cada070516d45b76d752430c31c9198933c78f8d54b17 node-v6.9.1-linux-x64.tar.gz' | sha256sum -c
tar -C /usr/local -xz --strip-components 1 < node-v6.9.1-linux-x64.tar.gz
-node -v # verify
+
+
+# Verify.
+node -v
npm -v
--- a/taskcluster/docs/docker-images.rst
+++ b/taskcluster/docs/docker-images.rst
@@ -5,18 +5,179 @@ Docker Images
=============
TaskCluster Docker images are defined in the source directory under
``taskcluster/docker``. Each directory therein contains the name of an
image used as part of the task graph.
More information is available in the ``README.md`` file in that directory.
+Using YAML to Define Images
+===========================
+
+If an ``image.yml`` file is present in a directory under ``taskcluster/docker``,
+it will be used to define how the Docker image is built. **This is the
+preferred mechanism for defining Docker images.**
+
+When ``image.yml`` files are used, the Docker image is built using a
+custom image build mechanism that is tailored to Mozilla's needs. This
+image building mechanism ensures consistent standards and practices
+across all images and reduces the amount of copy and paste needed to
+piece together images.
+
+Essentially:
+
+* All images have a standard ``worker`` user/group with a consistent
+ UID and GID.
+* Version control packages and configuration are set to an optimal state.
+* Environment variables facilitate interactive use.
+* Combining standalone *recipes* for reuse among multiple images is
+ trivial.
+
+When ``image.yml`` images are built, a set of base files are added to the
+``/image-build`` directory in the image. See the
+``taskcluster/docker/bootstrap`` directory and the
+:py:mod:`taskgraph.util.docker` module's source code for more details.
+
+In addition, the content of the ``image.yml`` results in various
+additional files being written at well-defined locations. For example,
+all *recipes* (reusable scripts) are written to ``/image-build/recipes/``.
+
+When the image is built, a single command is run in the container. This
+script (``build-image-bootstrap``) performs basic environment configuration,
+downloads a self-contained Python 3.6 interpreter, then invokes the
+``build-image-main`` Python script.
+
+``build-image-main`` is a harness of sorts for performing additional
+actions. It will perform tooltool downloads, install system packages,
+invoke recipes, normalize permissions, and clean up after itself.
+
+The last thing ``build-image-main`` does is recursively delete
+``/image-build`` so no trace of image building is left inside the
+Docker image.
+
+``image.yml`` Content
+---------------------
+
+``image.yml`` files are dictionaries with the following keys:
+
+``flavor`` (required)
+ The type of Docker image. This essentially defines the base image.
+ Supported values are ``ubuntu1604`` (Ubuntu 16.04).
+
+``volumes``
+ A list of paths that should be declared as Docker volumes.
+
+``tooltool-manifests``
+ A list of in-tree tooltool manifest files to process. Each item in
+ referenced manifests will be downloaded. If ``unpack`` is set, the
+ item will be extracted (assuming the archive type is known from its
+ filename extension) to a directory corresponding to the filename of the
+ entry.
+
+``extra-files``
+ A list of extra files in the source repository to add to the build
+ directory. Each file is saved to its basename rather than its full
+ relative path.
+
+``recipes``
+ A list of extra files in ``taskcluster/docker/recipes`` to execute during
+ image building. Files will be executed in the order they are defined.
+
+``local-recipes``
+ A list of extra files in the image directory to be treated as recipes. These
+ are essentially image-specific recipes. They will be executed after
+ ``recipes`` using the same execution mechanism.
+
+Paths in ``image.yml`` Images
+-----------------------------
+
+All image building activity occurs in ``/image-build``.
+
+All tooltool downloads are placed in ``/image-build/downloads/``.
+
+``extra-files`` are copied to ``/image-build/<basename>``.
+
+The main user that tasks run as is ``worker:worker``. Its ``${HOME}`` is
+``/builds/worker``.
+
+Writing Recipes
+---------------
+
+Most customization in ``image.yml`` defined images is via *recipes*.
+A *recipe* is a self-contained executable (typically a shell script)
+that does some work.
+
+There are 2 flavors of recipes: *global* and *local*. Global recipes
+exist in the ``taskcluster/docker/recipes``. An entry in the ``recipes``
+list in ``image.yml`` will copy a file from this directory. *Local*
+recipes come from files in the image's directory in the source repo.
+
+Recipes are executed in the order they are defined in ``image.yml``.
+All ``recipes`` execute before all ``local-recipes``.
+
+A goal of recipes is to define self-contained and reusable functionality
+once and then to leverage that recipe from multiple Docker images just
+by running the recipe. Examples of common recipes include logical actions
+like *install Mercurial* and *install Python*.
+
+Recipes should:
+
+* Do one thing and do it well
+* Be as self-contained as possible
+
+Recipes should not:
+
+* Download anything via ``curl``, ``wget``, etc. All downloads should be
+ mediated through supported and secure download channels (namely tooltool
+ and the system package manager).
+* Invoke the system package directory. Use the wrapper exported from the
+ bootstrap framework (see below).
+
+All receipes are executed as an immediate child of the main bootstrap
+script. So if recipes are shells, they execute with a fresh environment.
+The initial working directory of a recipe is ``/image-build``.
+
+Helpers for Shell Recipes
+-------------------------
+
+It is common to write recipes in shell. The bootstrap frameworks makes
+a ``shell-helper.sh`` helper script available for providing common
+utilities. To use it, simply::
+
+ . shell-helper.sh
+
+The following functions are defined:
+
+install_system_packages $1 ...
+ Install the named packages via the system package manager.
+
+install_temporary_system_packages $1 ...
+ Install system packages but mark them as temporary such that they will be
+ removed at the end of the image build mechanism. Temporary packages will
+ be available to all subsequent recipes but they won't be part of the final
+ Docker image.
+
+tooltool_download
+ Redirect a tooltool manifest to stdin of this function to have it
+ download files within. Downloaded files will be written to
+ ``downloads/<filename>``. If ``unpack`` is set, the extracted files
+ will be in ``downloads/<basename>/``. (This behavior differs from a
+ direct tooltool.py invocation, where archives are extracted directly
+ to the destination directory.)
+
+Dockerfile Based Images
+=======================
+
+If an ``image.yml`` file is not present, then Docker images are built
+using ``Dockerfile``s. This is the standard mechanism in the Docker
+community for building images.
+
Adding Extra Files to Images
-============================
+----------------------------
Dockerfile syntax has been extended to allow *any* file from the
source checkout to be added to the image build *context*. (Traditionally
you can only ``ADD`` files from the same directory as the Dockerfile.)
Simply add the following syntax as a comment in a Dockerfile::
# %include <path>
--- a/taskcluster/taskgraph/util/docker.py
+++ b/taskcluster/taskgraph/util/docker.py
@@ -1,26 +1,46 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
from __future__ import absolute_import, print_function, unicode_literals
import hashlib
+import json
import os
import shutil
import subprocess
import tarfile
import tempfile
+from voluptuous import (
+ Any,
+ Optional,
+ Required,
+)
+
+import yaml
+
from mozbuild.util import memoize
from mozpack.archive import (
create_tar_gz_from_files,
)
+from mozpack.files import (
+ ExecutableFile,
+ FileFinder,
+ GeneratedFile,
+)
+import mozpack.path as mozpath
+
from .. import GECKO
+from .schema import (
+ Schema,
+ validate_schema,
+)
IMAGE_DIR = os.path.join(GECKO, 'taskcluster', 'docker')
INDEX_PREFIX = 'docker.images.v2'
def docker_image(name, by_tag=False):
'''
@@ -59,38 +79,48 @@ def generate_context_hash(topsrcdir, ima
os.close(fd)
try:
return create_context_tar(topsrcdir, image_path, p, image_name)
finally:
os.unlink(p)
def create_context_tar(topsrcdir, context_dir, out_path, prefix):
- """Create a context tarball.
+ """Create a context tarball for building an image.
- A directory ``context_dir`` containing a Dockerfile will be assembled into
- a gzipped tar file at ``out_path``. Files inside the archive will be
- prefixed by directory ``prefix``.
+ A directory ``context_dir`` containing a Docker image definition will be
+ assembled into a gzipped tar file at ``out_path``. Files inside the archive
+ will be prefixed by directory ``prefix``.
- We also scan the source Dockerfile for special syntax that influences
+ The directory is either traditional or Mozilla style.
+
+ In a traditional directory, a ``Dockerfile`` is used to build the image.
+ We scan the source ``Dockerfile`` for special syntax that influences
context generation.
If a line in the ``Dockerfile`` has the form ``# %include <path>``,
the relative path specified on that line will be matched against
files in the source repository and added to the context under the
path ``topsrcdir/``. If an entry is a directory, we add all files
under that directory.
+ In a Mozilla-style directory, an ``image.yml`` file is used to define
+ the Docker image. See the ReST documentation for the content of this
+ file. Content from the ``image.yml`` file is used to dynamically
+ generate a ``Dockerfile`` to be used to build the Docker image.
+
Returns the SHA-256 hex digest of the created archive.
"""
- if os.path.exists(os.path.join(context_dir, 'Dockerfile')):
+ if os.path.exists(os.path.join(context_dir, 'image.yml')):
+ archive_files = _archive_files_from_yaml(topsrcdir, context_dir)
+ elif os.path.exists(os.path.join(context_dir, 'Dockerfile')):
archive_files = _archive_files_from_dockerfile(topsrcdir, context_dir)
else:
raise Exception('do not know how to build docker image from %s; must '
- 'have Dockerfile' % context_dir)
+ 'have Dockerfile or image.yml' % context_dir)
archive_files = {os.path.join(prefix, k): v
for k, v in archive_files.items()}
with open(out_path, 'wb') as fh:
create_tar_gz_from_files(fh, archive_files, '%s.tar.gz' % prefix)
h = hashlib.sha256()
@@ -139,16 +169,171 @@ def _archive_files_from_dockerfile(topsr
archive_files[archive_path] = source_path
else:
archive_path = os.path.join('topsrcdir', p)
archive_files[archive_path] = fs_path
return archive_files
+# Base Docker images we support.
+IMAGE_FLAVORS = {
+ 'ubuntu1604': {
+ 'docker-image': 'ubuntu:16.04',
+ },
+}
+
+IMAGE_SCHEMA = Schema({
+ # The base Docker image. Defined as name and a hash.
+ Required('flavor'): Any(*IMAGE_FLAVORS.keys()),
+
+ # Paths that are volumes or caches.
+ Optional('volumes'): [basestring],
+
+ Optional('tooltool-manifests'): [basestring],
+
+ # Files in recipes directory to execute.
+ Optional('recipes'): [basestring],
+
+ Optional('local-recipes'): [basestring],
+
+ Optional('extra-files'): [basestring],
+})
+
+# Directories and files to always be added to image.yml contexts.
+COMMON_DIRS = {
+ 'taskcluster/docker/bootstrap',
+}
+
+COMMON_FILES = {
+ 'taskcluster/docker/recipes/run-task',
+ 'testing/mozharness/external_tools/robustcheckout.py',
+}
+
+# Common variables to set in the image. These are normally configured at
+# login. The set is taken from the GNU su manual.
+COMMON_ENV = {
+ 'HOME': '/builds/worker',
+ 'SHELL': '/bin/bash',
+ 'USER': 'worker',
+ 'LOGNAME': 'worker',
+ 'HOSTNAME': 'taskcluster-worker',
+ 'LANG': 'en_US.UTF-8',
+ 'LC_ALL': 'en_US.UTF-8',
+}
+
+
+def _archive_files_from_yaml(topsrcdir, context_dir):
+ """Assemble an image build context from our custom YAML.
+
+ Goals:
+
+ * Isolate all file ADDs to /image-build
+ * Minimize Dockerfile content
+ * Keep all images as similar as possible
+ """
+ with open(os.path.join(context_dir, 'image.yml'), 'rb') as fh:
+ image = yaml.load(fh)
+
+ validate_schema(IMAGE_SCHEMA, image, '%s/image.yml:' % os.path.basename(
+ context_dir))
+
+ flavor = IMAGE_FLAVORS[image['flavor']]
+
+ lines = []
+ archive_files = {}
+ recipe_index = [0]
+
+ def add_recipe(name, source=None):
+ recipe_index[0] += 1
+ p = mozpath.join('image-build', 'recipes', '%03d-%s' % (recipe_index[0],
+ name))
+ if not source:
+ source = os.path.join(topsrcdir, 'taskcluster', 'docker', 'recipes',
+ name)
+
+ archive_files[p] = ExecutableFile(source)
+
+ lines.append('FROM %s' % flavor['docker-image'])
+
+ # Add common directories and files.
+ for d in sorted(COMMON_DIRS):
+ finder = FileFinder(os.path.join(topsrcdir, d), find_dotfiles=True)
+ for p, f in finder.find('**'):
+ p = mozpath.join('image-build', p)
+ if p in archive_files:
+ raise Exception('path %s already exists in archive' % p)
+ archive_files[p] = f
+
+ for f in sorted(COMMON_FILES):
+ p = mozpath.join('image-build', mozpath.basename(f))
+ if p in archive_files:
+ raise Exception('path %s already exists in archive' % p)
+
+ archive_files[p] = os.path.join(topsrcdir, f)
+
+ # Tooltool downloads are aggregated and added to a central file.
+ downloads = []
+
+ # And entries from referenced tooltool manifests.
+ for path in image.get('tooltool-manifests', []):
+ with open(os.path.join(topsrcdir, path), 'rb') as fh:
+ m = json.load(fh)
+
+ downloads.extend(m)
+
+ archive_files['image-build/tooltool-manifest.tt'] = GeneratedFile(
+ json.dumps(downloads, sort_keys=True))
+
+ # Recipes are installed into the recipes directory. The filename is prefixed
+ # with the order the recipe is defined in so recipes execute in that order.
+
+ # Always install Mercurial.
+ add_recipe('install-mercurial.sh')
+
+ for recipe in image.get('recipes', []):
+ add_recipe(recipe)
+
+ for recipe in image.get('local-recipes', []):
+ add_recipe('local-%s' % recipe, os.path.join(context_dir, recipe))
+
+ # Add all other extra files.
+ for path in image.get('extra-files', []):
+ p = mozpath.join('image-build', mozpath.basename(path))
+ archive_files[p] = os.path.join(topsrcdir, path)
+
+ # Add all our files to the image.
+ lines.append('ADD image-build/ /image-build/')
+
+ # Run our main image builder as one step. This should be the only RUN
+ # in the Dockerfile.
+ lines.append('RUN /image-build/build-image-bootstrap')
+
+ # Convert volumes to VOLUME.
+ #
+ # This needs to be after RUN so /builds/worker doesn't exist during
+ # image building.
+ if image.get('volumes'):
+ lines.append('VOLUME %s' % ' '.join(image['volumes']))
+
+ lines.append('ENV %s' % ' '.join(
+ '='.join(t) for t in sorted(COMMON_ENV.items())))
+
+ # Set a default command useful for debugging.
+ lines.append('CMD ["/bin/bash", "--login"]')
+
+ # All meaningful actions should be done as part of the building framework.
+ # Verify that.
+ assert len([s for s in lines if s.startswith('RUN ')]) == 1
+
+ # Add generated Dockerfile.
+ archive_files['Dockerfile'] = GeneratedFile(
+ b'\n'.join(s.encode('utf-8') for s in lines))
+
+ return archive_files
def build_from_context(docker_bin, context_path, prefix, tag=None):
"""Build a Docker image from a context archive.
Given the path to a `docker` binary, a image build tar.gz (produced with
``create_context_tar()``, a prefix in that context containing files, and
an optional ``tag`` for the produced image, build that Docker image.
@@ -177,17 +362,28 @@ def build_from_context(docker_bin, conte
if res:
raise Exception('error building image')
finally:
shutil.rmtree(d)
@memoize
def parse_volumes(image):
- """Parse VOLUME entries from a Dockerfile for an image."""
+ """Determine volumes in a Docker image.
+
+ Parses either an image.yml or a Dockerfile.
+ """
+ image_yml = os.path.join(IMAGE_DIR, image, 'image.yml')
+
+ if os.path.exists(image_yml):
+ with open(image_yml, 'rb') as fh:
+ image = yaml.load(fh)
+
+ return set(image.get('volumes', []))
+
volumes = set()
with open(os.path.join(IMAGE_DIR, image, 'Dockerfile'), 'rb') as fh:
for line in fh:
line = line.strip()
if not line.startswith(b'VOLUME '):
continue
