Bug 1460777 - Taskgraph tasks for retrieving remote content; r?dustin, glandium
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--- a/build/unix/build-gcc/build-gcc.sh
+++ b/build/unix/build-gcc/build-gcc.sh
@@ -2,99 +2,16 @@
set -e
set -x
make_flags="-j$(nproc)"
. $data_dir/download-tools.sh
-prepare() {
- pushd $root_dir
- download_and_check https://ftp.gnu.org/gnu/binutils binutils-$binutils_version.tar.$binutils_ext.sig
- tar xaf $TMPDIR/binutils-$binutils_version.tar.$binutils_ext
-
- case "$gcc_version" in
- *-*)
- download https://gcc.gnu.org/pub/gcc/snapshots/$gcc_version/gcc-$gcc_version.tar.$gcc_ext
- ;;
- *)
- download_and_check https://ftp.gnu.org/gnu/gcc/gcc-$gcc_version gcc-$gcc_version.tar.$gcc_ext.sig
- ;;
- esac
- tar xaf $TMPDIR/gcc-$gcc_version.tar.$gcc_ext
- cd gcc-$gcc_version
-
- (
- # Divert commands that download_prerequisites use
- ln() { :; }
- tar() { :; }
- sed() { :; }
- wget() {
- # Get last argument given to wget.
- eval echo \$$# >&3
- }
-
- # In GCC >= 7, the download_prerequisites script tried to do its own
- # verification, but we have ours, so disable it.
- set -- --no-verify
- . ./contrib/download_prerequisites
- ) 3>&1 > /dev/null | while read url; do
- file=$(basename $url)
- case "$file" in
- gmp-*.tar.*)
- # If download_prerequisites wants 4.3.2, use 5.1.3 instead.
- file=${file/4.3.2/5.1.3}
- download_and_check https://ftp.gnu.org/gnu/gmp/ $file.sig
- ;;
- mpfr-*.tar.*)
- # If download_prerequisites wants 2.4.2, use 3.1.5 instead.
- file=${file/2.4.2/3.1.5}
- download_and_check https://ftp.gnu.org/gnu/mpfr/ $file.sig
- ;;
- mpc-*.tar.*)
- # If download_prerequisites wants 0.8.1, use 0.8.2 instead.
- file=${file/0.8.1/0.8.2}
- case "$file" in
- *-0.8.2.tar*|*-0.9.tar*|*-1.0.tar*)
- ext=asc
- ;;
- *)
- ext=sig
- ;;
- esac
- case "$file" in
- *-0.8.2.tar*)
- # The ftp.gnu.org mirror doesn't have 0.8.2, so we use
- # a debian source tarball instead. It has a different name,
- # so we can't use `download`. Manually do what it does, but
- # handling the difference in file name.
- wget -c --progress=dot:mega -O $TMPDIR/$file http://snapshot.debian.org/archive/debian/20100527T162226Z/pool/main/m/mpclib/mpclib_0.8.2.orig.tar.gz
- (cd $TMPDIR; sha256sum $file) >> $root_dir/downloads
- ;;
- *)
- download_and_check https://ftp.gnu.org/gnu/mpc $file.$ext
- ;;
- esac
- ;;
- *)
- download $(dirname $url) $file
- ;;
- esac
- tar xaf $TMPDIR/$file
- ln -sf ${file%.tar.*} ${file%-*}
- done
-
- # Check all the downloads we did are in the checksums list, and that the
- # checksums match.
- diff -u <(sort -k 2 $root_dir/downloads) $root_dir/checksums
-
- popd
-}
-
prepare_mingw() {
export prefix=/tools/mingw32
export install_dir=$root_dir$prefix
mkdir -p $install_dir
export PATH=$PATH:$install_dir/bin/
cd $root_dir
--- a/taskcluster/ci/docker-image/kind.yml
+++ b/taskcluster/ci/docker-image/kind.yml
@@ -78,16 +78,19 @@ jobs:
BASE_TAG: '20170620'
SNAPSHOT: '20170830T000511Z'
packages:
- deb9-mercurial
- deb9-python-zstandard
android-build:
symbol: I(agb)
parent: debian9-base
+ fetch:
+ symbol: I(fetch)
+ parent: debian9-base
mingw32-build:
symbol: I(mingw)
parent: debian9-base
index-task:
symbol: I(idx)
funsize-update-generator:
symbol: I(pg)
google-play-strings:
new file mode 100644
--- /dev/null
+++ b/taskcluster/ci/fetch/kind.yml
@@ -0,0 +1,24 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+loader: taskgraph.loader.transform:loader
+
+transforms:
+ - taskgraph.transforms.try_job:transforms
+ - taskgraph.transforms.job:transforms
+ - taskgraph.transforms.task:transforms
+
+job-defaults:
+ run-on-projects: []
+ treeherder:
+ kind: build
+ platform: fetch/opt
+ tier: 1
+ worker-type: aws-provisioner-v1/gecko-{level}-images
+ worker:
+ docker-image: {in-tree: fetch}
+ max-run-time: 900
+
+jobs-from:
+ - toolchains.yml
new file mode 100644
--- /dev/null
+++ b/taskcluster/ci/fetch/toolchains.yml
@@ -0,0 +1,195 @@
+binutils-2.25.1:
+ description: binutils 2.25.1 source code
+ treeherder:
+ symbol: binutils2.25.1
+ run:
+ using: fetch-url
+ url: ftp://ftp.gnu.org/gnu/binutils/binutils-2.25.1.tar.bz2
+ sha256: b5b14added7d78a8d1ca70b5cb75fef57ce2197264f4f5835326b0df22ac9f22
+ size: 24163561
+ gpg-signature:
+ sig-url: "{url}.sig"
+ key-path: build/unix/build-gcc/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
+
+binutils-2.27:
+ description: binutils 2.27 source code
+ treeherder:
+ symbol: binutils2.27
+ run:
+ using: fetch-url
+ url: ftp://ftp.gnu.org/gnu/binutils/binutils-2.27.tar.bz2
+ sha256: 369737ce51587f92466041a97ab7d2358c6d9e1b6490b3940eb09fb0a9a6ac88
+ size: 26099568
+ gpg-signature:
+ sig-url: "{url}.sig"
+ key-path: build/unix/build-gcc/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
+
+binutils-2.28.1:
+ description: binutils 2.28.1 source code
+ treeherder:
+ symbol: binutils2.28.1
+ run:
+ using: fetch-url
+ url: ftp://ftp.gnu.org/gnu/binutils/binutils-2.28.1.tar.xz
+ sha256: 16328a906e55a3c633854beec8e9e255a639b366436470b4f6245eb0d2fde942
+ size: 19440112
+ gpg-signature:
+ sig-url: "{url}.sig"
+ key-path: build/unix/build-gcc/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
+
+cloog-0.18.1:
+ description: cloog source code
+ treeherder:
+ symbol: cloog0.18.1
+ run:
+ using: fetch-url
+ url: ftp://gcc.gnu.org/pub/gcc/infrastructure/cloog-0.18.1.tar.gz
+ sha256: 02500a4edd14875f94fe84cbeda4290425cb0c1c2474c6f75d75a303d64b4196
+ size: 3857324
+
+gcc-4.9.4:
+ description: GCC 4.9.4 source code
+ treeherder:
+ symbol: gcc4.9.4
+ run:
+ using: fetch-url
+ url: ftp://ftp.gnu.org/gnu/gcc/gcc-4.9.4/gcc-4.9.4.tar.bz2
+ sha256: 6c11d292cd01b294f9f84c9a59c230d80e9e4a47e5c6355f046bb36d4f358092
+ size: 90097606
+ gpg-signature:
+ sig-url: "{url}.sig"
+ key-path: build/unix/build-gcc/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
+
+gcc-6.4.0:
+ description: GCC 6.4.0 source code
+ treeherder:
+ symbol: gcc6.4.0
+ run:
+ using: fetch-url
+ url: ftp://ftp.gnu.org/gnu/gcc/gcc-6.4.0/gcc-6.4.0.tar.xz
+ sha256: 850bf21eafdfe5cd5f6827148184c08c4a0852a37ccf36ce69855334d2c914d4
+ size: 76156220
+ gpg-signature:
+ sig-url: "{url}.sig"
+ key-path: build/unix/build-gcc/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
+
+gcc-7.3.0:
+ description: GCC 7.3.0 source code
+ treeherder:
+ symbol: gcc7.3.0
+ run:
+ using: fetch-url
+ url: ftp://ftp.gnu.org/gnu/gcc/gcc-7.3.0/gcc-7.3.0.tar.xz
+ sha256: 832ca6ae04636adbb430e865a1451adf6979ab44ca1c8374f61fba65645ce15c
+ size: 62462388
+ gpg-signature:
+ sig-url: "{url}.sig"
+ key-path: build/unix/build-gcc/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
+
+gmp-5.1.3:
+ description: GMP 5.1.3 source code
+ treeherder:
+ symbol: gmp5.1.3
+ run:
+ using: fetch-url
+ url: https://ftp.gnu.org/gnu/gmp/gmp-5.1.3.tar.bz2
+ sha256: 752079520b4690531171d0f4532e40f08600215feefede70b24fabdc6f1ab160
+ size: 2196480
+ gpg-signature:
+ sig-url: "{url}.sig"
+ key-path: build/unix/build-gcc/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
+
+gmp-6.1.0:
+ description: GMP 6.1.0 source code
+ treeherder:
+ symbol: gmp6.1.0
+ run:
+ using: fetch-url
+ url: https://ftp.gnu.org/gnu/gmp/gmp-6.1.0.tar.bz2
+ sha256: 498449a994efeba527885c10405993427995d3f86b8768d8cdf8d9dd7c6b73e8
+ size: 2383840
+ gpg-signature:
+ sig-url: "{url}.sig"
+ key-path: build/unix/build-gcc/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
+
+isl-0.12.2:
+ description: ISL 0.12.2 source code
+ treeherder:
+ symbol: isl0.12.2
+ run:
+ using: fetch-url
+ url: ftp://gcc.gnu.org/pub/gcc/infrastructure/isl-0.12.2.tar.bz2
+ sha256: f4b3dbee9712850006e44f0db2103441ab3d13b406f77996d1df19ee89d11fb4
+ size: 1319434
+
+isl-0.15:
+ description: ISL 0.15 source code
+ treeherder:
+ symbol: isl0.15
+ run:
+ using: fetch-url
+ url: ftp://gcc.gnu.org/pub/gcc/infrastructure/isl-0.15.tar.bz2
+ sha256: 8ceebbf4d9a81afa2b4449113cee4b7cb14a687d7a549a963deb5e2a41458b6b
+ size: 1574964
+
+isl-0.16.1:
+ description: ISL 0.16.1 source code
+ treeherder:
+ symbol: isl0.16.1
+ run:
+ using: fetch-url
+ url: ftp://gcc.gnu.org/pub/gcc/infrastructure/isl-0.16.1.tar.bz2
+ sha256: 412538bb65c799ac98e17e8cfcdacbb257a57362acfaaff254b0fcae970126d2
+ size: 1626446
+
+mpc-0.8.2:
+ description: mpc 0.8.2 source code
+ treeherder:
+ symbol: mpc0.8.2
+ run:
+ using: fetch-url
+ url: http://www.multiprecision.org/downloads/mpc-0.8.2.tar.gz
+ sha256: ae79f8d41d8a86456b68607e9ca398d00f8b7342d1d83bcf4428178ac45380c7
+ size: 548401
+ gpg-signature:
+ sig-url: "{url}.asc"
+ key-path: build/unix/build-gcc/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
+
+mpc-1.0.3:
+ description: mpc 1.0.3 source code
+ treeherder:
+ symbol: mpc1.0.3
+ run:
+ using: fetch-url
+ url: http://www.multiprecision.org/downloads/mpc-1.0.3.tar.gz
+ sha256: 617decc6ea09889fb08ede330917a00b16809b8db88c29c31bfbb49cbf88ecc3
+ size: 669925
+ gpg-signature:
+ sig-url: "{url}.sig"
+ key-path: build/unix/build-gcc/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
+
+mpfr-3.1.4:
+ description: mpfr 3.1.4 source code
+ treeherder:
+ symbol: mpfr3.1.4
+ run:
+ using: fetch-url
+ url: http://www.mpfr.org/mpfr-3.1.4/mpfr-3.1.4.tar.bz2
+ sha256: d3103a80cdad2407ed581f3618c4bed04e0c92d1cf771a65ead662cc397f7775
+ size: 1279284
+ gpg-signature:
+ sig-url: "{url}.asc"
+ key-path: build/unix/build-gcc/07F3DBBECC1A39605078094D980C197698C3739D.key
+
+mpfr-3.1.5:
+ description: mpfr 3.1.5 source code
+ treeherder:
+ symbol: mpfr3.1.5
+ run:
+ using: fetch-url
+ url: http://www.mpfr.org/mpfr-3.1.5/mpfr-3.1.5.tar.bz2
+ sha256: ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658
+ size: 1279489
+ gpg-signature:
+ sig-url: "{url}.asc"
+ key-path: build/unix/build-gcc/07F3DBBECC1A39605078094D980C197698C3739D.key
--- a/taskcluster/ci/toolchain/kind.yml
+++ b/taskcluster/ci/toolchain/kind.yml
@@ -1,16 +1,20 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
loader: taskgraph.loader.transform:loader
+kind-dependencies:
+ - fetch
+
transforms:
- taskgraph.transforms.try_job:transforms
+ - taskgraph.transforms.use_fetches:transforms
- taskgraph.transforms.use_toolchains:transforms
- taskgraph.transforms.job:transforms
- taskgraph.transforms.task:transforms
job-defaults:
run-on-projects: []
jobs-from:
--- a/taskcluster/ci/toolchain/linux.yml
+++ b/taskcluster/ci/toolchain/linux.yml
@@ -156,16 +156,24 @@ linux64-gcc-4.9:
worker:
max-run-time: 36000
run:
using: toolchain-script
script: build-gcc-4.9-linux.sh
resources:
- 'build/unix/build-gcc/build-gcc.sh'
toolchain-artifact: public/build/gcc.tar.xz
+ fetches:
+ - binutils-2.25.1
+ - cloog-0.18.1
+ - gcc-4.9.4
+ - gmp-5.1.3
+ - isl-0.12.2
+ - mpc-0.8.2
+ - mpfr-3.1.5
linux64-gcc-6:
description: "GCC 6 toolchain build"
treeherder:
kind: build
platform: toolchains/opt
symbol: TL(gcc6)
tier: 1
@@ -174,16 +182,23 @@ linux64-gcc-6:
max-run-time: 36000
run:
using: toolchain-script
script: build-gcc-6-linux.sh
resources:
- 'build/unix/build-gcc/build-gcc.sh'
toolchain-alias: linux64-gcc
toolchain-artifact: public/build/gcc.tar.xz
+ fetches:
+ - binutils-2.28.1
+ - gcc-6.4.0
+ - gmp-5.1.3
+ - isl-0.15
+ - mpc-0.8.2
+ - mpfr-3.1.5
linux64-gcc-7:
description: "GCC 7 toolchain build"
treeherder:
kind: build
platform: toolchains/opt
symbol: TL(gcc7)
tier: 1
@@ -191,16 +206,23 @@ linux64-gcc-7:
worker:
max-run-time: 36000
run:
using: toolchain-script
script: build-gcc-7-linux.sh
resources:
- 'build/unix/build-gcc/build-gcc.sh'
toolchain-artifact: public/build/gcc.tar.xz
+ fetches:
+ - binutils-2.28.1
+ - gcc-7.3.0
+ - gmp-6.1.0
+ - isl-0.16.1
+ - mpc-1.0.3
+ - mpfr-3.1.4
linux64-gcc-sixgill:
description: "sixgill GCC plugin build"
treeherder:
kind: build
platform: toolchains/opt
symbol: TL(sixgill)
tier: 1
@@ -211,16 +233,23 @@ linux64-gcc-sixgill:
using: toolchain-script
script: build-gcc-sixgill-plugin-linux.sh
resources:
- 'build/unix/build-gcc/build-gcc.sh'
- 'taskcluster/scripts/misc/build-gcc-sixgill-plugin-linux.sh'
toolchain-artifact: public/build/sixgill.tar.xz
toolchains:
- linux64-gcc-6
+ fetches:
+ - binutils-2.28.1
+ - isl-0.15
+ - gcc-6.4.0
+ - gmp-5.1.3
+ - mpc-0.8.2
+ - mpfr-3.1.5
linux64-llvm-dsymutil:
description: "llvm-dsymutil toolchain build"
treeherder:
kind: build
platform: toolchains/opt
symbol: TL(dsymutil)
tier: 1
@@ -565,16 +594,23 @@ linux64-mingw32-gcc:
docker-image: {in-tree: mingw32-build}
max-run-time: 10800
run:
using: toolchain-script
script: build-gcc-mingw32.sh
resources:
- 'build/unix/build-gcc/build-gcc.sh'
toolchain-artifact: public/build/mingw32.tar.xz
+ fetches:
+ - binutils-2.27
+ - gcc-6.4.0
+ - gmp-5.1.3
+ - isl-0.15
+ - mpc-0.8.2
+ - mpfr-3.1.5
linux64-mingw32-nsis:
description: "NSIS build for MinGW32 Cross Compile"
treeherder:
kind: build
platform: toolchains/opt
symbol: TMW(mingw32-nsis)
tier: 2
new file mode 100644
--- /dev/null
+++ b/taskcluster/docker/fetch/Dockerfile
@@ -0,0 +1,9 @@
+# %ARG DOCKER_IMAGE_PARENT
+FROM $DOCKER_IMAGE_PARENT
+
+RUN apt-get update && \
+ apt-get install \
+ gnupg
+
+# %include taskcluster/scripts/misc/fetch-content
+ADD topsrcdir/taskcluster/scripts/misc/fetch-content /builds/worker/bin/fetch-content
--- a/taskcluster/docs/attributes.rst
+++ b/taskcluster/docs/attributes.rst
@@ -171,16 +171,21 @@ stub-installer
Signals to the build system that this build is expected to have a stub installer
present, and informs followon tasks to expect it.
repackage_type
==============
This is the type of repackage. Can be ``repackage`` or
``repackage_signing``.
+fetch-artifact
+==============
+
+For fetch jobs, this is the path to the artifact for that fetch operation.
+
toolchain-artifact
==================
For toolchain jobs, this is the path to the artifact for that toolchain.
toolchain-alias
===============
For toolchain jobs, this optionally gives an alias that can be used instead of the
real toolchain job name in the toolchains list for build jobs.
--- a/taskcluster/docs/kinds.rst
+++ b/taskcluster/docs/kinds.rst
@@ -456,16 +456,23 @@ Dummy tasks to consolidate beetmover dep
post-beetmover-checksums-dummy
------------------------------
Dummy tasks to consolidate beetmover-checksums dependencies to avoid taskcluster limits on number of dependencies per task.
post-langpack-dummy
------------------------------
Dummy tasks to consolidate language pack beetmover dependencies to avoid taskcluster limits on number of dependencies per task.
+fetch
+-----
+
+Tasks that obtain something from a remote service and re-expose it as a
+task artifact. These tasks are used to effectively cache and re-host
+remote content so it is reliably and deterministically available.
+
packages
--------
Tasks used to build packages for use in docker images.
diffoscope
----------
Tasks used to compare pairs of Firefox builds using https://diffoscope.org/.
As of writing, this is mainly meant to be used in try builds, by editing
--- a/taskcluster/docs/transforms.rst
+++ b/taskcluster/docs/transforms.rst
@@ -141,16 +141,17 @@ following ``run-using`` are available
* ``mach``
* ``mozharness``
* ``mozharness-test``
* ``run-task``
* ``spidermonkey`` or ``spidermonkey-package`` or ``spidermonkey-mozjs-crate`` or ``spidermonkey-rust-bindings``
* ``debian-package``
* ``toolchain-script``
* ``always-optimized``
+ * ``fetch-url``
Task Descriptions
-----------------
Every kind needs to create tasks, and all of those tasks have some things in
common. They all run on one of a small set of worker implementations, each
with their own idiosyncracies. And they all report to TreeHerder in a similar
--- a/taskcluster/scripts/misc/build-gcc-4.9-linux.sh
+++ b/taskcluster/scripts/misc/build-gcc-4.9-linux.sh
@@ -12,37 +12,25 @@ data_dir=$HOME_DIR/src/build/unix/build-
. $data_dir/build-gcc.sh
gcc_version=4.9.4
gcc_ext=bz2
binutils_version=2.25.1
binutils_ext=bz2
-# GPG key used to sign GCC
-$GPG --import $data_dir/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
-# GPG key used to sign binutils
-$GPG --import $data_dir/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
-# GPG key used to sign GMP
-$GPG --import $data_dir/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
-# GPG key used to sign MPFR
-$GPG --import $data_dir/07F3DBBECC1A39605078094D980C197698C3739D.key
-# GPG key used to sign MPC
-$GPG --import $data_dir/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
+$HOME_DIR/src/taskcluster/scripts/misc/fetch-content task-artifacts -d $root_dir $MOZ_FETCHES
-cat > $HOME_DIR/checksums <<EOF
-b5b14added7d78a8d1ca70b5cb75fef57ce2197264f4f5835326b0df22ac9f22 binutils-2.25.1.tar.bz2
-02500a4edd14875f94fe84cbeda4290425cb0c1c2474c6f75d75a303d64b4196 cloog-0.18.1.tar.gz
-6c11d292cd01b294f9f84c9a59c230d80e9e4a47e5c6355f046bb36d4f358092 gcc-4.9.4.tar.bz2
-752079520b4690531171d0f4532e40f08600215feefede70b24fabdc6f1ab160 gmp-5.1.3.tar.bz2
-f4b3dbee9712850006e44f0db2103441ab3d13b406f77996d1df19ee89d11fb4 isl-0.12.2.tar.bz2
-ae79f8d41d8a86456b68607e9ca398d00f8b7342d1d83bcf4428178ac45380c7 mpc-0.8.2.tar.gz
-ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658 mpfr-3.1.5.tar.bz2
-EOF
+pushd $root_dir/gcc-$gcc_version
+ln -sf ../cloog-0.18.1 cloog
+ln -sf ../gmp-5.1.3 gmp
+ln -sf ../mpc-0.8.2 mpc
+ln -sf ../isl-0.12.2 isl
+ln -sf ../mpfr-3.1.5 mpfr
+popd
-prepare
apply_patch $data_dir/PR64905.patch
build_binutils
build_gcc
# Put a tarball in the artifacts dir
mkdir -p $UPLOAD_DIR
cp $HOME_DIR/gcc.tar.* $UPLOAD_DIR
--- a/taskcluster/scripts/misc/build-gcc-6-linux.sh
+++ b/taskcluster/scripts/misc/build-gcc-6-linux.sh
@@ -12,35 +12,23 @@ data_dir=$HOME_DIR/src/build/unix/build-
. $data_dir/build-gcc.sh
gcc_version=6.4.0
gcc_ext=xz
binutils_version=2.28.1
binutils_ext=xz
-# GPG key used to sign GCC
-$GPG --import $data_dir/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
-# GPG key used to sign binutils
-$GPG --import $data_dir/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
-# GPG key used to sign GMP
-$GPG --import $data_dir/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
-# GPG key used to sign MPFR
-$GPG --import $data_dir/07F3DBBECC1A39605078094D980C197698C3739D.key
-# GPG key used to sign MPC
-$GPG --import $data_dir/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
+$HOME_DIR/src/taskcluster/scripts/misc/fetch-content task-artifacts -d $root_dir $MOZ_FETCHES
-cat > $HOME_DIR/checksums <<EOF
-16328a906e55a3c633854beec8e9e255a639b366436470b4f6245eb0d2fde942 binutils-2.28.1.tar.xz
-850bf21eafdfe5cd5f6827148184c08c4a0852a37ccf36ce69855334d2c914d4 gcc-6.4.0.tar.xz
-752079520b4690531171d0f4532e40f08600215feefede70b24fabdc6f1ab160 gmp-5.1.3.tar.bz2
-8ceebbf4d9a81afa2b4449113cee4b7cb14a687d7a549a963deb5e2a41458b6b isl-0.15.tar.bz2
-ae79f8d41d8a86456b68607e9ca398d00f8b7342d1d83bcf4428178ac45380c7 mpc-0.8.2.tar.gz
-ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658 mpfr-3.1.5.tar.bz2
-EOF
+pushd $root_dir/gcc-$gcc_version
+ln -sf ../gmp-5.1.3 gmp
+ln -sf ../isl-0.15 isl
+ln -sf ../mpc-0.8.2 mpc
+ln -sf ../mpfr-3.1.5 mpfr
+popd
-prepare
build_binutils
build_gcc
# Put a tarball in the artifacts dir
mkdir -p $UPLOAD_DIR
cp $HOME_DIR/gcc.tar.* $UPLOAD_DIR
--- a/taskcluster/scripts/misc/build-gcc-7-linux.sh
+++ b/taskcluster/scripts/misc/build-gcc-7-linux.sh
@@ -12,35 +12,23 @@ data_dir=$HOME_DIR/src/build/unix/build-
. $data_dir/build-gcc.sh
gcc_version=7.3.0
gcc_ext=xz
binutils_version=2.28.1
binutils_ext=xz
-# GPG key used to sign GCC
-$GPG --import $data_dir/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
-# GPG key used to sign binutils
-$GPG --import $data_dir/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
-# GPG key used to sign GMP
-$GPG --import $data_dir/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
-# GPG key used to sign MPFR
-$GPG --import $data_dir/07F3DBBECC1A39605078094D980C197698C3739D.key
-# GPG key used to sign MPC
-$GPG --import $data_dir/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
+$HOME_DIR/src/taskcluster/scripts/misc/fetch-content task-artifacts -d $root_dir $MOZ_FETCHES
-cat > $HOME_DIR/checksums <<EOF
-16328a906e55a3c633854beec8e9e255a639b366436470b4f6245eb0d2fde942 binutils-2.28.1.tar.xz
-832ca6ae04636adbb430e865a1451adf6979ab44ca1c8374f61fba65645ce15c gcc-7.3.0.tar.xz
-498449a994efeba527885c10405993427995d3f86b8768d8cdf8d9dd7c6b73e8 gmp-6.1.0.tar.bz2
-412538bb65c799ac98e17e8cfcdacbb257a57362acfaaff254b0fcae970126d2 isl-0.16.1.tar.bz2
-617decc6ea09889fb08ede330917a00b16809b8db88c29c31bfbb49cbf88ecc3 mpc-1.0.3.tar.gz
-d3103a80cdad2407ed581f3618c4bed04e0c92d1cf771a65ead662cc397f7775 mpfr-3.1.4.tar.bz2
-EOF
+pushd $root_dir/gcc-$gcc_version
+ln -sf ../gmp-6.1.0 gmp
+ln -sf ../isl-0.16.1 isl
+ln -sf ../mpc-1.0.3 mpc
+ln -sf ../mpfr-3.1.4 mpfr
+popd
-prepare
build_binutils
build_gcc
# Put a tarball in the artifacts dir
mkdir -p $UPLOAD_DIR
cp $HOME_DIR/gcc.tar.* $UPLOAD_DIR
--- a/taskcluster/scripts/misc/build-gcc-mingw32.sh
+++ b/taskcluster/scripts/misc/build-gcc-mingw32.sh
@@ -15,37 +15,24 @@ data_dir=$HOME_DIR/src/build/unix/build-
gcc_version=6.4.0
gcc_ext=xz
binutils_version=2.27
binutils_ext=bz2
binutils_configure_flags="--target=i686-w64-mingw32"
mingw_version=bcf1f29d6dc80b6025b416bef104d2314fa9be57
-# GPG keys used to sign GCC (collected from 5.1.0, 5.4.0, 6.4.0)
-$GPG --import $data_dir/33C235A34C46AA3FFB293709A328C3A2C3C45C06.key
-$GPG --import $data_dir/7F74F97C103468EE5D750B583AB00996FC26A641.key
-# GPG key used to sign binutils
-$GPG --import $data_dir/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
-# GPG key used to sign GMP
-$GPG --import $data_dir/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
-# GPG key used to sign MPFR
-$GPG --import $data_dir/07F3DBBECC1A39605078094D980C197698C3739D.key
-# GPG key used to sign MPC
-$GPG --import $data_dir/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
+$HOME_DIR/src/taskcluster/scripts/misc/fetch-content task-artifacts -d $root_dir $MOZ_FETCHES
-cat > $root_dir/checksums <<EOF
-369737ce51587f92466041a97ab7d2358c6d9e1b6490b3940eb09fb0a9a6ac88 binutils-2.27.tar.bz2
-850bf21eafdfe5cd5f6827148184c08c4a0852a37ccf36ce69855334d2c914d4 gcc-6.4.0.tar.xz
-752079520b4690531171d0f4532e40f08600215feefede70b24fabdc6f1ab160 gmp-5.1.3.tar.bz2
-8ceebbf4d9a81afa2b4449113cee4b7cb14a687d7a549a963deb5e2a41458b6b isl-0.15.tar.bz2
-ae79f8d41d8a86456b68607e9ca398d00f8b7342d1d83bcf4428178ac45380c7 mpc-0.8.2.tar.gz
-ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658 mpfr-3.1.5.tar.bz2
-EOF
+pushd $root_dir/gcc-$gcc_version
+ln -sf ../gmp-5.1.3 gmp
+ln -sf ../isl-0.15 isl
+ln -sf ../mpc-0.8.2 mpc
+ln -sf ../mpfr-3.1.5 mpfr
+popd
-prepare
prepare_mingw
build_binutils
build_gcc_and_mingw
# Put a tarball in the artifacts dir
mkdir -p $UPLOAD_DIR
cp $root_dir/mingw32.tar.* $UPLOAD_DIR
--- a/taskcluster/scripts/misc/build-gcc-sixgill-plugin-linux.sh
+++ b/taskcluster/scripts/misc/build-gcc-sixgill-plugin-linux.sh
@@ -22,38 +22,25 @@ gcc_version=6.4.0
gcc_ext=xz
binutils_version=2.28.1
binutils_ext=xz
sixgill_rev=ab06fc42cf0f
sixgill_repo=https://hg.mozilla.org/users/sfink_mozilla.com/sixgill
. $data_dir/build-gcc.sh
-# GPG key used to sign GCC
-$GPG --import $data_dir/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
-# GPG key used to sign binutils
-$GPG --import $data_dir/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
-# GPG key used to sign GMP
-$GPG --import $data_dir/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
-# GPG key used to sign MPFR
-$GPG --import $data_dir/07F3DBBECC1A39605078094D980C197698C3739D.key
-# GPG key used to sign MPC
-$GPG --import $data_dir/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
+$HOME_DIR/src/taskcluster/scripts/misc/fetch-content task-artifacts -d $root_dir $MOZ_FETCHES
-cat > $HOME_DIR/checksums <<EOF
-16328a906e55a3c633854beec8e9e255a639b366436470b4f6245eb0d2fde942 binutils-2.28.1.tar.xz
-850bf21eafdfe5cd5f6827148184c08c4a0852a37ccf36ce69855334d2c914d4 gcc-6.4.0.tar.xz
-752079520b4690531171d0f4532e40f08600215feefede70b24fabdc6f1ab160 gmp-5.1.3.tar.bz2
-8ceebbf4d9a81afa2b4449113cee4b7cb14a687d7a549a963deb5e2a41458b6b isl-0.15.tar.bz2
-ae79f8d41d8a86456b68607e9ca398d00f8b7342d1d83bcf4428178ac45380c7 mpc-0.8.2.tar.gz
-ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658 mpfr-3.1.5.tar.bz2
-EOF
-
-# Download GCC + related, and unpack.
-prepare
+pushd $root_dir/gcc-$gcc_version
+ln -sf ../binutils-2.28.1 binutils
+ln -sf ../gmp-5.3.1 gmp
+ln -sf ../isl-0.15 isl
+ln -sf ../mpc-0.8.2 mpc
+ln -sf ../mpfr-3.1.5 mpfr
+popd
export TMPDIR=${TMPDIR:-/tmp/}
export gcc_bindir=$root_dir/src/gcc/bin
export gmp_prefix=/tools/gmp
export gmp_dir=$root_dir$gmp_prefix
prepare_sixgill() {(
cd $root_dir
new file mode 100755
--- /dev/null
+++ b/taskcluster/scripts/misc/fetch-content
@@ -0,0 +1,268 @@
+#!/usr/bin/python3 -u
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+import argparse
+import concurrent.futures
+import hashlib
+import multiprocessing
+import os
+import pathlib
+import re
+import subprocess
+import sys
+import tempfile
+import time
+import urllib.request
+
+
+ARTIFACT_URL = ('https://queue.taskcluster.net/v1/task/{task}/artifacts/'
+ '{artifact}')
+
+CONCURRENCY = multiprocessing.cpu_count()
+
+
+class IntegrityError(Exception):
+ """Represents an integrity error when downloading a URL."""
+
+
+def stream_download(url, sha256=None, size=None):
+ """Download a URL to a generator, optionally with content verification.
+
+ If ``sha256`` or ``size`` are defined, the downloaded URL will be
+ validated against those requirements and ``IntegrityError`` will be
+ raised if expectations do not match.
+
+ Because verification cannot occur until the file is completely downloaded
+ it is recommended for consumers to not do anything meaningful with the
+ data if content verification is being used. To securely handle retrieved
+ content, it should be streamed to a file or memory and only operated
+ on after the generator is exhausted without raising.
+ """
+ print('downloading %s' % url)
+
+ h = hashlib.sha256()
+ length = 0
+
+ t0 = time.time()
+ with urllib.request.urlopen(url) as fh:
+ while True:
+ chunk = fh.read(65536)
+ if not chunk:
+ break
+
+ h.update(chunk)
+ length += len(chunk)
+
+ yield chunk
+
+ duration = time.time() - t0
+ digest = h.hexdigest()
+
+ print('%s resolved to %d bytes with sha256 %s in %.3fs' % (
+ url, length, digest, duration))
+
+ if size:
+ if size == length:
+ print('verified size of %s' % url)
+ else:
+ raise IntegrityError('size mismatch on %s: wanted %d; got %d' % (
+ url, size, length))
+
+ if sha256:
+ if digest == sha256:
+ print('verified sha256 integrity of %s' % url)
+ else:
+ raise IntegrityError('sha256 mismatch on %s: wanted %s; got %s' % (
+ url, sha256, digest))
+
+
+def download_to_path(url, path, sha256=None, size=None):
+ """Download a URL to a filesystem path, possibly with verification."""
+
+ # We download to a temporary file and rename at the end so there's
+ # no chance of the final file being partially written or containing
+ # bad data.
+ try:
+ path.unlink()
+ except FileNotFoundError:
+ pass
+
+ tmp = path.with_name('%s.tmp' % path.name)
+
+ print('downloading %s to %s' % (url, tmp))
+
+ try:
+ with tmp.open('wb') as fh:
+ for chunk in stream_download(url, sha256=sha256, size=size):
+ fh.write(chunk)
+
+ print('renaming to %s' % path)
+ tmp.rename(path)
+ except IntegrityError:
+ tmp.unlink()
+ raise
+
+
+def gpg_verify_path(path: pathlib.Path, public_key_data: bytes,
+ signature_data: bytes):
+ """Verify that a filesystem path verifies using GPG.
+
+ Takes a Path defining a file to verify. ``public_key_data`` contains
+ bytes with GPG public key data. ``signature_data`` contains a signed
+ GPG document to use with ``gpg --verify``.
+ """
+ print('validating GPG signature of %s' % path)
+ print('GPG key data:\n%s' % public_key_data.decode('ascii'))
+
+ with tempfile.TemporaryDirectory() as td:
+ try:
+ # --batch since we're running unattended.
+ gpg_args = ['gpg', '--homedir', td, '--batch']
+
+ print('importing GPG key...')
+ subprocess.run(gpg_args + ['--import'],
+ input=public_key_data,
+ check=True)
+
+ print('verifying GPG signature...')
+ subprocess.run(gpg_args + ['--verify', '-', '%s' % path],
+ input=signature_data,
+ check=True)
+
+ print('GPG signature verified!')
+ finally:
+ # There is a race between the agent self-terminating and
+ # shutil.rmtree() from the temporary directory cleanup that can
+ # lead to exceptions. Kill the agent before cleanup to prevent this.
+ env = dict(os.environ)
+ env['GNUPGHOME'] = td
+ subprocess.run(['gpgconf', '--kill', 'gpg-agent'], env=env)
+
+
+def extract_archive(path, dest_dir):
+ """Extract an archive to a destination directory."""
+
+ if re.search('\.tar\..*$', path.name):
+ args = ['tar', 'xaf', str(path)]
+ elif path.name.endswith('.zip'):
+ args = ['unzip', str(path)]
+ else:
+ print('%s is unknown archive format; ignoring' % path)
+ return False
+
+ print('extracting %s to %s using %r' % (path, dest_dir, args))
+ t0 = time.time()
+ subprocess.check_call(args, cwd=str(dest_dir), bufsize=0)
+ print('%s extracted in %.3fs' % (path, time.time() - t0))
+ return True
+
+
+def fetch_and_extract(url, dest_dir, sha256=None, size=None):
+ """Fetch a URL and extract it to a destination path.
+
+ If the downloaded URL is an archive, it is extracted automatically
+ and the archive is deleted. Otherwise the file remains in place in
+ the destination directory.
+ """
+
+ basename = url.split('/')[-1]
+ dest_path = dest_dir / basename
+
+ download_to_path(url, dest_path, sha256=sha256, size=size)
+
+ if extract_archive(dest_path, dest_dir):
+ print('removing %s' % dest_path)
+ dest_path.unlink()
+
+
+def fetch_urls(urls, dest):
+ """Fetch URLs pairs to a pathlib.Path."""
+ dest.mkdir(parents=True, exist_ok=True)
+
+ with concurrent.futures.ThreadPoolExecutor(CONCURRENCY) as e:
+ fs = []
+
+ for url in urls:
+ fs.append(e.submit(fetch_and_extract, url, dest))
+
+ for f in fs:
+ f.result()
+
+
+def command_static_url(args):
+ gpg_sig_url = args.gpg_sig_url
+ gpg_env_key = args.gpg_key_env
+
+ if bool(gpg_sig_url) != bool(gpg_env_key):
+ print('--gpg-sig-url and --gpg-key-env must both be defined')
+ return 1
+
+ if gpg_sig_url:
+ gpg_signature = b''.join(stream_download(gpg_sig_url))
+ gpg_key = os.environb[gpg_env_key.encode('ascii')]
+
+ dest = pathlib.Path(args.dest)
+ dest.parent.mkdir(parents=True, exist_ok=True)
+
+ try:
+ download_to_path(args.url, dest, sha256=args.sha256, size=args.size)
+
+ if gpg_sig_url:
+ gpg_verify_path(dest, gpg_key, gpg_signature)
+
+ except Exception:
+ try:
+ dest.unlink()
+ except FileNotFoundError:
+ pass
+
+ raise
+
+
+def command_task_artifacts(args):
+ urls = []
+
+ for word in args.artifacts:
+ # Format is name@task
+ artifact, task_id = word.split('@', 1)
+
+ urls.append(ARTIFACT_URL.format(task=task_id,
+ artifact=artifact))
+
+ fetch_urls(urls, pathlib.Path(args.dest))
+
+
+def main():
+ parser = argparse.ArgumentParser()
+ subparsers = parser.add_subparsers(title='sub commands')
+
+ url = subparsers.add_parser('static-url', help='Download a static URL')
+ url.set_defaults(func=command_static_url)
+ url.add_argument('--sha256', required=True,
+ help='SHA-256 of downloaded content')
+ url.add_argument('--size', required=True, type=int,
+ help='Size of downloaded content, in bytes')
+ url.add_argument('--gpg-sig-url',
+ help='URL containing signed GPG document validating '
+ 'URL to fetch')
+ url.add_argument('--gpg-key-env',
+ help='Environment variable containing GPG key to validate')
+ url.add_argument('url', help='URL to fetch')
+ url.add_argument('dest', help='Destination path')
+
+ artifacts = subparsers.add_parser('task-artifacts',
+ help='Fetch task artifacts')
+ artifacts.set_defaults(func=command_task_artifacts)
+ artifacts.add_argument('-d', '--dest', help='Destination directory')
+ artifacts.add_argument('artifacts', nargs='+',
+ help='Artifacts to fetch. Of form path@task_id')
+
+ args = parser.parse_args()
+
+ return args.func(args)
+
+
+if __name__ == '__main__':
+ sys.exit(main())
new file mode 100644
--- /dev/null
+++ b/taskcluster/taskgraph/transforms/job/fetch.py
@@ -0,0 +1,138 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+# Support for running tasks that download remote content and re-export
+# it as task artifacts.
+
+from __future__ import absolute_import, unicode_literals
+
+import os
+
+from voluptuous import (
+ Optional,
+ Required,
+)
+
+import taskgraph
+
+from . import (
+ run_job_using,
+)
+from ...util.cached_tasks import (
+ add_optimization,
+)
+from ...util.schema import (
+ Schema,
+)
+
+
+CACHE_TYPE = 'content.v1'
+
+
+url_schema = Schema({
+ Required('using'): 'fetch-url',
+
+ # Base work directory used to set up the task.
+ Required('workdir'): basestring,
+
+ # The URL to download.
+ Required('url'): basestring,
+
+ # The SHA-256 of the downloaded content.
+ Required('sha256'): basestring,
+
+ # Size of the downloaded entity, in bytes.
+ Required('size'): int,
+
+ # GPG signature verification.
+ Optional('gpg-signature'): {
+ # URL where GPG signature document can be obtained. Can contain the
+ # value ``{url}``, which will be substituted with the value from
+ # ``url``.
+ Required('sig-url'): basestring,
+ # Path to file containing GPG public key(s) used to validate download.
+ Required('key-path'): basestring,
+ },
+
+ # The name to give to the generated artifact.
+ Optional('artifact-name'): basestring,
+
+ # IMPORTANT: when adding anything that changes the behavior of the task,
+ # it is important to update the digest data used to compute cache hits.
+})
+
+
+@run_job_using('docker-worker', 'fetch-url',
+ schema=url_schema)
+def cache_url(config, job, taskdesc):
+ """Configure a task to download a URL and expose it as an artifact."""
+ run = job['run']
+
+ worker = taskdesc['worker']
+ worker['chain-of-trust'] = True
+
+ # Fetch tasks are idempotent and immutable. Have them live for
+ # essentially forever.
+ if config.params['level'] == '3':
+ expires = '1000 years'
+ else:
+ expires = '28 days'
+
+ taskdesc['expires-after'] = expires
+
+ artifact_name = run.get('artifact-name')
+ if not artifact_name:
+ artifact_name = run['url'].split('/')[-1]
+
+ worker.setdefault('artifacts', []).append({
+ 'type': 'directory',
+ 'name': 'public',
+ 'path': '/builds/worker/artifacts',
+ })
+
+ env = worker.setdefault('env', {})
+
+ args = [
+ '/builds/worker/bin/fetch-content', 'static-url',
+ '--sha256', run['sha256'],
+ '--size', '%d' % run['size'],
+ ]
+
+ if 'gpg-signature' in run:
+ sig_url = run['gpg-signature']['sig-url'].format(url=run['url'])
+ key_path = os.path.join(taskgraph.GECKO, run['gpg-signature'][
+ 'key-path'])
+
+ with open(key_path, 'rb') as fh:
+ gpg_key = fh.read()
+
+ env['FETCH_GPG_KEY'] = gpg_key
+ args.extend([
+ '--gpg-sig-url', sig_url,
+ '--gpg-key-env', 'FETCH_GPG_KEY',
+ ])
+
+ args.extend([
+ run['url'], '/builds/worker/artifacts/%s' % artifact_name,
+ ])
+
+ worker['command'] = ['/builds/worker/bin/run-task', '--'] + args
+
+ attributes = taskdesc.setdefault('attributes', {})
+ attributes['fetch-artifact'] = 'public/%s' % artifact_name
+
+ if not taskgraph.fast:
+ cache_name = taskdesc['label'].replace('{}-'.format(config.kind), '', 1)
+
+ # This adds the level to the index path automatically.
+ add_optimization(
+ config,
+ taskdesc,
+ cache_type=CACHE_TYPE,
+ cache_name=cache_name,
+ # We don't include the GPG signature in the digest because it isn't
+ # materially important for caching: GPG signatures are supplemental
+ # trust checking beyond what the shasum already provides.
+ digest_data=[run['sha256'], '%d' % run['size'], artifact_name],
+ )
new file mode 100644
--- /dev/null
+++ b/taskcluster/taskgraph/transforms/use_fetches.py
@@ -0,0 +1,55 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+from __future__ import absolute_import, unicode_literals
+
+from .base import (
+ TransformSequence,
+)
+
+
+transforms = TransformSequence()
+
+
+def get_attribute(dict, key, attributes, attribute_name):
+ '''Get `attribute_name` from the given `attributes` dict, and if there
+ is a corresponding value, set `key` in `dict` to that value.'''
+ value = attributes.get(attribute_name)
+ if value:
+ dict[key] = value
+
+
+@transforms.add
+def use_fetches(config, jobs):
+ artifacts = {}
+
+ for task in config.kind_dependencies_tasks:
+ if task.kind != 'fetch':
+ continue
+
+ name = task.label.replace('%s-' % task.kind, '')
+ get_attribute(artifacts, name, task.attributes, 'fetch-artifact')
+
+ for job in jobs:
+ fetches = job.pop('fetches', [])
+
+ for fetch in fetches:
+ if fetch not in artifacts:
+ raise Exception('Missing fetch job for %s-%s: %s' % (
+ config.kind, job['name'], fetch))
+
+ if not artifacts[fetch].startswith('public/'):
+ raise Exception('non-public artifacts not supported')
+
+ if fetches:
+ job.setdefault('dependencies', {}).update(
+ ('fetch-%s' % f, 'fetch-%s' % f)
+ for f in fetches)
+
+ env = job.setdefault('worker', {}).setdefault('env', {})
+ env['MOZ_FETCHES'] = {'task-reference': ' '.join(
+ '%s@<fetch-%s>' % (artifacts[f], f)
+ for f in fetches)}
+
+ yield job