Bug 1355482 - documentation cleanups for release promotion, r?aki
Various fixes for the docs to make docs better and sphinx happier
* remove unused targets which were duplicated in balrog, partials
* fix up malformed targets and links
* convert docstrings to comments so sphinx ignores example response in utils/partials.py
* section underlines should match titles
MozReview-Commit-ID: GSYqsocBC4I
--- a/taskcluster/docs/balrog.rst
+++ b/taskcluster/docs/balrog.rst
@@ -1,13 +1,11 @@
Balrog in Release Promotion
===========================
-.. _overview
-
Overview
--------
Balrog is Mozilla's update server. It is responsible for delivering newer versions of Firefox to existing Firefox installations. If you are not already, it would be useful to be familiar with Balrog's core concepts before continuing with this doc. You can find that information on `Balrog's official documentation`_.
The basic interactions that Release Promotion has with Balrog are as follows:
#. Submit new release metadata to Balrog with a number of ``balrog`` tasks and the ``release-balrog-submit-toplevel`` task.
#. Update test channels to point at the new release in the ``release-balrog-submit-toplevel`` task.
@@ -34,14 +32,14 @@ After a release has been pushed to cdns,
Schedule Shipping
-----------------
When we're ready to ship a release we need to let Balrog know about it by scheduling a change to the appropriate Balrog rule. If ``release_eta`` is set it will be used as the ship date and time. If not, the release will be scheduled for shipping 5 minutes in the future. In either case, signoff will need to be done in Balrog by multiple parties before the release is actually made live.
This step is done by the ``release-balrog-scheduling`` task in the ``ship`` phase.
``secondary`` tasks
------------------
+-------------------
You may have noticed ``secondary`` variants of the ``release-balrog-submit-toplevel``, ``release-update-verify``, ``release-final-verify``, and ``release-balrog-scheduling`` tasks. These fulfill the same function as their primary counterparts, but for the "beta" update channel. They are only used when we build Release Candidates.
.. _Balrog's official documentation: http://mozilla-balrog.readthedocs.io/en/latest/
-S _balrogscript workers: https://github.com/mozilla-releng/balrogscript
+.. _balrogscript workers: https://github.com/mozilla-releng/balrogscript
--- a/taskcluster/docs/kinds.rst
+++ b/taskcluster/docs/kinds.rst
@@ -86,17 +86,17 @@ static-analysis
---------------
Static analysis builds use the compiler to perform some detailed analysis of
the source code while building. The useful output from these tasks are their
build logs, and while they produce a binary, they do not upload it as an
artifact.
static-analysis-autotest
----------------
+------------------------
Static analysis autotest utility in order to be sure that there is no regression
when upgrading utilities that impact static-analysis.
toolchain
---------
Toolchain builds create the compiler toolchains used to build Firefox. These
@@ -198,40 +198,40 @@ of the Beetmover kind above and the repa
release-beetmover-push-to-release
---------------------------------
release-beetmover-push-to-release publishes promoted releases from the
candidates directory to the release directory. This is part of release
promotion.
beetmover-source
--------------------
+----------------
Beetmover-source publishes release source. This is part of release promotion.
checksums-signing
-----------------
Checksums-signing take as input the checksums file generated by beetmover tasks
and sign it via the signing scriptworkers. Returns the same file signed and
additional detached signature.
release-source-checksums-signing
--------------------------------
+--------------------------------
release-source-checksums-signing take as input the checksums file generated by
source-related beetmover task and sign it via the signing scriptworkers.
Returns the same file signed and additional detached signature.
beetmover-checksums
-------------------
Beetmover, takes specific artifact checksums and pushes it to a location outside
of Taskcluster's task artifacts (archive.mozilla.org as one place) and in the
process determines the final location and "pretty" names it (version product name)
release-beetmover-source-checksums
----------------------------------
+----------------------------------
Beetmover, takes source specific artifact checksums and pushes it to a location outside
of Taskcluster's task artifacts (archive.mozilla.org as one place) and in the
process determines the final location and "pretty" names it (version product name)
google-play-strings
-------------------
Download strings to display on Google Play from https://l10n.mozilla-community.org/stores_l10n/.
Artifact is then used by push-apk.
@@ -270,29 +270,29 @@ release-binary-transparency
Binary transparency creates a publicly verifiable log of binary shas for downstream
release auditing. https://wiki.mozilla.org/Security/Binary_Transparency
release-snap-repackage
----------------------
Generate an installer using Ubuntu's Snap format.
release-snap-push
-----------------------
+-----------------
Pushes Snap repackage on Snap store.
release-notify-push
-----------------------
+-------------------
Notify when a release has been pushed to CDNs.
release-notify-ship
-----------------------
+-------------------
Notify when a release has been shipped.
release-secondary-notify-ship
-----------------------
+-----------------------------
Notify when an RC release has been shipped to the beta channel.
release-notify-promote
----------------------
Notify when a release has been promoted.
release-bouncer-sub
-------------------
@@ -338,17 +338,17 @@ release-beetmover-signed-langpacks
----------------------------------
Publishes signed langpacks to archive.mozilla.org
release-update-verify
---------------------
Verifies the contents and package of release update MARs.
release-secondary-update-verify
----------------------
+-------------------------------
Verifies the contents and package of release update MARs.
release-update-verify-config
----------------------------
Creates configs for release-update-verify tasks
release-secondary-update-verify-config
--------------------------------------
@@ -358,69 +358,69 @@ release-updates-builder
-----------------------
Top level Balrog blob submission & patcher/update verify config updates.
release-version-bump
--------------------
Bumps to the next version.
release-source
---------------------
+--------------
Generates source for the release
release-source-signing
---------------------
+----------------------
Signs source for the release
release-partner-repack
----------------------
Generates customized versions of releases for partners.
release-partner-repack-chunking-dummy
-----------------------
+-------------------------------------
Chunks the partner repacks by locale.
release-partner-repack-signing
------------------------------
Internal signing of partner repacks.
release-partner-repack-repackage
-------------------------------
+--------------------------------
Repackaging of partner repacks.
release-partner-repack-repackage-signing
-------------------------------
+----------------------------------------
External signing of partner repacks.
release-partner-repack-beetmover
-------------------------------
+--------------------------------
Moves the partner repacks to S3 buckets.
release-early-tagging
---------------------
Utilises treescript to perform tagging that should happen near the start of a release.
release-eme-free-repack
-----------------------
+-----------------------
Generates customized versions of releases for eme-free repacks.
release-eme-free-repack-signing
-------------------------------
+-------------------------------
Internal signing of eme-free repacks
release-eme-free-repack-repackage
-------------------------------
+---------------------------------
Repackaging of eme-free repacks.
release-eme-free-repack-repackage-signing
-------------------------------
+-----------------------------------------
External signing of eme-free repacks.
release-eme-free-repack-beetmover
-------------------------------
+---------------------------------
Moves the eme-free repacks to S3 buckets.
release-eme-free-repack-beetmover-checksums
-------------------------------------------
Moves the beetmover checksum for eme-free repacks to S3 buckets.
repackage
---------
@@ -449,34 +449,33 @@ Partials takes the complete.mar files pr
updates between previous nightly releases and the new one. Requires a release_history
in the parameters. See ``mach release-history`` if doing this manually.
partials-signing
----------------
Partials-signing takes the partial updates produced in Partials and signs them.
post-balrog-dummy
---------------------
+-----------------
Dummy tasks to consolidate balrog dependencies to avoid taskcluster limits on number of dependencies per task.
post-beetmover-dummy
--------------------
Dummy tasks to consolidate beetmover dependencies to avoid taskcluster limits on number of dependencies per task.
post-beetmover-checksums-dummy
------------------------------
Dummy tasks to consolidate beetmover-checksums dependencies to avoid taskcluster limits on number of dependencies per task.
post-langpack-dummy
-------------------------------
+-------------------
Dummy tasks to consolidate language pack beetmover dependencies to avoid taskcluster limits on number of dependencies per task.
fetch
-----
-
Tasks that obtain something from a remote service and re-expose it as a
task artifact. These tasks are used to effectively cache and re-host
remote content so it is reliably and deterministically available.
packages
--------
Tasks used to build packages for use in docker images.
--- a/taskcluster/docs/partials.rst
+++ b/taskcluster/docs/partials.rst
@@ -1,27 +1,23 @@
Partial Update Generation
=========================
-.. _overview
-
Overview
--------
Windows, Mac and Linux releases have partial updates, to reduce
the file size end-users have to download in order to receive new
versions. These are created using a docker image, some Python,
``mbsdiff``, and the tools in ``tools/update-packaging``
The task has been called 'Funsize' for quite some time. This might
make sense depending on what brands of chocolate bar are available
near you.
-.. _how the task works
-
How the Task Works
------------------
Funsize uses a docker image that's built in-tree, named funsize-update-generator.
The image contains some Python to examine the task definition and determine
what needs to be done, but it downloads tools like ``mar`` and ``mbsdiff``
from either locations specified in the task definition, or default mozilla-central
locations.
@@ -40,17 +36,17 @@ artifact.
"dest_mar": "target-60.0b8.partial.mar",
"locale": "ach",
"from_mar": "http://archive.mozilla.org/pub/firefox/candidates/60.0b8-candidates/build1/update/linux-i686/ach/firefox-60.0b8.complete.mar",
"update_number": 2,
"platform": "linux32",
"previousVersion": "60.0b8",
"previousBuildNumber": "1",
"branch": "mozilla-beta"
- },
+ }
The 'update number' indicates how many released versions there are between 'to' and the current 'from'.
For example, if we are building a partial update for the current nightly from the previous one, the update
number will be 1. For the release before that, it will be 2. This lets us use generic output artifact
names that we can rename in the later ``beetmover`` tasks.
Inside the task, for each partial it has been told to generate, it will download, unpack and virus
scan the 'from_mar' and 'to_mar', download the tools, and run ``make_incremental_update.sh`` from
@@ -62,36 +58,32 @@ and this allows us to save a lot of comp
For Releases
------------
Partials are made as part of the ``promote`` task group. The previous
versions used to create the update are specified in ship-it by
Release Management.
-.. _data and metrics
-
Data About Partials
-------------------
Some metrics are collected in Datadog_ about partial update tasks.
The prefix used is ``releng.releases.partials``, so the relevant metric names
will all start with that.
Some dashboards in Datadog are public, some require a login. If you need
access, file a bug under 'Cloud Services :: Operations: Metrics/Monitoring'
Some examples of potentially useful metrics:
* ``releng.releases.partials.partial_mar_size`` (tagged with branch, platform and update-number)
* ``releng.releases.partials.task_duration`` - the time the task took, running partial generation concurrently.
* ``releng.releases.partials.generate_partial.time`` - the time taken to make one partial update
-.. _nightly partials
-
Nightly Partials
----------------
Since nightly releases don't appear in ship-it, the partials to create
are determined in the decision task. This was controversial, and so here
are the assumptions and reasons, so that when an alternative solution is
discovered, we can assess it in context:
--- a/taskcluster/docs/release-promotion-action.rst
+++ b/taskcluster/docs/release-promotion-action.rst
@@ -2,25 +2,25 @@ Release Promotion Action
========================
The release promotion action is how Releng triggers `release promotion`_
taskgraphs. The one action covers all release promotion needs: different
*flavors* allow for us to trigger the different :ref:`release promotion phases`
for each product. The input schema and release promotion flavors are defined in
the `release promotion action`_.
+.. _snowman model:
+
The snowman model
-----------------
The `release promotion action`_ allows us to chain multiple taskgraphs (aka graphs, aka task groups) together.
Essentially, we're using `optimization`_ logic to replace task labels in the
current taskgraph with task IDs from the previous taskgraph(s).
-.. _snowman model:
-
This is the ``snowman`` model. If you request the body of
the snowman and point at the base, we only create the middle section of the snowman.
If you request the body of the snowman and don't point it at the base, we build the
first base and body of the snowman from scratch.
For example, let's generate a task ``t2`` that depends on ``t1``. Let's call our new taskgraph ``G``::
G
--- a/taskcluster/docs/release-promotion.rst
+++ b/taskcluster/docs/release-promotion.rst
@@ -11,17 +11,17 @@ increased the end-to-end time for a give
promotion removes these anti-patterns.
By running our continuous integration tests against our shippable builds, we
have a higher degree of confidence at release time. By separating the build
phase tasks (compilation, packaging, and related tests) from the promotion
phase tasks, we can schedule each phase at their own independent cadence, as
needed, and the end-to-end time for promotion is reduced significantly.
-.. _release promotion phases
+.. _release promotion phases:
Release Promotion Phases
------------------------
Currently, we have the ``build``, ``promote``, ``push``, and ``ship`` phases.
The ``build`` phase creates ``shippable builds``. These optimize for correctness
over speed, and are designed to be of shipping quality, should we decide to
--- a/taskcluster/docs/signing.rst
+++ b/taskcluster/docs/signing.rst
@@ -1,26 +1,22 @@
Signing
=======
-.. _overview
-
Overview
--------
Our `code signing`_ happens in discrete tasks, for both performance reasons
and to limit which machines have access to the signing servers and keys.
In general, the binary-to-be-signed is generated in one task, and the request
to sign it is in a second task. We verify the request via the `chain of trust`_,
sign the binary, then upload the signed binary or original binary + detached
signature as artifacts.
-.. _how the task works
-
How the Task Works
------------------
Scriptworker_ verifies the task definition and the upstream tasks until it
determines the graph comes from a trusted tree; this is `chain of trust`_
verification. Part of this verification is downloading and verifying the shas
of the ``upstreamArtifacts`` in the task payload.
@@ -81,17 +77,17 @@ file will also need to be signed. These
`release-source-signing` and `partials-signing` sign the release source tarball
and partial update MARs.
We generate signed checksums at the top of the releases directories, like
in `60.0`_. To generate these, we have the checksums signing kinds, including
`release-generate-checksums-signing`, `checksums-signing`, and
`release-source-checksums-signing`
-.. _signing formats
+.. _signing formats:
Signing formats
---------------
The known signingscript formats are listed in the fourth column of the
`signing password files`_.
The formats are specified in the ``upstreamArtifacts`` list-of-dicts. The task
@@ -109,17 +105,17 @@ This includes the ``focus-jar`` format,
set of keys for the Focus app.
``macapp`` signing accepts either a ``dmg`` or ``tar.gz``; it converts ``dmg``
files to ``tar.gz`` before submitting to the signing server. The signed binary
is a ``tar.gz``.
``signcode`` signing takes individual binaries or a zipfile. We sign the
individual file or internals of the zipfile, skipping any already-signed files
-and a select few blocklisted files (using the `_should_sign_windows`_ function).
+and a select few blocklisted files (using the `should_sign_windows`_ function).
It returns a signed individual binary or zipfile with signed internals, depending
on the input. This format includes ``signcode``, ``osslsigncode``,
``sha2signcode``, and ``sha2signcodestub``.
``mar`` signing signs our update files (Mozilla ARchive). ``mar_sha384`` is
the same, but with a different hashing algorithm.
``widevine`` and ``widevine_blessed`` are also video-related; see the
@@ -174,17 +170,17 @@ development use. Because it isn't used o
Mozilla Releng is able to make breaking changes on this pool without affecting
any other team.
.. _60.0: https://archive.mozilla.org/pub/firefox/releases/60.0/
.. _addonscript: https://github.com/mozilla-releng/addonscript/
.. _code signing: https://en.wikipedia.org/wiki/Code_signing
.. _chain of trust: https://scriptworker.readthedocs.io/en/latest/chain_of_trust.html
.. _depsigning: https://tools.taskcluster.net/provisioners/scriptworker-prov-v1/worker-types/depsigning
-.. __should_sign_windows: https://github.com/mozilla-releng/signingscript/blob/65cbb99ea53896fda9f4844e050a9695c762d24f/signingscript/sign.py#L369
+.. _should_sign_windows: https://github.com/mozilla-releng/signingscript/blob/65cbb99ea53896fda9f4844e050a9695c762d24f/signingscript/sign.py#L369
.. _Encrypted Media Extensions: https://hacks.mozilla.org/2014/05/reconciling-mozillas-mission-and-w3c-eme/
.. _signing password files: https://github.com/mozilla/build-puppet/tree/feff5e12ab70f2c060b29940464e77208c7f0ef2/modules/signing_scriptworker/templates
.. _signingscript: https://github.com/mozilla-releng/signingscript/
.. _signing-linux-dev: https://tools.taskcluster.net/provisioners/scriptworker-prov-v1/worker-types/signing-linux-dev
.. _signing-linux-v1: https://tools.taskcluster.net/provisioners/scriptworker-prov-v1/worker-types/signing-linux-v1
.. _signtool: https://github.com/mozilla-releng/signtool
.. _Scriptworker: https://github.com/mozilla-releng/scriptworker/
.. _widevine site: https://www.widevine.com/wv_drm.html
--- a/taskcluster/taskgraph/util/partners.py
+++ b/taskcluster/taskgraph/util/partners.py
@@ -49,36 +49,33 @@ MANIFEST_QUERY = """query {
object(expression: "master:default.xml") {
... on Blob {
text
}
}
}
}
"""
-
-r"""
-Example response:
-{
- "data": {
- "repository": {
- "object": {
- "text": "<?xml version=\"1.0\" ?>\n<manifest>\n " +
- "<remote fetch=\"git@github.com:mozilla-partners/\" name=\"mozilla-partners\"/>\n " +
- "<remote fetch=\"git@github.com:mozilla/\" name=\"mozilla\"/>\n\n " +
- "<project name=\"repack-scripts\" path=\"scripts\" remote=\"mozilla-partners\" " +
- "revision=\"master\"/>\n <project name=\"build-tools\" path=\"scripts/tools\" " +
- "remote=\"mozilla\" revision=\"master\"/>\n <project name=\"mozilla-EME-free\" " +
- "path=\"partners/mozilla-EME-free\" remote=\"mozilla-partners\" " +
- "revision=\"master\"/>\n</manifest>\n"
- }
- }
- }
-}
-"""
+# Example response:
+# {
+# "data": {
+# "repository": {
+# "object": {
+# "text": "<?xml version=\"1.0\" ?>\n<manifest>\n " +
+# "<remote fetch=\"git@github.com:mozilla-partners/\" name=\"mozilla-partners\"/>\n " +
+# "<remote fetch=\"git@github.com:mozilla/\" name=\"mozilla\"/>\n\n " +
+# "<project name=\"repack-scripts\" path=\"scripts\" remote=\"mozilla-partners\" " +
+# "revision=\"master\"/>\n <project name=\"build-tools\" path=\"scripts/tools\" " +
+# "remote=\"mozilla\" revision=\"master\"/>\n <project name=\"mozilla-EME-free\" " +
+# "path=\"partners/mozilla-EME-free\" remote=\"mozilla-partners\" " +
+# "revision=\"master\"/>\n</manifest>\n"
+# }
+# }
+# }
+# }
# Returns the contents of desktop/*/repack.cfg for a partner repository
REPACK_CFG_QUERY = """query{
repository(owner:"%(owner)s", name:"%(repo)s") {
object(expression: "master:desktop/"){
... on Tree {
entries {
name
@@ -95,51 +92,49 @@ REPACK_CFG_QUERY = """query{
}
}
}
}
}
}
}
"""
-r"""
-Example response:
-{
- "data": {
- "repository": {
- "object": {
- "entries": [
- {
- "name": "mozilla-EME-free",
- "object": {
- "entries": [
- {
- "name": "distribution",
- "object": {}
- },
- {
- "name": "repack.cfg",
- "object": {
- "text": "aus=\"mozilla-EMEfree\"\ndist_id=\"mozilla-EMEfree\"\n" +
- "dist_version=\"1.0\"\nlinux-i686=true\nlinux-x86_64=true\n" +
- " locales=\"ach af de en-US\"\nmac=true\nwin32=true\nwin64=true\n" +
- "output_dir=\"%(platform)s-EME-free/%(locale)s\"\n\n" +
- "# Upload params\nbucket=\"net-mozaws-prod-delivery-firefox\"\n" +
- "upload_to_candidates=true\n"
- }
- }
- ]
- }
- }
- ]
- }
- }
- }
-}
-"""
+# Example response:
+# {
+# "data": {
+# "repository": {
+# "object": {
+# "entries": [
+# {
+# "name": "mozilla-EME-free",
+# "object": {
+# "entries": [
+# {
+# "name": "distribution",
+# "object": {}
+# },
+# {
+# "name": "repack.cfg",
+# "object": {
+# "text": "aus=\"mozilla-EMEfree\"\ndist_id=\"mozilla-EMEfree\"\n" +
+# "dist_version=\"1.0\"\nlinux-i686=true\nlinux-x86_64=true\n" +
+# " locales=\"ach af de en-US\"\nmac=true\nwin32=true\nwin64=true\n" +
+# "output_dir=\"%(platform)s-EME-free/%(locale)s\"\n\n" +
+# "# Upload params\nbucket=\"net-mozaws-prod-delivery-firefox\"\n" +
+# "upload_to_candidates=true\n"
+# }
+# }
+# ]
+# }
+# }
+# ]
+# }
+# }
+# }
+# }
# Map platforms in repack.cfg into their equivalents in taskcluster
TC_PLATFORM_PER_FTP = {
'linux-i686': 'linux-nightly',
'linux-x86_64': 'linux64-nightly',
'mac': 'macosx64-nightly',
'win32': 'win32-nightly',
'win64': 'win64-nightly',