Bug 1355482 - documentation cleanups for release promotion, r?aki draft
authorNick Thomas <nthomas@mozilla.com>
Wed, 27 Jun 2018 17:42:08 +1200
changeset 811609 d7bd75a905ab1551b34318d3c30c536ad89da1b0
parent 811608 fcd8efdb459b969aa2148ad70bcd1a5c51b0573d
push id114372
push usernthomas@mozilla.com
push dateWed, 27 Jun 2018 22:40:10 +0000
reviewersaki
bugs1355482
milestone63.0a1
Bug 1355482 - documentation cleanups for release promotion, r?aki Various fixes for the docs to make docs better and sphinx happier * remove unused targets which were duplicated in balrog, partials * fix up malformed targets and links * convert docstrings to comments so sphinx ignores example response in utils/partials.py * section underlines should match titles MozReview-Commit-ID: GSYqsocBC4I
taskcluster/docs/balrog.rst
taskcluster/docs/kinds.rst
taskcluster/docs/partials.rst
taskcluster/docs/release-promotion-action.rst
taskcluster/docs/release-promotion.rst
taskcluster/docs/signing.rst
taskcluster/taskgraph/util/partners.py
--- a/taskcluster/docs/balrog.rst
+++ b/taskcluster/docs/balrog.rst
@@ -1,13 +1,11 @@
 Balrog in Release Promotion
 ===========================
 
-.. _overview
-
 Overview
 --------
 Balrog is Mozilla's update server. It is responsible for delivering newer versions of Firefox to existing Firefox installations. If you are not already, it would be useful to be familiar with Balrog's core concepts before continuing with this doc. You can find that information on `Balrog's official documentation`_.
 
 The basic interactions that Release Promotion has with Balrog are as follows:
 
 #. Submit new release metadata to Balrog with a number of ``balrog`` tasks and the ``release-balrog-submit-toplevel`` task.
 #. Update test channels to point at the new release in the ``release-balrog-submit-toplevel`` task.
@@ -34,14 +32,14 @@ After a release has been pushed to cdns,
 
 Schedule Shipping
 -----------------
 When we're ready to ship a release we need to let Balrog know about it by scheduling a change to the appropriate Balrog rule. If ``release_eta`` is set it will be used as the ship date and time. If not, the release will be scheduled for shipping 5 minutes in the future. In either case, signoff will need to be done in Balrog by multiple parties before the release is actually made live.
 
 This step is done by the ``release-balrog-scheduling`` task in the ``ship`` phase.
 
 ``secondary`` tasks
------------------
+-------------------
 You may have noticed ``secondary`` variants of the ``release-balrog-submit-toplevel``, ``release-update-verify``, ``release-final-verify``, and ``release-balrog-scheduling`` tasks. These fulfill the same function as their primary counterparts, but for the "beta" update channel. They are only used when we build Release Candidates.
 
 
 .. _Balrog's official documentation: http://mozilla-balrog.readthedocs.io/en/latest/
-S  _balrogscript workers: https://github.com/mozilla-releng/balrogscript
+.. _balrogscript workers: https://github.com/mozilla-releng/balrogscript
--- a/taskcluster/docs/kinds.rst
+++ b/taskcluster/docs/kinds.rst
@@ -86,17 +86,17 @@ static-analysis
 ---------------
 
 Static analysis builds use the compiler to perform some detailed analysis of
 the source code while building.  The useful output from these tasks are their
 build logs, and while they produce a binary, they do not upload it as an
 artifact.
 
 static-analysis-autotest
----------------
+------------------------
 
 Static analysis autotest utility in order to be sure that there is no regression
 when upgrading utilities that impact static-analysis.
 
 toolchain
 ---------
 
 Toolchain builds create the compiler toolchains used to build Firefox.  These
@@ -198,40 +198,40 @@ of the Beetmover kind above and the repa
 release-beetmover-push-to-release
 ---------------------------------
 
 release-beetmover-push-to-release publishes promoted releases from the
 candidates directory to the release directory. This is part of release
 promotion.
 
 beetmover-source
--------------------
+----------------
 
 Beetmover-source publishes release source. This is part of release promotion.
 
 checksums-signing
 -----------------
 Checksums-signing take as input the checksums file generated by beetmover tasks
 and sign it via the signing scriptworkers. Returns the same file signed and
 additional detached signature.
 
 release-source-checksums-signing
--------------------------------
+--------------------------------
 release-source-checksums-signing take as input the checksums file generated by
 source-related beetmover task and sign it via the signing scriptworkers.
 Returns the same file signed and additional detached signature.
 
 beetmover-checksums
 -------------------
 Beetmover, takes specific artifact checksums and pushes it to a location outside
 of Taskcluster's task artifacts (archive.mozilla.org as one place) and in the
 process determines the final location and "pretty" names it (version product name)
 
 release-beetmover-source-checksums
----------------------------------
+----------------------------------
 Beetmover, takes source specific artifact checksums and pushes it to a location outside
 of Taskcluster's task artifacts (archive.mozilla.org as one place) and in the
 process determines the final location and "pretty" names it (version product name)
 
 google-play-strings
 -------------------
 Download strings to display on Google Play from https://l10n.mozilla-community.org/stores_l10n/.
 Artifact is then used by push-apk.
@@ -270,29 +270,29 @@ release-binary-transparency
 Binary transparency creates a publicly verifiable log of binary shas for downstream
 release auditing. https://wiki.mozilla.org/Security/Binary_Transparency
 
 release-snap-repackage
 ----------------------
 Generate an installer using Ubuntu's Snap format.
 
 release-snap-push
-----------------------
+-----------------
 Pushes Snap repackage on Snap store.
 
 release-notify-push
-----------------------
+-------------------
 Notify when a release has been pushed to CDNs.
 
 release-notify-ship
-----------------------
+-------------------
 Notify when a release has been shipped.
 
 release-secondary-notify-ship
-----------------------
+-----------------------------
 Notify when an RC release has been shipped to the beta channel.
 
 release-notify-promote
 ----------------------
 Notify when a release has been promoted.
 
 release-bouncer-sub
 -------------------
@@ -338,17 +338,17 @@ release-beetmover-signed-langpacks
 ----------------------------------
 Publishes signed langpacks to archive.mozilla.org
 
 release-update-verify
 ---------------------
 Verifies the contents and package of release update MARs.
 
 release-secondary-update-verify
----------------------
+-------------------------------
 Verifies the contents and package of release update MARs.
 
 release-update-verify-config
 ----------------------------
 Creates configs for release-update-verify tasks
 
 release-secondary-update-verify-config
 --------------------------------------
@@ -358,69 +358,69 @@ release-updates-builder
 -----------------------
 Top level Balrog blob submission & patcher/update verify config updates.
 
 release-version-bump
 --------------------
 Bumps to the next version.
 
 release-source
---------------------
+--------------
 Generates source for the release
 
 release-source-signing
---------------------
+----------------------
 Signs source for the release
 
 release-partner-repack
 ----------------------
 Generates customized versions of releases for partners.
 
 release-partner-repack-chunking-dummy
-----------------------
+-------------------------------------
 Chunks the partner repacks by locale.
 
 release-partner-repack-signing
 ------------------------------
 Internal signing of partner repacks.
 
 release-partner-repack-repackage
-------------------------------
+--------------------------------
 Repackaging of partner repacks.
 
 release-partner-repack-repackage-signing
-------------------------------
+----------------------------------------
 External signing of partner repacks.
 
 release-partner-repack-beetmover
-------------------------------
+--------------------------------
 Moves the partner repacks to S3 buckets.
 
 release-early-tagging
 ---------------------
 Utilises treescript to perform tagging that should happen near the start of a release.
 
 release-eme-free-repack
-----------------------
+-----------------------
 Generates customized versions of releases for eme-free repacks.
 
 release-eme-free-repack-signing
-------------------------------
+-------------------------------
 Internal signing of eme-free repacks
 
 release-eme-free-repack-repackage
-------------------------------
+---------------------------------
 Repackaging of eme-free repacks.
 
 release-eme-free-repack-repackage-signing
-------------------------------
+-----------------------------------------
 External signing of eme-free repacks.
 
 release-eme-free-repack-beetmover
-------------------------------
+---------------------------------
 Moves the eme-free repacks to S3 buckets.
 
 release-eme-free-repack-beetmover-checksums
 -------------------------------------------
 Moves the beetmover checksum for eme-free repacks to S3 buckets.
 
 repackage
 ---------
@@ -449,34 +449,33 @@ Partials takes the complete.mar files pr
 updates between previous nightly releases and the new one. Requires a release_history
 in the parameters. See ``mach release-history`` if doing this manually.
 
 partials-signing
 ----------------
 Partials-signing takes the partial updates produced in Partials and signs them.
 
 post-balrog-dummy
---------------------
+-----------------
 Dummy tasks to consolidate balrog dependencies to avoid taskcluster limits on number of dependencies per task.
 
 post-beetmover-dummy
 --------------------
 Dummy tasks to consolidate beetmover dependencies to avoid taskcluster limits on number of dependencies per task.
 
 post-beetmover-checksums-dummy
 ------------------------------
 Dummy tasks to consolidate beetmover-checksums dependencies to avoid taskcluster limits on number of dependencies per task.
 
 post-langpack-dummy
-------------------------------
+-------------------
 Dummy tasks to consolidate language pack beetmover dependencies to avoid taskcluster limits on number of dependencies per task.
 
 fetch
 -----
-
 Tasks that obtain something from a remote service and re-expose it as a
 task artifact. These tasks are used to effectively cache and re-host
 remote content so it is reliably and deterministically available.
 
 packages
 --------
 Tasks used to build packages for use in docker images.
 
--- a/taskcluster/docs/partials.rst
+++ b/taskcluster/docs/partials.rst
@@ -1,27 +1,23 @@
 Partial Update Generation
 =========================
 
-.. _overview
-
 Overview
 --------
 
 Windows, Mac and Linux releases have partial updates, to reduce
 the file size end-users have to download in order to receive new
 versions. These are created using a docker image, some Python,
 ``mbsdiff``, and the tools in ``tools/update-packaging``
 
 The task has been called 'Funsize' for quite some time. This might
 make sense depending on what brands of chocolate bar are available
 near you.
 
-.. _how the task works
-
 How the Task Works
 ------------------
 
 Funsize uses a docker image that's built in-tree, named funsize-update-generator.
 The image contains some Python to examine the task definition and determine
 what needs to be done, but it downloads tools like ``mar`` and ``mbsdiff``
 from either locations specified in the task definition, or default mozilla-central
 locations.
@@ -40,17 +36,17 @@ artifact.
        "dest_mar": "target-60.0b8.partial.mar",
        "locale": "ach",
        "from_mar": "http://archive.mozilla.org/pub/firefox/candidates/60.0b8-candidates/build1/update/linux-i686/ach/firefox-60.0b8.complete.mar",
        "update_number": 2,
        "platform": "linux32",
        "previousVersion": "60.0b8",
        "previousBuildNumber": "1",
        "branch": "mozilla-beta"
-     },
+     }
 
 The 'update number' indicates how many released versions there are between 'to' and the current 'from'.
 For example, if we are building a partial update for the current nightly from the previous one, the update
 number will be 1. For the release before that, it will be 2. This lets us use generic output artifact
 names that we can rename in the later ``beetmover`` tasks.
 
 Inside the task, for each partial it has been told to generate, it will download, unpack and virus
 scan the 'from_mar' and 'to_mar', download the tools, and run ``make_incremental_update.sh`` from
@@ -62,36 +58,32 @@ and this allows us to save a lot of comp
 
 For Releases
 ------------
 
 Partials are made as part of the ``promote`` task group. The previous
 versions used to create the update are specified in ship-it by
 Release Management.
 
-.. _data and metrics
-
 Data About Partials
 -------------------
 
 Some metrics are collected in Datadog_ about partial update tasks.
 The prefix used is ``releng.releases.partials``, so the relevant metric names
 will all start with that.
 
 Some dashboards in Datadog are public, some require a login. If you need
 access, file a bug under 'Cloud Services :: Operations: Metrics/Monitoring'
 
 Some examples of potentially useful metrics:
 
 * ``releng.releases.partials.partial_mar_size`` (tagged with branch, platform and update-number)
 * ``releng.releases.partials.task_duration`` - the time the task took, running partial generation concurrently.
 * ``releng.releases.partials.generate_partial.time`` - the time taken to make one partial update
 
-.. _nightly partials
-
 Nightly Partials
 ----------------
 
 Since nightly releases don't appear in ship-it, the partials to create
 are determined in the decision task. This was controversial, and so here
 are the assumptions and reasons, so that when an alternative solution is
 discovered, we can assess it in context:
 
--- a/taskcluster/docs/release-promotion-action.rst
+++ b/taskcluster/docs/release-promotion-action.rst
@@ -2,25 +2,25 @@ Release Promotion Action
 ========================
 
 The release promotion action is how Releng triggers `release promotion`_
 taskgraphs. The one action covers all release promotion needs: different
 *flavors* allow for us to trigger the different :ref:`release promotion phases`
 for each product. The input schema and release promotion flavors are defined in
 the `release promotion action`_.
 
+.. _snowman model:
+
 The snowman model
 -----------------
 
 The `release promotion action`_ allows us to chain multiple taskgraphs (aka graphs, aka task groups) together.
 Essentially, we're using `optimization`_ logic to replace task labels in the
 current taskgraph with task IDs from the previous taskgraph(s).
 
-.. _snowman model:
-
 This is the ``snowman`` model. If you request the body of
 the snowman and point at the base, we only create the middle section of the snowman.
 If you request the body of the snowman and don't point it at the base, we build the
 first base and body of the snowman from scratch.
 
 For example, let's generate a task ``t2`` that depends on ``t1``. Let's call our new taskgraph ``G``::
 
     G
--- a/taskcluster/docs/release-promotion.rst
+++ b/taskcluster/docs/release-promotion.rst
@@ -11,17 +11,17 @@ increased the end-to-end time for a give
 promotion removes these anti-patterns.
 
 By running our continuous integration tests against our shippable builds, we
 have a higher degree of confidence at release time. By separating the build
 phase tasks (compilation, packaging, and related tests) from the promotion
 phase tasks, we can schedule each phase at their own independent cadence, as
 needed, and the end-to-end time for promotion is reduced significantly.
 
-.. _release promotion phases
+.. _release promotion phases:
 
 Release Promotion Phases
 ------------------------
 
 Currently, we have the ``build``, ``promote``, ``push``, and ``ship`` phases.
 
 The ``build`` phase creates ``shippable builds``. These optimize for correctness
 over speed, and are designed to be of shipping quality, should we decide to
--- a/taskcluster/docs/signing.rst
+++ b/taskcluster/docs/signing.rst
@@ -1,26 +1,22 @@
 Signing
 =======
 
-.. _overview
-
 Overview
 --------
 
 Our `code signing`_ happens in discrete tasks, for both performance reasons
 and to limit which machines have access to the signing servers and keys.
 
 In general, the binary-to-be-signed is generated in one task, and the request
 to sign it is in a second task. We verify the request via the `chain of trust`_,
 sign the binary, then upload the signed binary or original binary + detached
 signature as artifacts.
 
-.. _how the task works
-
 How the Task Works
 ------------------
 
 Scriptworker_ verifies the task definition and the upstream tasks until it
 determines the graph comes from a trusted tree; this is `chain of trust`_
 verification. Part of this verification is downloading and verifying the shas
 of the ``upstreamArtifacts`` in the task payload.
 
@@ -81,17 +77,17 @@ file will also need to be signed. These 
 `release-source-signing` and `partials-signing` sign the release source tarball
 and partial update MARs.
 
 We generate signed checksums at the top of the releases directories, like
 in `60.0`_. To generate these, we have the checksums signing kinds, including
 `release-generate-checksums-signing`, `checksums-signing`, and
 `release-source-checksums-signing`
 
-.. _signing formats
+.. _signing formats:
 
 Signing formats
 ---------------
 
 The known signingscript formats are listed in the fourth column of the
 `signing password files`_.
 
 The formats are specified in the ``upstreamArtifacts`` list-of-dicts. The task
@@ -109,17 +105,17 @@ This includes the ``focus-jar`` format, 
 set of keys for the Focus app.
 
 ``macapp`` signing accepts either a ``dmg`` or ``tar.gz``; it converts ``dmg``
 files to ``tar.gz`` before submitting to the signing server. The signed binary
 is a ``tar.gz``.
 
 ``signcode`` signing takes individual binaries or a zipfile. We sign the
 individual file or internals of the zipfile, skipping any already-signed files
-and a select few blocklisted files (using the `_should_sign_windows`_ function).
+and a select few blocklisted files (using the `should_sign_windows`_ function).
 It returns a signed individual binary or zipfile with signed internals, depending
 on the input. This format includes ``signcode``, ``osslsigncode``,
 ``sha2signcode``, and ``sha2signcodestub``.
 
 ``mar`` signing signs our update files (Mozilla ARchive). ``mar_sha384`` is
 the same, but with a different hashing algorithm.
 
 ``widevine`` and ``widevine_blessed`` are also video-related; see the
@@ -174,17 +170,17 @@ development use. Because it isn't used o
 Mozilla Releng is able to make breaking changes on this pool without affecting
 any other team.
 
 .. _60.0: https://archive.mozilla.org/pub/firefox/releases/60.0/
 .. _addonscript: https://github.com/mozilla-releng/addonscript/
 .. _code signing: https://en.wikipedia.org/wiki/Code_signing
 .. _chain of trust: https://scriptworker.readthedocs.io/en/latest/chain_of_trust.html
 .. _depsigning: https://tools.taskcluster.net/provisioners/scriptworker-prov-v1/worker-types/depsigning
-.. __should_sign_windows: https://github.com/mozilla-releng/signingscript/blob/65cbb99ea53896fda9f4844e050a9695c762d24f/signingscript/sign.py#L369
+.. _should_sign_windows: https://github.com/mozilla-releng/signingscript/blob/65cbb99ea53896fda9f4844e050a9695c762d24f/signingscript/sign.py#L369
 .. _Encrypted Media Extensions: https://hacks.mozilla.org/2014/05/reconciling-mozillas-mission-and-w3c-eme/
 .. _signing password files: https://github.com/mozilla/build-puppet/tree/feff5e12ab70f2c060b29940464e77208c7f0ef2/modules/signing_scriptworker/templates
 .. _signingscript: https://github.com/mozilla-releng/signingscript/
 .. _signing-linux-dev: https://tools.taskcluster.net/provisioners/scriptworker-prov-v1/worker-types/signing-linux-dev
 .. _signing-linux-v1: https://tools.taskcluster.net/provisioners/scriptworker-prov-v1/worker-types/signing-linux-v1
 .. _signtool: https://github.com/mozilla-releng/signtool
 .. _Scriptworker: https://github.com/mozilla-releng/scriptworker/
 .. _widevine site: https://www.widevine.com/wv_drm.html
--- a/taskcluster/taskgraph/util/partners.py
+++ b/taskcluster/taskgraph/util/partners.py
@@ -49,36 +49,33 @@ MANIFEST_QUERY = """query {
     object(expression: "master:default.xml") {
       ... on Blob {
         text
       }
     }
   }
 }
 """
-
-r"""
-Example response:
-{
-  "data": {
-    "repository": {
-      "object": {
-        "text": "<?xml version=\"1.0\" ?>\n<manifest>\n  " +
-        "<remote fetch=\"git@github.com:mozilla-partners/\" name=\"mozilla-partners\"/>\n  " +
-        "<remote fetch=\"git@github.com:mozilla/\" name=\"mozilla\"/>\n\n  " +
-        "<project name=\"repack-scripts\" path=\"scripts\" remote=\"mozilla-partners\" " +
-        "revision=\"master\"/>\n  <project name=\"build-tools\" path=\"scripts/tools\" " +
-        "remote=\"mozilla\" revision=\"master\"/>\n  <project name=\"mozilla-EME-free\" " +
-        "path=\"partners/mozilla-EME-free\" remote=\"mozilla-partners\" " +
-        "revision=\"master\"/>\n</manifest>\n"
-      }
-    }
-  }
-}
-"""
+# Example response:
+# {
+#   "data": {
+#     "repository": {
+#       "object": {
+#         "text": "<?xml version=\"1.0\" ?>\n<manifest>\n  " +
+#           "<remote fetch=\"git@github.com:mozilla-partners/\" name=\"mozilla-partners\"/>\n  " +
+#           "<remote fetch=\"git@github.com:mozilla/\" name=\"mozilla\"/>\n\n  " +
+#           "<project name=\"repack-scripts\" path=\"scripts\" remote=\"mozilla-partners\" " +
+#           "revision=\"master\"/>\n  <project name=\"build-tools\" path=\"scripts/tools\" " +
+#           "remote=\"mozilla\" revision=\"master\"/>\n  <project name=\"mozilla-EME-free\" " +
+#           "path=\"partners/mozilla-EME-free\" remote=\"mozilla-partners\" " +
+#           "revision=\"master\"/>\n</manifest>\n"
+#       }
+#     }
+#   }
+# }
 
 # Returns the contents of desktop/*/repack.cfg for a partner repository
 REPACK_CFG_QUERY = """query{
   repository(owner:"%(owner)s", name:"%(repo)s") {
     object(expression: "master:desktop/"){
       ... on Tree {
         entries {
           name
@@ -95,51 +92,49 @@ REPACK_CFG_QUERY = """query{
             }
           }
         }
       }
     }
   }
 }
 """
-r"""
-Example response:
-{
-  "data": {
-    "repository": {
-      "object": {
-        "entries": [
-          {
-            "name": "mozilla-EME-free",
-            "object": {
-              "entries": [
-                {
-                  "name": "distribution",
-                  "object": {}
-                },
-                {
-                  "name": "repack.cfg",
-                  "object": {
-                    "text": "aus=\"mozilla-EMEfree\"\ndist_id=\"mozilla-EMEfree\"\n" +
-                            "dist_version=\"1.0\"\nlinux-i686=true\nlinux-x86_64=true\n" +
-                            " locales=\"ach af de en-US\"\nmac=true\nwin32=true\nwin64=true\n" +
-                            "output_dir=\"%(platform)s-EME-free/%(locale)s\"\n\n" +
-                            "# Upload params\nbucket=\"net-mozaws-prod-delivery-firefox\"\n" +
-                            "upload_to_candidates=true\n"
-                  }
-                }
-              ]
-            }
-          }
-        ]
-      }
-    }
-  }
-}
-"""
+# Example response:
+# {
+#   "data": {
+#     "repository": {
+#       "object": {
+#         "entries": [
+#           {
+#             "name": "mozilla-EME-free",
+#             "object": {
+#               "entries": [
+#                 {
+#                   "name": "distribution",
+#                   "object": {}
+#                 },
+#                 {
+#                   "name": "repack.cfg",
+#                   "object": {
+#                     "text": "aus=\"mozilla-EMEfree\"\ndist_id=\"mozilla-EMEfree\"\n" +
+#                             "dist_version=\"1.0\"\nlinux-i686=true\nlinux-x86_64=true\n" +
+#                             " locales=\"ach af de en-US\"\nmac=true\nwin32=true\nwin64=true\n" +
+#                             "output_dir=\"%(platform)s-EME-free/%(locale)s\"\n\n" +
+#                             "# Upload params\nbucket=\"net-mozaws-prod-delivery-firefox\"\n" +
+#                             "upload_to_candidates=true\n"
+#                   }
+#                 }
+#               ]
+#             }
+#           }
+#         ]
+#       }
+#     }
+#   }
+# }
 
 # Map platforms in repack.cfg into their equivalents in taskcluster
 TC_PLATFORM_PER_FTP = {
     'linux-i686': 'linux-nightly',
     'linux-x86_64': 'linux64-nightly',
     'mac': 'macosx64-nightly',
     'win32': 'win32-nightly',
     'win64': 'win64-nightly',