Bug 1471767 - taskcluster documentation fixes, r?dustin draft
authorNick Thomas <nthomas@mozilla.com>
Wed, 27 Jun 2018 21:48:10 +1200
changeset 811685 f48a897e049a5d156675f58ac7e760dbf35c67b4
parent 811684 347196a3c2c004364d07bd1d6a8e1a94dbcb3fb2
push id114387
push usernthomas@mozilla.com
push dateThu, 28 Jun 2018 00:15:32 +0000
reviewersdustin
bugs1471767
milestone63.0a1
Bug 1471767 - taskcluster documentation fixes, r?dustin Assorted fixes from trawling the sphinx logs - malformed formatting, broken references, leftovers from renaming action-task to action-callback and removing yaml-templates, docstring fixes to make sphinx happier, and typos. MozReview-Commit-ID: 6jUOljdLoE2
taskcluster/docs/caches.rst
taskcluster/docs/docker-images.rst
taskcluster/docs/how-tos.rst
taskcluster/docs/kinds.rst
taskcluster/docs/loading.rst
taskcluster/docs/mach.rst
taskcluster/docs/optimization-schedules.rst
taskcluster/docs/taskcluster-config.rst
taskcluster/docs/taskgraph.rst
taskcluster/docs/transforms.rst
taskcluster/taskgraph/transforms/job/__init__.py
taskcluster/taskgraph/transforms/upload_generated_sources.py
taskcluster/taskgraph/transforms/upload_symbols.py
taskcluster/taskgraph/util/schema.py
--- a/taskcluster/docs/caches.rst
+++ b/taskcluster/docs/caches.rst
@@ -82,17 +82,17 @@ Workspace Caches
 ``level-{{level}}-*-workspace``
    These caches (of various names typically ending with ``workspace``)
    contain state to be shared between task invocations. Use cases are
    dependent on the task.
 
 Other
 :::::
 
-``level-{{level}}-tooltool-cache-{{hash}}
+``level-{{level}}-tooltool-cache-{{hash}}``
    Tooltool invocations should use this cache. Tooltool will store files here
    indexed by their hash.
 
    This cache name pattern is reserved for use with ``run-task`` and must only
    be used by ``run-task``
 
 ``tooltool-cache`` (deprecated)
    Legacy location for tooltool files. Use the per-level one instead.
--- a/taskcluster/docs/docker-images.rst
+++ b/taskcluster/docs/docker-images.rst
@@ -22,61 +22,66 @@ Dockerfile in each folder.
 
 Images could either be an image intended for pushing to a docker registry, or
 one that is meant either for local testing or being built as an artifact when
 pushed to vcs.
 
 Task Images (build-on-push)
 :::::::::::::::::::::::::::
 
-Images can be uploaded as a task artifact, [indexed](#task-image-index-namespace) under
+Images can be uploaded as a task artifact, :ref:`indexed <task-image-index-namespace>` under
 a given namespace, and used in other tasks by referencing the task ID.
 
 Important to note, these images do not require building and pushing to a docker registry, and are
 built per push (if necessary) and uploaded as task artifacts.
 
-The decision task that is run per push will [determine](#context-directory-hashing)
+The decision task that is run per push will :ref:`determine <context-directory-hashing>`
 if the image needs to be built based on the hash of the context directory and if the image
 exists under the namespace for a given branch.
 
 As an additional convenience, and a precaution to loading images per branch, if an image
 has been indexed with a given context hash for mozilla-central, any tasks requiring that image
 will use that indexed task.  This is to ensure there are not multiple images built/used
 that were built from the same context. In summary, if the image has been built for mozilla-central,
 pushes to any branch will use that already built image.
 
 To use within an in-tree task definition, the format is:
 
-```yaml
-image:
-  type: 'task-image'
-  path: 'public/image.tar.zst'
-  taskId: '{{#task_id_for_image}}builder{{/task_id_for_image}}'
-```
+.. code-block:: yaml
+
+    image:
+        type: 'task-image'
+        path: 'public/image.tar.zst'
+        taskId: '<task_id_for_image_builder>'
+
+.. _context-directory-hashing:
 
 Context Directory Hashing
 .........................
 
 Decision tasks will calculate the sha256 hash of the contents of the image
 directory and will determine if the image already exists for a given branch and hash
 or if a new image must be built and indexed.
 
 Note: this is the contents of *only* the context directory, not the
 image contents.
 
 The decision task will:
+
 1. Recursively collect the paths of all files within the context directory
 2. Sort the filenames alphabetically to ensure the hash is consistently calculated
-3. Generate a sha256 hash of the contents of each file.
-4. All file hashes will then be combined with their path and used to update the hash
-of the context directory.
+3. Generate a sha256 hash of the contents of each file
+4. All file hashes will then be combined with their path and used to update the
+   hash of the context directory
 
 This ensures that the hash is consistently calculated and path changes will result
 in different hashes being generated.
 
+.. _task-image-index-namespace:
+
 Task Image Index Namespace
 ..........................
 
 Images that are built on push and uploaded as an artifact of a task will be indexed under the
 following namespaces.
 
 * gecko.cache.level-{level}.docker.v2.{name}.hash.{digest}
 * gecko.cache.level-{level}.docker.v2.{name}.latest
@@ -100,17 +105,17 @@ Example:
 
 .. code-block:: none
 
     image: taskcluster/decision:0.1.10@sha256:c5451ee6c655b3d97d4baa3b0e29a5115f23e0991d4f7f36d2a8f793076d6854
 
 Each image has a repo digest, an image hash, and a version. The repo digest is
 stored in the ``HASH`` file in the image directory  and used to refer to the
 image as above.  The version is in ``VERSION``.  The image hash is used in
-`chain-of-trust verification <http://scriptworker.readthedocs.io/en/latest/chain_of_trust.html>`
+`chain-of-trust verification <https://scriptworker.readthedocs.io/en/latest/chain_of_trust.html>`_
 in `scriptworker <https://github.com/mozilla-releng/scriptworker>`_.
 
 The version file only serves to provide convenient names, such that old
 versions are easy to discover in the registry (and ensuring old versions aren't
 deleted by garbage-collection).
 
 Each image directory also has a ``REGISTRY``, defaulting to the ``REGISTRY`` in
 the ``taskcluster/docker`` directory, and specifying the image registry to
@@ -163,17 +168,17 @@ Landing docker registry images takes a l
 
 Once a new version of the image has been built and tested locally, push it to
 the docker registry and make note of the resulting repo digest.  Put this value
 in the ``HASH`` file, and update any references to the image in the code or
 task definitions.
 
 The change is now safe to use in Try pushes.  However, if the image is used in
 building releases then it is *not* safe to land to an integration branch until
-the whitelists in `scriptworker
+the whitelists in `scriptworker's constants.py
 <https://github.com/mozilla-releng/scriptworker/blob/master/scriptworker/constants.py>`_
 have also been updated. These whitelists use the image hash, not the repo
 digest.
 
 Special Dockerfile Syntax
 -------------------------
 
 Dockerfile syntax has been extended to allow *any* file from the
--- a/taskcluster/docs/how-tos.rst
+++ b/taskcluster/docs/how-tos.rst
@@ -52,17 +52,17 @@ 4. When you are satisfied with the chang
 Hacking Actions
 ...............
 
 If you are working on an action task and wish to test it out locally, use the
 ``./mach taskgraph test-action-callback`` command:
 
    .. code-block:: none
 
-        ./mach taskgraph test-action-task \
+        ./mach taskgraph test-action-callback \
             --task-id I4gu9KDmSZWu3KHx6ba6tw --task-group-id sMO4ybV9Qb2tmcI1sDHClQ \
             --input input.yml hello_world_action
 
 This invocation will run the hello world callback with the given inputs and
 print any created tasks to stdout, rather than actually creating them.
 
 Common Changes
 --------------
--- a/taskcluster/docs/kinds.rst
+++ b/taskcluster/docs/kinds.rst
@@ -154,21 +154,17 @@ Tasks of the ``docker-image`` kind build
 Docker tasks run.
 
 The tasks to generate each docker image have predictable labels:
 ``build-docker-image-<name>``.
 
 Docker images are built from subdirectories of ``taskcluster/docker``, using
 ``docker build``.  There is currently no capability for one Docker image to
 depend on another in-tree docker image, without uploading the latter to a
-Docker repository
-
-The task definition used to create the image-building tasks is given in
-``image.yml`` in the kind directory, and is interpreted as a :doc:`YAML
-Template <yaml-templates>`.
+Docker repository.
 
 balrog
 ------
 
 Balrog tasks are responsible for submitting metadata to our update server (Balrog).
 They are typically downstream of a beetmover job that moves signed MARs somewhere
 (eg: beetmover and beetmover-l10n for releases, beetmover-repackage for nightlies).
 
--- a/taskcluster/docs/loading.rst
+++ b/taskcluster/docs/loading.rst
@@ -11,17 +11,17 @@ a Python function like::
 
 The ``kind`` is the name of the kind; the configuration for that kind
 named this class.
 
 The ``path`` is the path to the configuration directory for the kind. This
 can be used to load extra data, templates, etc.
 
 The ``parameters`` give details on which to base the task generation. See
-:ref:`parameters` for details.
+:doc:`parameters` for details.
 
 At the time this method is called, all kinds on which this kind depends
 (that is, specified in the ``kind-dependencies`` key in ``config``)
 have already loaded their tasks, and those tasks are available in
 the list ``loaded_tasks``.
 
 The return value is a list of inputs to the transforms listed in the kind's
 ``transforms`` property. The specific format for the input depends on the first
--- a/taskcluster/docs/mach.rst
+++ b/taskcluster/docs/mach.rst
@@ -17,17 +17,17 @@ graph-generation process and output the 
 
 ``mach taskgraph target-graph``
    Get the target task graph
 
 ``mach taskgraph optimized``
    Get the optimized task graph
 
 ``mach taskgraph morphed``
-   Get the morhped task graph
+   Get the morphed task graph
 
 See :doc:`how-tos` for further practical tips on debugging task-graph mechanics
 locally.
 
 Parameters
 ----------
 
 Each of these commands takes an optional ``--parameters`` argument giving a file
--- a/taskcluster/docs/optimization-schedules.rst
+++ b/taskcluster/docs/optimization-schedules.rst
@@ -57,33 +57,33 @@ Specification
 
 Components are defined as either inclusive or exclusive in :py:mod:`mozbuild.schedules`.
 
 File Annotation
 :::::::::::::::
 
 Files are annotated with their affected components in ``moz.build`` files with stanzas like ::
 
-    
     with Files('**/*.py'):
         SCHEDULES.inclusive += ['py-lint']
 
 for inclusive components and ::
 
     with Files('*gradle*'):
         SCHEDULES.exclusive = ['android']
 
 for exclusive components.
 Note the use of ``+=`` for inclusive compoenents (as this is adding to the existing set of affected components) but ``=`` for exclusive components (as this is resetting the affected set to something smaller).
 For cases where an inclusive component is affected exclusively (such as the python-lint configuration in the example above), that component can be assigned to ``SCHEDULES.exclusive``::
 
     with Files('**/pep8rc'):
         SCHEDULES.exclusive = ['py-lint']
 
-If multiple stanzas set ``SCHEDULES.exclusive``, the last one will take precedence.  Thus the following will set ``SCHEDULES.exclusive`` to ``hpux`` for all files except those under ``docs/``.
+If multiple stanzas set ``SCHEDULES.exclusive``, the last one will take precedence.  Thus the following
+will set ``SCHEDULES.exclusive`` to ``hpux`` for all files except those under ``docs/``. ::
 
     with Files('**'):
         SCHEDULES.exclusive = ['hpux']
 
     with Files('**/docs'):
         SCHEDULES.exclusive = ['docs']
 
 Task Annotation
--- a/taskcluster/docs/taskcluster-config.rst
+++ b/taskcluster/docs/taskcluster-config.rst
@@ -27,9 +27,9 @@ Part of the landing process is for someo
 
 You can test your patches with something like this, assuming ``.`` is a checkout of the `ci-configuration`_ repository containing your changes:
 
 .. code-block: shell
 
   ci-admin diff --ci-configuration-directory .
 
 .. _ci-configuration: https://hg.mozilla.org/build/ci-configuration/file
-.. _ci-configuration: https://hg.mozilla.org/build/ci-admin/file
+.. _ci-admin: https://hg.mozilla.org/build/ci-admin/file
--- a/taskcluster/docs/taskgraph.rst
+++ b/taskcluster/docs/taskgraph.rst
@@ -92,17 +92,17 @@ Graph generation, as run via ``mach task
    result is the "full task graph".
 #. Filter the target tasks (based on a series of filters, such as try syntax,
    tree-specific specifications, etc). The result is the "target task set".
 #. Based on the full task graph, calculate the transitive closure of the target
    task set.  That is, the target tasks and all requirements of those tasks.
    The result is the "target task graph".
 #. Optimize the target task graph using task-specific optimization methods.
    The result is the "optimized task graph" with fewer nodes than the target
-   task graph.  See :ref:`optimization`.
+   task graph.  See :doc:`optimization`.
 #. Morph the graph. Morphs are like syntactic sugar: they keep the same meaning,
    but express it in a lower-level way. These generally work around limitations
    in the TaskCluster platform, such as number of dependencies or routes in
    a task.
 #. Create tasks for all tasks in the morphed task graph.
 
 Transitive Closure
 ..................
@@ -132,17 +132,17 @@ Action Tasks
 
 Action Tasks are tasks which help you to schedule new jobs via Treeherder's
 "Add New Jobs" feature. The Decision Task creates a YAML file named
 ``action.yml`` which can be used to schedule Action Tasks after suitably replacing
 ``{{decision_task_id}}`` and ``{{task_labels}}``, which correspond to the decision
 task ID of the push and a comma separated list of task labels which need to be
 scheduled.
 
-This task invokes ``mach taskgraph action-task`` which builds up a task graph of
+This task invokes ``mach taskgraph action-callback`` which builds up a task graph of
 the requested tasks. This graph is optimized using the tasks running initially in
 the same push, due to the decision task.
 
 So for instance, if you had already requested a build task in the ``try`` command,
 and you wish to add a test which depends on this build, the original build task
 is re-used.
 
 
@@ -178,26 +178,26 @@ using simple parameterized values, as fo
 
 .. _taskgraph-graph-config:
 
 Graph Configuration
 -------------------
 
 There are several configuration settings that are pertain to the entire
 taskgraph. These are specified in :file:`config.yml` at the root of the
-taskgraph configuration (typically :file:`taskcluster/ci`). The available
+taskgraph configuration (typically :file:`taskcluster/ci/`). The available
 settings are documented inline in `taskcluster/taskgraph/config.py
 <https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/config.py>`_.
 
 .. _taskgraph-trust-domain:
 
 Trust Domain
 ------------
 
 When publishing and signing releases, that tasks verify their definition and
 all upstream tasks come from a decision task based on a trusted tree. (see
-`chain-of-trust verification <http://scriptworker.readthedocs.io/en/latest/chain_of_trust.html>`_).
+`chain-of-trust verification <https://scriptworker.readthedocs.io/en/latest/chain_of_trust.html>`_).
 Firefox and Thunderbird share the taskgraph code and in particular, they have
 separate taskgraph configurations and in particular distinct decision tasks.
 Although they use identical docker images and toolchains, in order to track the
 province of those artifacts when verifying the chain of trust, they use
 different index paths to cache those artifacts. The ``trust-domain`` graph
 configuration controls the base path for indexing these cached artifacts.
--- a/taskcluster/docs/transforms.rst
+++ b/taskcluster/docs/transforms.rst
@@ -18,17 +18,17 @@ items up into smaller items (for example
 transforms rewrite the items entirely, with the final result being a task
 definition.
 
 Transform Functions
 ...................
 
 Each transformation looks like this:
 
-.. code-block::
+.. code-block:: python
 
     @transforms.add
     def transform_an_item(config, items):
         """This transform ..."""  # always a docstring!
         for item in items:
             # ..
             yield item
 
--- a/taskcluster/taskgraph/transforms/job/__init__.py
+++ b/taskcluster/taskgraph/transforms/job/__init__.py
@@ -178,18 +178,18 @@ def make_task_description(config, jobs):
 registry = {}
 
 
 def run_job_using(worker_implementation, run_using, schema=None, defaults={}):
     """Register the decorated function as able to set up a task description for
     jobs with the given worker implementation and `run.using` property.  If
     `schema` is given, the job's run field will be verified to match it.
 
-    The decorated function should have the signature `using_foo(config, job,
-    taskdesc) and should modify the task description in-place.  The skeleton of
+    The decorated function should have the signature `using_foo(config, job, taskdesc)`
+    and should modify the task description in-place.  The skeleton of
     the task description is already set up, but without a payload."""
     def wrap(func):
         for_run_using = registry.setdefault(run_using, {})
         if worker_implementation in for_run_using:
             raise Exception("run_job_using({!r}, {!r}) already exists: {!r}".format(
                 run_using, worker_implementation, for_run_using[run_using]))
         for_run_using[worker_implementation] = (func, schema, defaults)
         return func
--- a/taskcluster/taskgraph/transforms/upload_generated_sources.py
+++ b/taskcluster/taskgraph/transforms/upload_generated_sources.py
@@ -1,15 +1,14 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 """
 Transform the upload-generated-files task description template,
-  taskcluster/ci/upload-generated-sources/kind.yml
-into an actual task description.
+taskcluster/ci/upload-generated-sources/kind.yml, into an actual task description.
 """
 
 from __future__ import absolute_import, print_function, unicode_literals
 
 from taskgraph.transforms.base import TransformSequence
 from taskgraph.util.taskcluster import get_artifact_url
 
 
--- a/taskcluster/taskgraph/transforms/upload_symbols.py
+++ b/taskcluster/taskgraph/transforms/upload_symbols.py
@@ -1,15 +1,14 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 """
 Transform the upload-symbols task description template,
-  taskcluster/ci/upload-symbols/job-template.yml
-into an actual task description.
+taskcluster/ci/upload-symbols/job-template.yml into an actual task description.
 """
 
 from __future__ import absolute_import, print_function, unicode_literals
 
 from taskgraph.transforms.base import TransformSequence
 
 
 transforms = TransformSequence()
--- a/taskcluster/taskgraph/util/schema.py
+++ b/taskcluster/taskgraph/util/schema.py
@@ -70,17 +70,17 @@ def resolve_keyed_by(item, field, item_n
         job:
             test-platform: linux128
             chunks:
                 by-test-platform:
                     macosx-10.11/debug: 13
                     win.*: 6
                     default: 12
 
-    a call to `resolve_keyed_by(item, 'job.chunks', item['thing-name'])
+    a call to `resolve_keyed_by(item, 'job.chunks', item['thing-name'])`
     would mutate item in-place to::
 
         job:
             chunks: 12
 
     The `item_name` parameter is used to generate useful error messages.
 
     If extra_values are supplied, they represent additional values available