Bug 1377875: update taskgraph how-to documentation; r?rforbes
Minor modificatinons based on conversations at the SF all-hands.
MozReview-Commit-ID: AnkwXppUPJd
--- a/taskcluster/docs/how-tos.rst
+++ b/taskcluster/docs/how-tos.rst
@@ -150,66 +150,80 @@ Existing Kind
The simplest option is to add your task to an existing kind. This is most
practical when the task "makes sense" as part of that kind -- for example, if
your task is building an installer for a new platform using mozharness scripts
similar to the existing build tasks, it makes most sense to add your task to
the ``build`` kind. If you need some additional functionality in the kind,
it's OK to modify the implementation as necessary, as long as the modification
is complete and useful to the next developer to come along.
+Tasks in the ``build`` kind generate Firefox installers, and the ``test`` kind
+will add a full set of Firefox tests for each ``build`` task.
+
New Kind
````````
The next option to consider is adding a new kind. A distinct kind gives you
some isolation from other task types, which can be nice if you are adding an
experimental kind of task.
-Kinds can range in complexity. The simplest sort of kind uses the
-``TransformTask`` implementation to read a list of jobs from the ``jobs`` key,
-and applies the standard ``job`` and ``task`` transforms:
+Kinds can range in complexity. The simplest sort of kind uses the transform
+loader to read a list of jobs from the ``jobs`` key, and applies the standard
+``job`` and ``task`` transforms:
.. code-block:: yaml
implementation: taskgraph.task.transform:TransformTask
transforms:
- taskgraph.transforms.job:transforms
- taskgraph.transforms.task:transforms
jobs:
- ..your job description here..
-Custom Kind Implementation
-``````````````````````````
+Job descriptions are defined and documented in
+``taskcluster/taskgraph/transforms/job/__init__.py``.
+
+Custom Kind Loader
+``````````````````
If your task depends on other tasks, then the decision of which tasks to create
-may require some code. For example, the ``upload-symbols`` kind iterates over
-the builds in the graph, generating a task for each one. This specific
-post-build behavior is implemented in the general
-``taskgraph.task.post_build:PostBuildTask`` kind implementation. If your task
-needs something more purpose-specific, then it may be time to write a new kind
-implementation.
+may require some code. For example, the ``test`` kind iterates over
+the builds in the graph, generating a full set of test tasks for each one. This specific
+post-build behavior is implemented as a loader defined in ``taskcluster/taskgraph/loader/test.py``.
+
+A custom loader is useful when the set of tasks you want to create is not
+static but based on something else (such as the available builds) or when the
+dependency relationships for your tasks are complex.
Custom Transforms
`````````````````
-If your task needs to create many tasks from a single description, for example
-to implement chunking, it is time to implement some custom transforms. Ideally
-those transforms will produce job descriptions, so you can use the existing ``job``
-and ``task`` transforms:
+Most loaders apply a series of ":doc:`transforms <transforms>`" that start with
+an initial human-friendly description of a task and end with a task definition
+suitable for insertion into a Taskcluster queue.
+
+Custom transforms can be useful to apply defaults, simplifying the YAML files
+in your kind. They can also apply business logic that is more easily expressed
+in code than in YAML.
+
+Transforms need not be one-to-one: a transform can produce zero or more outputs
+for each input. For example, the test transforms perform chunking by producing
+an output for each chunk of a given input.
+
+Ideally those transforms will produce job descriptions, so you can use the
+existing ``job`` and ``task`` transforms:
.. code-block:: yaml
transforms:
- taskgraph.transforms.my_stuff:transforms
- taskgraph.transforms.job:transforms
- taskgraph.transforms.task:transforms
-Similarly, if you need to include dynamic task defaults -- perhaps some feature
-is only available in level-3 repositories, or on specific projects -- then
-custom transforms are the appropriate tool. Try to keep transforms simple,
-single-purpose and well-documented!
+Try to keep transforms simple, single-purpose and well-documented!
Custom Run-Using
````````````````
If the way your task is executed is unique (so, not a mach command or
mozharness invocation), you can add a new implementation of the job
description's "run" section. Before you do this, consider that it might be a
better investment to modify your task to support invocation via mozharness or