Bug 1377875: update taskgraph how-to documentation; r?rforbes draft
authorDustin J. Mitchell <dustin@mozilla.com>
Sat, 01 Jul 2017 16:40:39 -0400
changeset 603287 d1f55c6d3977cf32bf881be4f9d9346b7e0a0264
parent 581313 095aca8a91e68b8b4a723dc016becb37b9c5d1ff
child 635869 3af8297501181bb9e38224b03785201b37812635
push id66722
push userdmitchell@mozilla.com
push dateMon, 03 Jul 2017 14:12:56 +0000
reviewersrforbes
bugs1377875
milestone55.0a1
Bug 1377875: update taskgraph how-to documentation; r?rforbes Minor modificatinons based on conversations at the SF all-hands. MozReview-Commit-ID: AnkwXppUPJd
taskcluster/docs/how-tos.rst
--- a/taskcluster/docs/how-tos.rst
+++ b/taskcluster/docs/how-tos.rst
@@ -150,66 +150,80 @@ Existing Kind
 The simplest option is to add your task to an existing kind.  This is most
 practical when the task "makes sense" as part of that kind -- for example, if
 your task is building an installer for a new platform using mozharness scripts
 similar to the existing build tasks, it makes most sense to add your task to
 the ``build`` kind.  If you need some additional functionality in the kind,
 it's OK to modify the implementation as necessary, as long as the modification
 is complete and useful to the next developer to come along.
 
+Tasks in the ``build`` kind generate Firefox installers, and the ``test`` kind
+will add a full set of Firefox tests for each ``build`` task.
+
 New Kind
 ````````
 
 The next option to consider is adding a new kind.  A distinct kind gives you
 some isolation from other task types, which can be nice if you are adding an
 experimental kind of task.
 
-Kinds can range in complexity.  The simplest sort of kind uses the
-``TransformTask`` implementation to read a list of jobs from the ``jobs`` key,
-and applies the standard ``job`` and ``task`` transforms:
+Kinds can range in complexity.  The simplest sort of kind uses the transform
+loader to read a list of jobs from the ``jobs`` key, and applies the standard
+``job`` and ``task`` transforms:
 
 .. code-block:: yaml
 
     implementation: taskgraph.task.transform:TransformTask
     transforms:
        - taskgraph.transforms.job:transforms
        - taskgraph.transforms.task:transforms
     jobs:
        - ..your job description here..
 
-Custom Kind Implementation
-``````````````````````````
+Job descriptions are defined and documented in
+``taskcluster/taskgraph/transforms/job/__init__.py``.
+
+Custom Kind Loader
+``````````````````
 
 If your task depends on other tasks, then the decision of which tasks to create
-may require some code.  For example, the ``upload-symbols`` kind iterates over
-the builds in the graph, generating a task for each one.  This specific
-post-build behavior is implemented in the general
-``taskgraph.task.post_build:PostBuildTask`` kind implementation.  If your task
-needs something more purpose-specific, then it may be time to write a new kind
-implementation.
+may require some code.  For example, the ``test`` kind iterates over
+the builds in the graph, generating a full set of test tasks for each one.  This specific
+post-build behavior is implemented as a loader defined in ``taskcluster/taskgraph/loader/test.py``.
+
+A custom loader is useful when the set of tasks you want to create is not
+static but based on something else (such as the available builds) or when the
+dependency relationships for your tasks are complex.
 
 Custom Transforms
 `````````````````
 
-If your task needs to create many tasks from a single description, for example
-to implement chunking, it is time to implement some custom transforms.  Ideally
-those transforms will produce job descriptions, so you can use the existing ``job``
-and ``task`` transforms:
+Most loaders apply a series of ":doc:`transforms <transforms>`" that start with
+an initial human-friendly description of a task and end with a task definition
+suitable for insertion into a Taskcluster queue.
+
+Custom transforms can be useful to apply defaults, simplifying the YAML files
+in your kind. They can also apply business logic that is more easily expressed
+in code than in YAML.
+
+Transforms need not be one-to-one: a transform can produce zero or more outputs
+for each input. For example, the test transforms perform chunking by producing
+an output for each chunk of a given input.
+
+Ideally those transforms will produce job descriptions, so you can use the
+existing ``job`` and ``task`` transforms:
 
 .. code-block:: yaml
 
     transforms:
        - taskgraph.transforms.my_stuff:transforms
        - taskgraph.transforms.job:transforms
        - taskgraph.transforms.task:transforms
 
-Similarly, if you need to include dynamic task defaults -- perhaps some feature
-is only available in level-3 repositories, or on specific projects -- then
-custom transforms are the appropriate tool.  Try to keep transforms simple,
-single-purpose and well-documented!
+Try to keep transforms simple, single-purpose and well-documented!
 
 Custom Run-Using
 ````````````````
 
 If the way your task is executed is unique (so, not a mach command or
 mozharness invocation), you can add a new implementation of the job
 description's "run" section.  Before you do this, consider that it might be a
 better investment to modify your task to support invocation via mozharness or