Bug 1391776: cleanup of taskgraph docs; r=ahal
* eliminate heading for test kinds, of which there is now only one
* make the caches document have a single heading in the TOC
* break out mach commands into a separate document, add ./mach taskgraph morphed
* remove docs for YAML templates support (the .yml file wasn't actually
used -- I expect it was a merge leftover); these are still used for actions.yml,
but once that is gone the code should be removed, too.
* break try out into its own document, edit to distinguish "how to run try"
from "how to generate config"
MozReview-Commit-ID: 76ZopWA9TPL
deleted file mode 100644
--- a/taskcluster/ci/nightly-fennec/docker_build.yml
+++ /dev/null
@@ -1,23 +0,0 @@
-$inherits:
- from: 'build.yml'
-
-
-task:
- workerType: b2gbuild
-
- scopes:
- # docker build tasks use tc-vcs so include the scope.
- - 'docker-worker:cache:level-{{level}}-{{project}}-tc-vcs'
-
- payload:
-
- cache:
- level-{{level}}-{{project}}-tc-vcs: '/home/worker/.tc-vcs'
-
- # All docker builds share a common artifact directory for ease of uploading.
- artifacts:
- 'public/build':
- type: directory
- path: '/home/worker/artifacts/'
- expires:
- relative-datestamp: '1 year'
--- a/taskcluster/docs/caches.rst
+++ b/taskcluster/docs/caches.rst
@@ -11,17 +11,17 @@ tasks. For example, if 2 tasks run on a
and both tasks request the same cache, the subsequent task will be
able to see files in the cache that were created by the first task.
It's also worth noting that TaskCluster workers ensure a cache can only
be used by 1 task at a time. If a worker is simultaneously running
multiple tasks requesting the same named cache, the worker will
have multiple caches of the same name on the worker.
Caches and ``run-task``
-=======================
+-----------------------
``run-task`` is our generic task wrapper script. It does common activities
like ensure a version control checkout is present.
One of the roles of ``run-task`` is to verify and sanitize caches.
It does this by storing state in a cache on its first use. If the recorded
*capabilities* of an existing cache don't match expectations for the
current task, ``run-task`` bails. This ensures that caches are only
@@ -34,20 +34,20 @@ In addition, the hash of ``run-task`` is
So any time ``run-task`` changes, a new set of caches are used. This
ensures that any backwards incompatible changes or bug fixes to
``run-task`` result in fresh caches.
Some caches are reserved for use with run-task. That property will be denoted
below.
Common Caches
-=============
+-------------
Version Control Caches
-----------------------
+::::::::::::::::::::::
``level-{{level}}-checkouts-{{hash}}``
This cache holds version control checkouts, each in a subdirectory named
after the repo (e.g., ``gecko``).
Checkouts should be read-only. If a task needs to create new files from
content of a checkout, this content should be written in a separate
directory/cache (like a workspace).
@@ -72,25 +72,25 @@ Version Control Caches
directory. If you are using ``hg robustcheckout``, pass this directory to the
``--sharebase`` option.
``level-{{level}}-{{project}}-tc-vcs`` (deprecated)
This cache is used internally by ``tc-vcs``. This tool is deprecated and
should be replaced with ``hg robustcheckout``.
Workspace Caches
-----------------
+::::::::::::::::
``level-{{level}}-*-workspace``
These caches (of various names typically ending with ``workspace``)
contain state to be shared between task invocations. Use cases are
dependent on the task.
Other
------
+:::::
``level-{{level}}-tooltool-cache-{{hash}}
Tooltool invocations should use this cache. Tooltool will store files here
indexed by their hash.
This cache name pattern is reserved for use with ``run-task`` and must only
be used by ``run-task``
--- a/taskcluster/docs/how-tos.rst
+++ b/taskcluster/docs/how-tos.rst
@@ -26,54 +26,45 @@ more common changes to the task graph wi
gradual transition, please open a new bug to remind yourself to remove the
complexity when the transition is complete.
Hacking Task Graphs
-------------------
The recommended process for changing task graphs is this:
-1. Find a recent decision task on the project or branch you are working on, and
- download its ``parameters.yml`` artifact. Alternately, you
- can simply take note of the artifact URL, or just the decision task's
- ``task-id``. This file contains all of the inputs to the task-graph
- generation process. Its contents are simple enough if you would like to
- modify it, and it is documented in :doc:`parameters`.
-
-2. Run one of the ``mach taskgraph`` subcommands (see :doc:`taskgraph`) to
- generate a baseline against which to measure your changes, passing the
- parameters you found in the previous step. For example:
+1. Run one of the ``mach taskgraph`` subcommands (see :doc:`mach`) to
+ generate a baseline against which to measure your changes.
.. code-block:: none
- ./mach taskgraph tasks --json -p parameters.yml > old-tasks.json
- ./mach taskgraph tasks --json -p url/to/parameters.yml > old-tasks.json
- ./mach taskgraph tasks --json -p task-id=<task-id> > old-tasks.json
+ ./mach taskgraph tasks --json > old-tasks.json
-3. Make your modifications under ``taskcluster/``.
+2. Make your modifications under ``taskcluster/``.
-4. Run the same ``mach taskgraph`` command, sending the output to a new file,
+3. Run the same ``mach taskgraph`` command, sending the output to a new file,
and use ``diff`` to compare the old and new files. Make sure your changes
- have the desired effect and no undesirable side-effects.
+ have the desired effect and no undesirable side-effects. A plain unified
+ diff should be useful for most changes, but in some cases it may be helpful
+ to post-process the JSON to strip distracting changes.
-5. When you are satisfied with the changes, push them to try to ensure that the
+4. When you are satisfied with the changes, push them to try to ensure that the
modified tasks work as expected.
Hacking Actions
...............
If you are working on an action task and wish to test it out locally, use the
``./mach taskgraph test-action-callback`` command:
.. code-block:: none
./mach taskgraph test-action-task \
--task-id I4gu9KDmSZWu3KHx6ba6tw --task-group-id sMO4ybV9Qb2tmcI1sDHClQ \
- -p parameters.yml --input input.yml \
- hello_world_action
+ --input input.yml hello_world_action
This invocation will run the hello world callback with the given inputs and
print any created tasks to stdout, rather than actually creating them.
Common Changes
--------------
Changing Test Characteristics
@@ -249,115 +240,8 @@ will make the new run-using option avail
Something Else?
...............
If you make another change not described here that turns out to be simple or
common, please include an update to this file in your patch.
-Schedule a Task on Try
-----------------------
-
-There are two methods for scheduling a task on try.
-
-The first method is a command line string called ``try syntax`` which is passed
-into the decision task via the commit message. An example try syntax might look
-like:
-
-.. parsed-literal::
-
- try: -b o -p linux64 -u mochitest-1 -t none
-
-This gets parsed by ``taskgraph.try_option_syntax:TryOptionSyntax`` and returns
-a list of matching task labels. For more information see the
-`TryServer wiki page <https://wiki.mozilla.org/Try>`_.
-
-The second method uses a checked-in file called ``try_task_config.json`` which
-lives at the root of the source dir. The format of this file is either a list
-of task labels, or a JSON object where task labels make up the keys. For
-example, the ``try_task_config.json`` file might look like:
-
-.. parsed-literal::
-
- {
- "tasks": [
- "test-windows10-64/opt-web-platform-tests-12",
- "test-windows7-32/opt-reftest-1",
- "test-windows7-32/opt-reftest-2",
- "test-windows7-32/opt-reftest-3",
- "build-linux64/debug",
- "source-test-mozlint-eslint"
- ]
- }
-
-Very simply, this will run any task label that gets passed in as well as their
-dependencies. While it is possible to manually commit this file and push to
-try, it is mainly meant to be a generation target for various `tryselect`_
-choosers.
-
-A list of all possible task labels can be obtained by running:
-
-.. parsed-literal::
-
- $ ./mach taskgraph tasks
-
-A list of task labels relevant to a tree (defaults to mozilla-central) can be
-obtained with:
-
-.. parsed-literal::
-
- $ ./mach taskgraph target
-
-Modifying Task Behavior on Try
-``````````````````````````````
-
-It's possible to alter the definition of a task with templates. Templates are
-`JSON-e`_ files that live in the `taskgraph module`_. Templates can be specified
-from the ``try_task_config.json`` like this:
-
-.. parsed-literal::
-
- {
- "tasks": [...],
- "templates": {
- artifact: {"enabled": 1}
- }
- }
-
-Each key in the templates object denotes a new template to apply, and the value
-denotes extra context to use while rendering. When specified, a template will
-be applied to every task no matter what. If the template should only be applied
-to certain kinds of tasks, this needs to be specified in the template itself
-using JSON-e `condition statements`_.
-
-The context available to the JSON-e render aims to match that of ``actions``.
-It looks like this:
-
-.. parsed-literal::
-
- {
- "task": {
- "payload": {
- "env": { ... },
- ...
- }
- "extra": {
- "treeherder": { ... },
- ...
- },
- "tags": { "kind": "<kind>", ... },
- ...
- },
- "input": {
- "enabled": 1,
- ...
- },
- "taskId": "<task id>"
- }
-
-See the `existing templates`_ for examples.
-
-.. _tryselect: https://dxr.mozilla.org/mozilla-central/source/tools/tryselect
-.. _JSON-e: https://taskcluster.github.io/json-e/
-.. _taskgraph module: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
-.. _condition statements: https://taskcluster.github.io/json-e/#%60$if%60%20-%20%60then%60%20-%20%60else%60
-.. _existing templates: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
--- a/taskcluster/docs/index.rst
+++ b/taskcluster/docs/index.rst
@@ -17,17 +17,18 @@ than you might suppose! This implementa
This section of the documentation describes the process in some detail,
referring to the source where necessary. If you are reading this with a
particular goal in mind and would rather avoid becoming a task-graph expert,
check out the :doc:`how-to section <how-tos>`.
.. toctree::
taskgraph
+ mach
loading
transforms
optimization
- yaml-templates
docker-images
cron
+ try
actions
how-tos
reference
--- a/taskcluster/docs/kinds.rst
+++ b/taskcluster/docs/kinds.rst
@@ -93,22 +93,25 @@ will eventually be dependencies of the b
are run manually via try pushes and the results uploaded to tooltool.
spidermonkey
------------
Spidermonkey tasks check out the full gecko source tree, then compile only the
spidermonkey portion. Each task runs specific tests after the build.
-Tests
------
+test
+----
-Test tasks for Gecko products are divided into several kinds, but share a
-common implementation. The process goes like this, based on a set of YAML
-files named in ``kind.yml``:
+The ``desktop-test`` kind defines tests for builds. Its ``tests.yml`` defines
+the full suite of desktop tests and their particulars, leaving it to the
+transforms to determine how those particulars apply to the various platforms.
+
+The process of generating tests goes like this, based on a set of YAML files
+named in ``kind.yml``:
* For each build task, determine the related test platforms based on the build
platform. For example, a Windows 2010 build might be tested on Windows 7
and Windows 10. Each test platform specifies "test sets" indicating which
tests to run. This is configured in the file named
``test-platforms.yml``.
* Each test set is expanded to a list of tests to run. This is configured in
@@ -125,27 +128,18 @@ files named in ``kind.yml``:
``kind.yml``. See :doc:`transforms`: for more information on these
transforms.
* The resulting tasks become a part of the task graph.
.. important::
This process generates *all* test jobs, regardless of tree or try syntax.
- It is up to a later stage of the task-graph generation (the target set) to
- select the tests that will actually be performed.
-
-test
-....
-
-The ``desktop-test`` kind defines tests for builds. Its ``tests.yml`` defines
-the full suite of desktop tests and their particulars, leaving it to the
-transforms to determine how those particulars apply to the various platforms.
-
-This kind includes both unit tests and talos.
+ It is up to a later stages of the task-graph generation (the target set and
+ optimization) to select the tests that will actually be performed.
docker-image
------------
Tasks of the ``docker-image`` kind build the Docker images in which other
Docker tasks run.
The tasks to generate each docker image have predictable labels:
new file mode 100644
--- /dev/null
+++ b/taskcluster/docs/mach.rst
@@ -0,0 +1,110 @@
+Mach commands
+=============
+
+A number of mach subcommands are available aside from ``mach taskgraph
+decision`` to make this complex system more accessible to those trying to
+understand or modify it. They allow you to run portions of the
+graph-generation process and output the results.
+
+``mach taskgraph tasks``
+ Get the full task set
+
+``mach taskgraph full``
+ Get the full task graph
+
+``mach taskgraph target``
+ Get the target task set
+
+``mach taskgraph target-graph``
+ Get the target task graph
+
+``mach taskgraph optimized``
+ Get the optimized task graph
+
+``mach taskgraph morphed``
+ Get the morhped task graph
+
+Each of these commands takes an optional ``--parameters`` option giving a file
+with parameters to guide the graph generation. The decision task helpfully
+produces such a file on every run, and that is generally the easiest way to get
+a parameter file. The parameter keys and values are described in
+:doc:`parameters`; using that information, you may modify an existing
+``parameters.yml`` or create your own. The ``--parameters`` option can also
+take an argument of the form ``project=<project>`` which will fetch the
+parameters from the latest push on that project; or ``task-id=<task-id>`` which
+will fetch the parameters from the given decision task. It defaults to
+``project=mozilla-central``.
+
+See :doc:`how-tos` for further practical tips on debugging task-graph mechanics
+locally.
+
+Taskgraph JSON Format
+---------------------
+By default, the above commands will only output a list of tasks. Use `-J` flag
+to output full task definitions. For example:
+
+.. code-block:: shell
+
+ $ ./mach taskgraph optimized -J
+
+
+Task graphs -- both the graph artifacts produced by the decision task and those
+output by the ``--json`` option to the ``mach taskgraph`` commands -- are JSON
+objects, keyed by label, or for optimized task graphs, by taskId. For
+convenience, the decision task also writes out ``label-to-taskid.json``
+containing a mapping from label to taskId. Each task in the graph is
+represented as a JSON object.
+
+Each task has the following properties:
+
+``kind``
+ The name of this task's kind
+
+``task_id``
+ The task's taskId (only for optimized task graphs)
+
+``label``
+ The task's label
+
+``attributes``
+ The task's attributes
+
+``dependencies``
+ The task's in-graph dependencies, represented as an object mapping
+ dependency name to label (or to taskId for optimized task graphs)
+
+``optimizations``
+ The optimizations to be applied to this task
+
+``task``
+ The task's TaskCluster task definition.
+
+The results from each command are in the same format, but with some differences
+in the content:
+
+* The ``tasks`` and ``target`` subcommands both return graphs with no edges.
+ That is, just collections of tasks without any dependencies indicated.
+
+* The ``optimized`` subcommand returns tasks that have been assigned taskIds.
+ The dependencies array, too, contains taskIds instead of labels, with
+ dependencies on optimized tasks omitted. However, the ``task.dependencies``
+ array is populated with the full list of dependency taskIds. All task
+ references are resolved in the optimized graph.
+
+The output of the ``mach taskgraph`` commands are suitable for processing with
+the `jq <https://stedolan.github.io/jq/>`_ utility. For example, to extract all
+tasks' labels and their dependencies:
+
+.. code-block:: shell
+
+ jq 'to_entries | map({label: .value.label, dependencies: .value.dependencies})'
+
+An alternate way of searching the output of ``mach taskgraph`` is
+`gron <https://github.com/tomnomnom/gron>`_, which converts json into a format
+that's easily searched with ``grep``
+
+.. code-block:: shell
+
+ gron taskgraph.json | grep -E 'test.*machine.platform = "linux64";'
+ ./mach taskgraph --json | gron | grep ...
+
--- a/taskcluster/docs/taskgraph.rst
+++ b/taskcluster/docs/taskgraph.rst
@@ -1,11 +1,11 @@
-======================
-TaskGraph Mach Command
-======================
+========
+Overview
+========
The task graph is built by linking different kinds of tasks together, pruning
out tasks that are not required, then optimizing by replacing subgraphs with
links to already-completed tasks.
Concepts
--------
@@ -145,55 +145,16 @@ the same push, due to the decision task.
So for instance, if you had already requested a build task in the ``try`` command,
and you wish to add a test which depends on this build, the original build task
is re-used.
Action Tasks are currently scheduled by
[pulse_actions](https://github.com/mozilla/pulse_actions). This feature is only
present on ``try`` pushes for now.
-Mach commands
--------------
-
-A number of mach subcommands are available aside from ``mach taskgraph
-decision`` to make this complex system more accessible to those trying to
-understand or modify it. They allow you to run portions of the
-graph-generation process and output the results.
-
-``mach taskgraph tasks``
- Get the full task set
-
-``mach taskgraph full``
- Get the full task graph
-
-``mach taskgraph target``
- Get the target task set
-
-``mach taskgraph target-graph``
- Get the target task graph
-
-``mach taskgraph optimized``
- Get the optimized task graph
-
-Each of these commands taskes a ``--parameters`` option giving a file with
-parameters to guide the graph generation. The decision task helpfully produces
-such a file on every run, and that is generally the easiest way to get a
-parameter file. The parameter keys and values are described in
-:doc:`parameters`; using that information, you may modify an existing
-``parameters.yml`` or create your own.
-
-By default, the above commands will only output a list of tasks. Use `-J` flag
-to output full task definitions. For example:
-
-.. code-block:: shell
-
- $ ./mach taskgraph optimized -J -p ~/Downloads/parameters.yml
-
-See :doc:`how-tos` for further practical tips.
-
Task Parameterization
---------------------
A few components of tasks are only known at the very end of the decision task
-- just before the ``queue.createTask`` call is made. These are specified
using simple parameterized values, as follows:
``{"relative-datestamp": "certain number of seconds/hours/days/years"}``
@@ -204,71 +165,8 @@ using simple parameterized values, as fo
``{"task-reference": "string containing <dep-name>"}``
The task definition may contain "task references" of this form. These will
be replaced during the optimization step, with the appropriate taskId for
the named dependency substituted for ``<dep-name>`` in the string.
Multiple labels may be substituted in a single string, and ``<<>`` can be
used to escape a literal ``<``.
-Taskgraph JSON Format
----------------------
-
-Task graphs -- both the graph artifacts produced by the decision task and those
-output by the ``--json`` option to the ``mach taskgraph`` commands -- are JSON
-objects, keyed by label, or for optimized task graphs, by taskId. For
-convenience, the decision task also writes out ``label-to-taskid.json``
-containing a mapping from label to taskId. Each task in the graph is
-represented as a JSON object.
-
-Each task has the following properties:
-
-``kind``
- The name of this task's kind
-
-``task_id``
- The task's taskId (only for optimized task graphs)
-
-``label``
- The task's label
-
-``attributes``
- The task's attributes
-
-``dependencies``
- The task's in-graph dependencies, represented as an object mapping
- dependency name to label (or to taskId for optimized task graphs)
-
-``optimizations``
- The optimizations to be applied to this task
-
-``task``
- The task's TaskCluster task definition.
-
-The results from each command are in the same format, but with some differences
-in the content:
-
-* The ``tasks`` and ``target`` subcommands both return graphs with no edges.
- That is, just collections of tasks without any dependencies indicated.
-
-* The ``optimized`` subcommand returns tasks that have been assigned taskIds.
- The dependencies array, too, contains taskIds instead of labels, with
- dependencies on optimized tasks omitted. However, the ``task.dependencies``
- array is populated with the full list of dependency taskIds. All task
- references are resolved in the optimized graph.
-
-The output of the ``mach taskgraph`` commands are suitable for processing with
-the `jq <https://stedolan.github.io/jq/>`_ utility. For example, to extract all
-tasks' labels and their dependencies:
-
-.. code-block:: shell
-
- jq 'to_entries | map({label: .value.label, dependencies: .value.dependencies})'
-
-An alternate way of searching the output of ``mach taskgraph`` is
-`gron <https://github.com/tomnomnom/gron>`_, which converts json into a format
-that's easily searched with ``grep``
-
-.. code-block:: shell
-
- gron taskgraph.json | grep -E 'test.*machine.platform = "linux64";'
- ./mach taskgraph -p parameters.yml --json | gron | grep ...
-
new file mode 100644
--- /dev/null
+++ b/taskcluster/docs/try.rst
@@ -0,0 +1,140 @@
+Try
+===
+
+"Try" is a way to "try out" a proposed change safely before review, without
+officialy landing it. This functionality has been around for a *long* time in
+various forms, and can sometimes show its age.
+
+Access to "push to try" is typically avilable to a much larger group of
+developers than those who can land changes in integration and release branches.
+Specifically, try pushes are allowed for anyone with `SCM Level`_ 1, while
+integration branches are at SCM level 3.
+
+Scheduling a Task on Try
+------------------------
+
+There are two methods for scheduling a task on try.
+
+Try Option Syntax
+:::::::::::::::::
+
+The first, older method is a command line string called ``try syntax`` which is passed
+into the decision task via the commit message. The resulting commit is then
+pushed to the https://hg.mozilla.org/try repository. An example try syntax
+might look like:
+
+.. parsed-literal::
+
+ try: -b o -p linux64 -u mochitest-1 -t none
+
+This gets parsed by ``taskgraph.try_option_syntax:TryOptionSyntax`` and returns
+a list of matching task labels. For more information see the
+`TryServer wiki page <https://wiki.mozilla.org/Try>`_.
+
+Try Task Config
+:::::::::::::::
+
+The second, more modern method specifies exactly the tasks to run. That list
+of tasks is usually generated locally with some `local tool <tryselect>`_ and
+attached to the commit pushed to the try repository. This gives finer control
+over exactly what runs and enables growth of an ecosystem of tooling
+appropriate to varied circumstances.
+
+Implementation
+,,,,,,,,,,,,,,
+
+This method uses a checked-in file called ``try_task_config.json`` which lives
+at the root of the source dir. The JSON object in this file contains a
+``tasks`` key giving the labels of the tasks to run. For example, the
+``try_task_config.json`` file might look like:
+
+.. parsed-literal::
+
+ {
+ "tasks": [
+ "test-windows10-64/opt-web-platform-tests-12",
+ "test-windows7-32/opt-reftest-1",
+ "test-windows7-32/opt-reftest-2",
+ "test-windows7-32/opt-reftest-3",
+ "build-linux64/debug",
+ "source-test-mozlint-eslint"
+ ]
+ }
+
+Very simply, this will run any task label that gets passed in as well as their
+dependencies. While it is possible to manually commit this file and push to
+try, it is mainly meant to be a generation target for various `tryselect`_
+choosers. For example:
+
+.. parsed-literal::
+
+ $ ./mach try fuzzy
+
+A list of all possible task labels can be obtained by running:
+
+.. parsed-literal::
+
+ $ ./mach taskgraph tasks
+
+A list of task labels relevant to a tree (defaults to mozilla-central) can be
+obtained with:
+
+.. parsed-literal::
+
+ $ ./mach taskgraph target
+
+Modifying Tasks in a Try Push
+,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
+
+It's possible to alter the definition of a task with templates. Templates are
+`JSON-e`_ files that live in the `taskgraph module`_. Templates can be specified
+from the ``try_task_config.json`` like this:
+
+.. parsed-literal::
+
+ {
+ "tasks": [...],
+ "templates": {
+ artifact: {"enabled": 1}
+ }
+ }
+
+Each key in the templates object denotes a new template to apply, and the value
+denotes extra context to use while rendering. When specified, a template will
+be applied to every task no matter what. If the template should only be applied
+to certain kinds of tasks, this needs to be specified in the template itself
+using JSON-e `condition statements`_.
+
+The context available to the JSON-e render aims to match that of ``actions``.
+It looks like this:
+
+.. parsed-literal::
+
+ {
+ "task": {
+ "payload": {
+ "env": { ... },
+ ...
+ }
+ "extra": {
+ "treeherder": { ... },
+ ...
+ },
+ "tags": { "kind": "<kind>", ... },
+ ...
+ },
+ "input": {
+ "enabled": 1,
+ ...
+ },
+ "taskId": "<task id>"
+ }
+
+See the `existing templates`_ for examples.
+
+.. _tryselect: https://dxr.mozilla.org/mozilla-central/source/tools/tryselect
+.. _JSON-e: https://taskcluster.github.io/json-e/
+.. _taskgraph module: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
+.. _condition statements: https://taskcluster.github.io/json-e/#%60$if%60%20-%20%60then%60%20-%20%60else%60
+.. _existing templates: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
+.. _SCM Level: https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/
deleted file mode 100644
--- a/taskcluster/docs/yaml-templates.rst
+++ /dev/null
@@ -1,49 +0,0 @@
-Task Definition YAML Templates
-==============================
-
-A few kinds of tasks are described using templated YAML files. These files
-allow some limited forms of inheritance and template substitution as well as
-the usual YAML features, as described below.
-
-Please do not use these features in new kinds. If you are tempted to use
-variable substitution over a YAML file to define tasks, please instead
-implement a new kind-specific transform to accopmlish your goal. For example,
-if the current push-id must be included as an argument in
-``task.payload.command``, write a transform function that makes that assignment
-while building a job description, rather than parameterizing that value in the
-input to the transforms.
-
-Inheritance
------------
-
-One YAML file can "inherit" from another by including a top-level ``$inherits``
-key. That key specifies the parent file in ``from``, and optionally a
-collection of variables in ``variables``. For example:
-
-.. code-block:: yaml
-
- $inherits:
- from: 'tasks/builds/base_linux32.yml'
- variables:
- build_name: 'linux32'
- build_type: 'dbg'
-
-Inheritance proceeds as follows: First, the child document has its template
-substitutions performed and is parsed as YAML. Then, the parent document is
-parsed, with substitutions specified by ``variables`` added to the template
-substitutions. Finally, the child document is merged with the parent.
-
-To merge two JSON objects (dictionaries), each value is merged individually.
-Lists are merged by concatenating the lists from the parent and child
-documents. Atomic values (strings, numbers, etc.) are merged by preferring the
-child document's value.
-
-Substitution
-------------
-
-Each document is expanded using the PyStache template engine before it is
-parsed as YAML. The parameters for this expansion are specific to the task
-kind.
-
-Simple value substitution looks like ``{{variable}}``. Function calls look
-like ``{{#function}}argument{{/function}}``.
--- a/taskcluster/taskgraph/decision.py
+++ b/taskcluster/taskgraph/decision.py
@@ -216,16 +216,17 @@ def write_artifact(filename, data):
elif filename.endswith('.json'):
with open(path, 'w') as f:
json.dump(data, f, sort_keys=True, indent=2, separators=(',', ': '))
else:
raise TypeError("Don't know how to write to {}".format(filename))
def get_action_yml(parameters):
+ # NOTE: when deleting this function, delete taskcluster/taskgraph/util/templates.py too
templates = Templates(os.path.join(GECKO, "taskcluster/taskgraph"))
action_parameters = parameters.copy()
match = re.match(r'https://(hg.mozilla.org)/(.*?)/?$', action_parameters['head_repository'])
if not match:
raise Exception('Unrecognized head_repository')
repo_scope = 'assume:repo:{}/{}:*'.format(
match.group(1), match.group(2))