Bug 1387135 - Add ability to apply templates to task definitions via try_task_config.json, r?dustin
This provides a mechanism to modify the behaviour of tasks from a try push. The try_task_config.json
looks something like:
{
"tasks": ["build-linux64/opt", "test-linux64/opt-mochitest-e10s-1"],
"templates": {
"artifact": {"enabled": 1}
}
}
This tells taskgraph to apply the 'artifact' template to all tasks. Templates are JSONe based
.yml files that live under taskcluster/taskgraph/templates. Taskgraph will render every template
against every task definition. The templates themselves can then use JSONe condition statements to
filter out which tasks they should or shouldn't apply to.
MozReview-Commit-ID: J8HVZzOt4mX
--- a/taskcluster/docs/how-tos.rst
+++ b/taskcluster/docs/how-tos.rst
@@ -273,33 +273,91 @@ a list of matching task labels. For more
The second method uses a checked-in file called ``try_task_config.json`` which
lives at the root of the source dir. The format of this file is either a list
of task labels, or a JSON object where task labels make up the keys. For
example, the ``try_task_config.json`` file might look like:
.. parsed-literal::
- [
- "test-windows10-64/opt-web-platform-tests-12",
- "test-windows7-32/opt-reftest-1",
- "test-windows7-32/opt-reftest-2",
- "test-windows7-32/opt-reftest-3",
- "build-linux64/debug",
- "source-test-mozlint-eslint"
- ]
+ {
+ "tasks": [
+ "test-windows10-64/opt-web-platform-tests-12",
+ "test-windows7-32/opt-reftest-1",
+ "test-windows7-32/opt-reftest-2",
+ "test-windows7-32/opt-reftest-3",
+ "build-linux64/debug",
+ "source-test-mozlint-eslint"
+ ]
+ }
Very simply, this will run any task label that gets passed in as well as their
dependencies. While it is possible to manually commit this file and push to
-try, it is mainly meant to be a generation target for various trychooser tools.
+try, it is mainly meant to be a generation target for various `tryselect`_
+choosers.
A list of all possible task labels can be obtained by running:
.. parsed-literal::
$ ./mach taskgraph tasks
A list of task labels relevant to a tree (defaults to mozilla-central) can be
obtained with:
.. parsed-literal::
$ ./mach taskgraph target
+
+Modifying Task Behavior on Try
+``````````````````````````````
+
+It's possible to alter the definition of a task with templates. Templates are
+`JSON-e`_ files that live in the `taskgraph module`_. Templates can be specified
+from the ``try_task_config.json`` like this:
+
+.. parsed-literal::
+
+ {
+ "tasks": [...],
+ "templates": {
+ artifact: {"enabled": 1}
+ }
+ }
+
+Each key in the templates object denotes a new template to apply, and the value
+denotes extra context to use while rendering. When specified, a template will
+be applied to every task no matter what. If the template should only be applied
+to certain kinds of tasks, this needs to be specified in the template itself
+using JSON-e `condition statements`_.
+
+The context available to the JSON-e render aims to match that of ``actions``.
+It looks like this:
+
+.. parsed-literal::
+
+ {
+ "task": {
+ "payload": {
+ "env": { ... },
+ ...
+ }
+ "extra": {
+ "treeherder": { ... },
+ ...
+ },
+ "tags": { "kind": "<kind>", ... },
+ ...
+ },
+ "input": {
+ "enabled": 1,
+ ...
+ },
+ "taskId": "<task id>"
+ }
+
+See the `existing templates`_ for examples.
+
+.. _tryselect: https://dxr.mozilla.org/mozilla-central/source/tools/tryselect
+.. _JSON-e: https://taskcluster.github.io/json-e/
+.. _taskgraph module: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
+.. _condition statements: https://taskcluster.github.io/json-e/#%60$if%60%20-%20%60then%60%20-%20%60else%60
+.. _existing templates: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/templates
--- a/taskcluster/docs/parameters.rst
+++ b/taskcluster/docs/parameters.rst
@@ -88,17 +88,30 @@ those in the target set, recursively. I
specified programmatically using one of a variety of methods (e.g., parsing try
syntax or reading a project-specific configuration file).
``filters``
List of filter functions (from ``taskcluster/taskgraph/filter_tasks.py``) to
apply. This is usually defined internally, as filters are typically
global.
+``target_task_labels``
+ List of task labels to select. Labels not listed will be filtered out.
+ Enabled on try only.
+
``target_tasks_method``
The method to use to determine the target task set. This is the suffix of
one of the functions in ``taskcluster/taskgraph/target_tasks.py``.
``optimize_target_tasks``
- If true, then target tasks are eligible for optimization.
+ If true, then target tasks are eligible for optimization.
``include_nightly``
- If true, then nightly tasks are eligible for optimization.
+ If true, then nightly tasks are eligible for optimization.
+
+Morphed Set
+-----------
+
+``morph_templates``
+ Dict of JSON-e templates to apply to each task, keyed by template name.
+ Values are extra context that will be available to the template under the
+ ``input.<template>`` key. Available templates live in
+ ``taskcluster/taskgraph/templates``. Enabled on try only.
--- a/taskcluster/taskgraph/decision.py
+++ b/taskcluster/taskgraph/decision.py
@@ -161,16 +161,18 @@ def get_decision_parameters(options):
] if n in options}
# Define default filter list, as most configurations shouldn't need
# custom filters.
parameters['filters'] = [
'check_servo',
'target_tasks_method',
]
+ parameters['target_task_labels'] = []
+ parameters['morph_templates'] = {}
# owner must be an email, but sometimes (e.g., for ffxbld) it is not, in which
# case, fake it
if '@' not in parameters['owner']:
parameters['owner'] += '@noreply.mozilla.org'
# use the pushdate as build_date if given, else use current time
parameters['build_date'] = parameters['pushdate'] or int(time.time())
@@ -182,16 +184,25 @@ def get_decision_parameters(options):
try:
parameters.update(PER_PROJECT_PARAMETERS[project])
except KeyError:
logger.warning("using default project parameters; add {} to "
"PER_PROJECT_PARAMETERS in {} to customize behavior "
"for this project".format(project, __file__))
parameters.update(PER_PROJECT_PARAMETERS['default'])
+ # morph_templates and target_task_labels are only used on try, so don't
+ # bother loading them elsewhere
+ task_config_file = os.path.join(GECKO, 'try_task_config.json')
+ if project == 'try' and os.path.isfile(task_config_file):
+ with open(task_config_file, 'r') as fh:
+ task_config = json.load(fh)
+ parameters['morph_templates'] = task_config.get('templates', {})
+ parameters['target_task_labels'] = task_config.get('tasks')
+
# `target_tasks_method` has higher precedence than `project` parameters
if options.get('target_tasks_method'):
parameters['target_tasks_method'] = options['target_tasks_method']
return Parameters(parameters)
def write_artifact(filename, data):
--- a/taskcluster/taskgraph/generator.py
+++ b/taskcluster/taskgraph/generator.py
@@ -275,17 +275,18 @@ class TaskGraphGenerator(object):
if not self.parameters.get('optimize_target_tasks', True):
do_not_optimize = target_task_set.graph.nodes
optimized_task_graph, label_to_taskid = optimize_task_graph(target_task_graph,
self.parameters,
do_not_optimize)
yield 'optimized_task_graph', optimized_task_graph
- morphed_task_graph, label_to_taskid = morph(optimized_task_graph, label_to_taskid)
+ morphed_task_graph, label_to_taskid = morph(
+ optimized_task_graph, label_to_taskid, self.parameters)
yield 'label_to_taskid', label_to_taskid
yield 'morphed_task_graph', morphed_task_graph
def _run_until(self, name):
while name not in self._run_results:
try:
k, v = self._run.next()
--- a/taskcluster/taskgraph/morph.py
+++ b/taskcluster/taskgraph/morph.py
@@ -14,23 +14,27 @@ the graph.
# Note that the translation of `{'task-reference': '..'}` is handled in the
# optimization phase (since optimization involves dealing with taskIds
# directly). Similarly, `{'relative-datestamp': '..'}` is handled at the last
# possible moment during task creation.
from __future__ import absolute_import, print_function, unicode_literals
import logging
+import os
import re
+import jsone
+import yaml
from slugid import nice as slugid
from .task import Task
from .graph import Graph
from .taskgraph import TaskGraph
+here = os.path.abspath(os.path.dirname(__file__))
logger = logging.getLogger(__name__)
MAX_ROUTES = 10
def amend_taskgraph(taskgraph, label_to_taskid, to_add):
"""Add the given tasks to the taskgraph, returning a new taskgraph"""
new_tasks = taskgraph.tasks.copy()
new_edges = taskgraph.graph.edges.copy()
@@ -236,17 +240,51 @@ def add_s3_uploader_task(taskgraph, labe
added = make_s3_uploader_task(task)
taskgraph, label_to_taskid = amend_taskgraph(
taskgraph, label_to_taskid, [added])
update_test_tasks(added.task_id, task.task_id, taskgraph)
logger.info('Added s3-uploader task')
return taskgraph, label_to_taskid
-def morph(taskgraph, label_to_taskid):
+class apply_jsone_templates(object):
+ """Apply a set of JSON-e templates to each task's `task` attribute.
+
+ :param templates: A dict with the template name as the key, and extra context
+ to use (in addition to task.to_json()) as the value.
+ """
+ template_dir = os.path.join(here, 'templates')
+
+ def __init__(self, templates):
+ self.templates = templates
+
+ def __call__(self, taskgraph, label_to_taskid):
+ if not self.templates:
+ return taskgraph, label_to_taskid
+
+ for task in taskgraph.tasks.itervalues():
+ for template in sorted(self.templates):
+ context = {
+ 'task': task.task,
+ 'taskGroup': None,
+ 'taskId': task.task_id,
+ 'kind': task.kind,
+ 'input': self.templates[template],
+ }
+
+ template_path = os.path.join(self.template_dir, template + '.yml')
+ with open(template_path) as f:
+ template = yaml.load(f)
+ task.task = jsone.render(template, context)
+
+ return taskgraph, label_to_taskid
+
+
+def morph(taskgraph, label_to_taskid, parameters):
"""Apply all morphs"""
morphs = [
add_index_tasks,
add_s3_uploader_task,
+ apply_jsone_templates(parameters.get('morph_templates')),
]
for m in morphs:
taskgraph, label_to_taskid = m(taskgraph, label_to_taskid)
return taskgraph, label_to_taskid
--- a/taskcluster/taskgraph/parameters.py
+++ b/taskcluster/taskgraph/parameters.py
@@ -16,33 +16,40 @@ PARAMETER_NAMES = set([
'build_date',
'filters',
'head_ref',
'head_repository',
'head_rev',
'include_nightly',
'level',
'message',
+ 'morph_templates',
'moz_build_date',
'optimize_target_tasks',
'owner',
'project',
'pushdate',
'pushlog_id',
+ 'target_task_labels',
'target_tasks_method',
])
+TRY_ONLY_PARAMETERS = set([
+ 'morph_templates',
+ 'target_task_labels',
+])
+
class Parameters(ReadOnlyDict):
"""An immutable dictionary with nicer KeyError messages on failure"""
def check(self):
names = set(self)
msg = []
- missing = PARAMETER_NAMES - names
+ missing = PARAMETER_NAMES - TRY_ONLY_PARAMETERS - names
if missing:
msg.append("missing parameters: " + ", ".join(missing))
extra = names - PARAMETER_NAMES
if extra:
msg.append("extra parameters: " + ", ".join(extra))
if msg:
--- a/taskcluster/taskgraph/target_tasks.py
+++ b/taskcluster/taskgraph/target_tasks.py
@@ -2,21 +2,21 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
from __future__ import absolute_import, print_function, unicode_literals
import os
-import json
from taskgraph import try_option_syntax
from taskgraph.util.attributes import match_run_on_projects
+here = os.path.abspath(os.path.dirname(__file__))
_target_task_methods = {}
def _target_task(name):
def wrap(func):
_target_task_methods[name] = func
return func
return wrap
@@ -48,30 +48,21 @@ def filter_upload_symbols(task, paramete
def standard_filter(task, parameters):
return all(
filter_func(task, parameters) for filter_func in
(filter_on_nightly, filter_for_project, filter_upload_symbols)
)
def _try_task_config(full_task_graph, parameters):
- task_config_file = os.path.join(os.getcwd(), 'try_task_config.json')
-
- if not os.path.isfile(task_config_file):
+ if not parameters.get('target_task_labels'):
return []
- with open(task_config_file, 'r') as fh:
- task_config = json.load(fh)
-
- target_task_labels = []
- for task in full_task_graph.tasks.itervalues():
- if task.label in task_config:
- target_task_labels.append(task.label)
-
- return target_task_labels
+ return [t.label for t in full_task_graph.tasks.itervalues()
+ if t.label in parameters['target_task_labels']]
def _try_option_syntax(full_task_graph, parameters):
"""Generate a list of target tasks based on try syntax in
parameters['message'] and, for context, the full task graph."""
options = try_option_syntax.TryOptionSyntax(parameters['message'], full_task_graph)
target_tasks_labels = [t.label for t in full_task_graph.tasks.itervalues()
if options.task_matches(t)]
--- a/taskcluster/taskgraph/test/test_target_tasks.py
+++ b/taskcluster/taskgraph/test/test_target_tasks.py
@@ -1,15 +1,14 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
from __future__ import absolute_import, print_function, unicode_literals
-import os
import unittest
from taskgraph import target_tasks
from taskgraph import try_option_syntax
from taskgraph.graph import Graph
from taskgraph.taskgraph import TaskGraph
from taskgraph.task import Task
from mozunit import main
@@ -71,39 +70,38 @@ class TestTargetTasks(unittest.TestCase)
'a': Task(kind=None, label='a', attributes={}, task={}),
'b': Task(kind=None, label='b', attributes={'at-at': 'yep'}, task={}),
'c': Task(kind=None, label='c', attributes={}, task={}),
}
graph = Graph(nodes=set('abc'), edges=set())
tg = TaskGraph(tasks, graph)
method = target_tasks.get_method('try_tasks')
- config = os.path.join(os.getcwd(), 'try_task_config.json')
+ params = {
+ 'message': '',
+ 'target_task_labels': [],
+ }
orig_TryOptionSyntax = try_option_syntax.TryOptionSyntax
try:
try_option_syntax.TryOptionSyntax = FakeTryOptionSyntax
# no try specifier
- self.assertEqual(method(tg, {'message': ''}), ['b'])
+ self.assertEqual(method(tg, params), ['b'])
# try syntax only
- self.assertEqual(method(tg, {'message': 'try: me'}), ['b'])
+ params['message'] = 'try: me'
+ self.assertEqual(method(tg, params), ['b'])
# try task config only
- with open(config, 'w') as fh:
- fh.write('["c"]')
- self.assertEqual(method(tg, {'message': ''}), ['c'])
-
- with open(config, 'w') as fh:
- fh.write('{"c": {}}')
- self.assertEqual(method(tg, {'message': ''}), ['c'])
+ params['message'] = ''
+ params['target_task_labels'] = ['c']
+ self.assertEqual(method(tg, params), ['c'])
# both syntax and config
- self.assertEqual(set(method(tg, {'message': 'try: me'})), set(['b', 'c']))
+ params['message'] = 'try: me'
+ self.assertEqual(set(method(tg, params)), set(['b', 'c']))
finally:
try_option_syntax.TryOptionSyntax = orig_TryOptionSyntax
- if os.path.isfile(config):
- os.remove(config)
if __name__ == '__main__':
main()