--- a/taskcluster/docs/transforms.rst
+++ b/taskcluster/docs/transforms.rst
@@ -40,25 +40,25 @@ true: items that are not yielded are eff
multiple items for each consumed item implements item duplication; this is how
test chunking is accomplished, for example.
The ``transforms`` object is an instance of
:class:`taskgraph.transforms.base.TransformSequence`, which serves as a simple
mechanism to combine a sequence of transforms into one.
Schemas
--------
+.......
The items used in transforms are validated against some simple schemas at
various points in the transformation process. These schemas accomplish two
things: they provide a place to add comments about the meaning of each field,
and they enforce that the fields are actually used in the documented fashion.
Keyed By
---------
+........
Several fields in the input items can be "keyed by" another value in the item.
For example, a test description's chunks may be keyed by ``test-platform``.
In the item, this looks like:
.. code-block:: yaml
chunks:
@@ -68,43 +68,29 @@ In the item, this looks like:
default: 10
This is a simple but powerful way to encode business rules in the items
provided as input to the transforms, rather than expressing those rules in the
transforms themselves. If you are implementing a new business rule, prefer
this mode where possible. The structure is easily resolved to a single value
using :func:`taskgraph.transform.base.get_keyed_by`.
-Task-Generation Transforms
---------------------------
-
-Every kind needs to create tasks, and all of those tasks have some things in
-common. They all run on one of a small set of worker implementations, each
-with their own idiosyncracies. And they all report to TreeHerder in a similar
-way.
+Organization
+-------------
-The transforms in ``taskcluster/taskgraph/transforms/task.py`` implement
-this common functionality. They expect a "task description", and produce a
-task definition. The schema for a task description is defined at the top of
-``task.py``, with copious comments. The parts of the task description
-that are specific to a worker implementation are isolated in a ``worker``
-object which has an ``implementation`` property naming the worker
-implementation. Thus the transforms that produce a task description must be
-aware of the worker implementation to be used, but need not be aware of the
-details of its payload format.
+Task creation operates broadly in a few phases, with the interfaces of those
+stages defined by schemas. The process begins with the raw data structures
+parsed from the YAML files in the kind configuration. This data can processed
+by kind-specific transforms resulting, for test jobs, in a "test description".
+The shared test-description transforms then create a "task description", which
+the task-generation transforms then convert into a task definition suitable for
+``queue.createTask``.
-The result is a dictionary with keys ``label``, ``attributes``, ``task``, and
-``dependencies``, with the latter having the same format as the input
-dependencies.
-
-These transforms assign names to treeherder groups using an internal list of
-group names. Feel free to add additional groups to this list as necessary.
-
-Test Transforms
----------------
+Test Descriptions
+-----------------
The transforms configured for test kinds proceed as follows, based on
configuration in ``kind.yml``:
* The test description is validated to conform to the schema in
``taskcluster/taskgraph/transforms/tests/test_description.py``. This schema
is extensively documented and is a the primary reference for anyone
modifying tests.
@@ -130,8 +116,44 @@ configuration in ``kind.yml``:
embodies the specifics of how test runs work: invoking mozharness, various
worker options, and so on.
* Finally, the ``taskgraph.transforms.task:transforms``, described above
under "Task-Generation Transforms", are applied.
Test dependencies are produced in the form of a dictionary mapping dependency
name to task label.
+
+Task Descriptions
+-----------------
+
+Every kind needs to create tasks, and all of those tasks have some things in
+common. They all run on one of a small set of worker implementations, each
+with their own idiosyncracies. And they all report to TreeHerder in a similar
+way.
+
+The transforms in ``taskcluster/taskgraph/transforms/task.py`` implement
+this common functionality. They expect a "task description", and produce a
+task definition. The schema for a task description is defined at the top of
+``task.py``, with copious comments. In general, these transforms handle
+functionality that is common to all Gecko tasks. While the schema is the
+definitive reference, the functionality includes:
+
+* TreeHerder metadata
+
+* Build index routes
+
+* Information about the projects on which this task should run
+
+* Optimizations
+
+* Defaults for ``expires-after`` and and ``deadline-after``, based on project
+
+* Worker configuration
+
+The parts of the task description that are specific to a worker implementation
+are isolated in a ``worker`` object which has an ``implementation`` property
+naming the worker implementation. Thus the transforms that produce a task
+description must be aware of the worker implementation to be used, but need not
+be aware of the details of its payload format.
+
+This file maps treeherder groups to group names using an internal list of group
+names. Feel free to add additional groups to this list as necessary.--- a/taskcluster/taskgraph/transforms/task.py
+++ b/taskcluster/taskgraph/transforms/task.py
@@ -5,16 +5,18 @@
These transformations take a task description and turn it into a TaskCluster
task definition (along with attributes, label, etc.). The input to these
transformations is generic to any kind of task, but abstracts away some of the
complexities of worker implementations, scopes, and treeherder annotations.
"""
from __future__ import absolute_import, print_function, unicode_literals
+import time
+
from taskgraph.util.treeherder import split_symbol
from taskgraph.transforms.base import (
validate_schema,
TransformSequence
)
from voluptuous import Schema, Any, Required, Optional, Extra
# shortcut for a string where task references are allowed
@@ -72,36 +74,76 @@ task_description_schema = Schema({
# treeherder.machine.platform and treeherder.collection or
# treeherder.labels
'platform': basestring,
# treeherder environments (defaults to both staging and production)
Required('environments', default=['production', 'staging']): ['production', 'staging'],
},
- # the provisioner-id/worker-type for the task
+ # information for indexing this build so its artifacts can be discovered;
+ # if omitted, the build will not be indexed.
+ Optional('index'): {
+ # the name of the product this build produces
+ 'product': Any('firefox', 'mobile', 'b2g'),
+
+ # the names to use for this job in the TaskCluster index
+ 'job-name': Any(
+ # Assuming the job is named "normally", this is the v2 job name,
+ # and the v1 and buildbot routes will be determined appropriately.
+ basestring,
+
+ # otherwise, give separate names for each of the legacy index
+ # routes; if a name is omitted, no corresponding route will be
+ # created.
+ {
+ # the name as it appears in buildbot routes
+ Optional('buildbot'): basestring,
+ Optional('gecko-v1'): basestring,
+ Required('gecko-v2'): basestring,
+ }
+ ),
+ },
+
+ # The `run_on_projects` attribute, defaulting to "all". This dictates the
+ # projects on which this task should be included in the target task set.
+ # See the attributes documentation for details.
+ Optional('run-on-projects'): [basestring],
+
+ # If the task can be coalesced, this is the name used in the coalesce key
+ # the project, etc. will be added automatically. Note that try (level 1)
+ # tasks are never coalesced
+ Optional('coalesce-name'): basestring,
+
+ # the provisioner-id/worker-type for the task. The following parameters will
+ # be substituted in this string:
+ # {level} -- the scm level of this push
'worker-type': basestring,
# information specific to the worker implementation that will run this task
'worker': Any({
Required('implementation'): Any('docker-worker', 'docker-engine'),
- # the docker image (in docker's `host/repo/image:tag` format) in which
- # to run the task; if omitted, this will be a reference to the image
- # generated by the 'docker-image' dependency, which must be defined in
- # 'dependencies'
- Optional('docker-image'): basestring,
+ # For tasks that will run in docker-worker or docker-engine, this is the
+ # name of the docker image or in-tree docker image to run the task in. If
+ # in-tree, then a dependency will be created automatically. This is
+ # generally `desktop-test`, or an image that acts an awful lot like it.
+ Required('docker-image'): Any(
+ # a raw Docker image path (repo/image:tag)
+ basestring,
+ # an in-tree generated docker image (from `testing/docker/<name>`)
+ {'in-tree': basestring}
+ ),
# worker features that should be enabled
Required('relengapi-proxy', default=False): bool,
Required('taskcluster-proxy', default=False): bool,
Required('allow-ptrace', default=False): bool,
Required('loopback-video', default=False): bool,
Required('loopback-audio', default=False): bool,
- Optional('superseder-url'): basestring,
# caches to set up for the task
Optional('caches'): [{
# only one type is supported by any of the workers right now
'type': 'persistent',
# name of the cache, allowing re-use by subsequent tasks naming the
# same cache
@@ -165,17 +207,16 @@ task_description_schema = Schema({
Optional('repository'): basestring,
Optional('project'): basestring,
},
'properties': {
'product': basestring,
Extra: basestring, # additional properties are allowed
},
}),
-
})
GROUP_NAMES = {
'tc': 'Executed by TaskCluster',
'tc-e10s': 'Executed by TaskCluster with e10s',
'tc-Fxfn-l': 'Firefox functional tests (local) executed by TaskCluster',
'tc-Fxfn-l-e10s': 'Firefox functional tests (local) executed by TaskCluster with e10s',
'tc-Fxfn-r': 'Firefox functional tests (remote) executed by TaskCluster',
@@ -184,95 +225,130 @@ GROUP_NAMES = {
'tc-M-e10s': 'Mochitests executed by TaskCluster with e10s',
'tc-R': 'Reftests executed by TaskCluster',
'tc-R-e10s': 'Reftests executed by TaskCluster with e10s',
'tc-VP': 'VideoPuppeteer tests executed by TaskCluster',
'tc-W': 'Web platform tests executed by TaskCluster',
'tc-W-e10s': 'Web platform tests executed by TaskCluster with e10s',
'tc-X': 'Xpcshell tests executed by TaskCluster',
'tc-X-e10s': 'Xpcshell tests executed by TaskCluster with e10s',
+ 'tc-Sim': 'Mulet simulator runs',
+ 'Cc': 'Toolchain builds',
+ 'SM-tc': 'Spidermonkey builds',
}
UNKNOWN_GROUP_NAME = "Treeherder group {} has no name; add it to " + __file__
+BUILDBOT_ROUTE_TEMPLATES = [
+ "index.buildbot.branches.{project}.{job-name-buildbot}",
+ "index.buildbot.revisions.{head_rev}.{project}.{job-name-buildbot}",
+]
+
+V1_ROUTE_TEMPLATES = [
+ "index.gecko.v1.{project}.latest.linux.{job-name-gecko-v1}",
+ "index.gecko.v1.{project}.revision.linux.{head_rev}.{job-name-gecko-v1}",
+]
+
+V2_ROUTE_TEMPLATES = [
+ "index.gecko.v2.{project}.latest.{product}.{job-name-gecko-v2}",
+ "index.gecko.v2.{project}.pushdate.{pushdate_long}.{product}.{job-name-gecko-v2}",
+ "index.gecko.v2.{project}.revision.{head_rev}.{product}.{job-name-gecko-v2}",
+]
+
+# the roots of the treeherder routes, keyed by treeherder environment
+TREEHERDER_ROUTE_ROOTS = {
+ 'production': 'tc-treeherder',
+ 'staging': 'tc-treeherder-stage',
+}
+
+COALESCE_KEY = 'builds.{project}.{name}'
# define a collection of payload builders, depending on the worker implementation
payload_builders = {}
def payload_builder(name):
def wrap(func):
payload_builders[name] = func
return func
return wrap
@payload_builder('docker-worker')
def build_docker_worker_payload(config, task, task_def):
worker = task['worker']
- if 'docker-image' in worker:
- # a literal image name
- image = {
- 'type': 'docker-image',
- 'name': worker['docker-image'],
- }
- else:
- assert 'docker-image' in task['dependencies'], 'no docker-worker dependency'
+ image = worker['docker-image']
+ if isinstance(image, dict):
+ docker_image_task = 'build-docker-image-' + image['in-tree']
+ task.setdefault('dependencies', {})['docker-image'] = docker_image_task
image = {
"path": "public/image.tar",
"taskId": {"task-reference": "<docker-image>"},
"type": "task-image",
}
features = {}
if worker.get('relengapi-proxy'):
features['relengAPIProxy'] = True
+ if worker.get('taskcluster-proxy'):
+ features['taskclusterProxy'] = True
+
if worker.get('allow-ptrace'):
features['allowPtrace'] = True
task_def['scopes'].append('docker-worker:feature:allowPtrace')
capabilities = {}
for lo in 'audio', 'video':
if worker.get('loopback-' + lo):
capitalized = 'loopback' + lo.capitalize()
devices = capabilities.setdefault('devices', {})
devices[capitalized] = True
task_def['scopes'].append('docker-worker:capability:device:' + capitalized)
- caches = {}
-
- for cache in worker['caches']:
- caches[cache['name']] = cache['mount-point']
- task_def['scopes'].append('docker-worker:cache:' + cache['name'])
-
- artifacts = {}
-
- for artifact in worker['artifacts']:
- artifacts[artifact['name']] = {
- 'path': artifact['path'],
- 'type': artifact['type'],
- 'expires': task_def['expires'], # always expire with the task
- }
-
task_def['payload'] = payload = {
'command': worker['command'],
- 'cache': caches,
- 'artifacts': artifacts,
'image': image,
'env': worker['env'],
- 'maxRunTime': worker['max-run-time'],
}
+
+ if 'max-run-time' in worker:
+ payload['maxRunTime'] = worker['max-run-time']
+
+ if 'artifacts' in worker:
+ artifacts = {}
+ for artifact in worker['artifacts']:
+ artifacts[artifact['name']] = {
+ 'path': artifact['path'],
+ 'type': artifact['type'],
+ 'expires': task_def['expires'], # always expire with the task
+ }
+ payload['artifacts'] = artifacts
+
+ if 'caches' in worker:
+ caches = {}
+ for cache in worker['caches']:
+ caches[cache['name']] = cache['mount-point']
+ task_def['scopes'].append('docker-worker:cache:' + cache['name'])
+ payload['cache'] = caches
+
if features:
payload['features'] = features
if capabilities:
payload['capabilities'] = capabilities
+ # coalesce / superseding
+ if 'coalesce-name' in task and int(config.params['level']) > 1:
+ key = COALESCE_KEY.format(
+ project=config.params['project'],
+ name=task['coalesce-name'])
+ payload['supersederUrl'] = "https://coalesce.mozilla-releng.net/v1/list/" + key
+
@payload_builder('generic-worker')
def build_generic_worker_payload(config, task, task_def):
worker = task['worker']
artifacts = []
for artifact in worker['artifacts']:
@@ -297,51 +373,113 @@ transforms = TransformSequence()
def validate(config, tasks):
for task in tasks:
yield validate_schema(
task_description_schema, task,
"In task {!r}:".format(task.get('label', '?no-label?')))
@transforms.add
+def add_index_routes(config, tasks):
+ for task in tasks:
+ index = task.get('index')
+ routes = task.setdefault('routes', [])
+
+ if not index:
+ yield task
+ continue
+
+ job_name = index['job-name']
+ # unpack the v2 name to v1 and buildbot names
+ if isinstance(job_name, basestring):
+ base_name, type_name = job_name.rsplit('-', 1)
+ job_name = {
+ 'buildbot': base_name,
+ 'gecko-v1': '{}.{}'.format(base_name, type_name),
+ 'gecko-v2': '{}-{}'.format(base_name, type_name),
+ }
+ subs = config.params.copy()
+ for n in job_name:
+ subs['job-name-' + n] = job_name[n]
+ subs['pushdate_long'] = time.strftime(
+ "%Y.%m.%d.%Y%m%d%H%M%S",
+ time.gmtime(config.params['pushdate']))
+ subs['product'] = index['product']
+
+ if 'buildbot' in job_name:
+ for tpl in BUILDBOT_ROUTE_TEMPLATES:
+ routes.append(tpl.format(**subs))
+ if 'gecko-v1' in job_name:
+ for tpl in V1_ROUTE_TEMPLATES:
+ routes.append(tpl.format(**subs))
+ if 'gecko-v2' in job_name:
+ for tpl in V2_ROUTE_TEMPLATES:
+ routes.append(tpl.format(**subs))
+
+ # rank is zero for non-tier-1 tasks and based on pushid for others;
+ # this sorts tier-{2,3} builds below tier-1 in the index
+ tier = task.get('treeherder', {}).get('tier', 3)
+ task.setdefault('extra', {})['index'] = {
+ 'rank': 0 if tier > 1 else int(config.params['pushdate'])
+ }
+ del task['index']
+ yield task
+
+
+@transforms.add
def build_task(config, tasks):
for task in tasks:
- provisioner_id, worker_type = task['worker-type'].split('/', 1)
+ worker_type = task['worker-type'].format(level=str(config.params['level']))
+ provisioner_id, worker_type = worker_type.split('/', 1)
+
routes = task.get('routes', [])
scopes = task.get('scopes', [])
# set up extra
extra = task.get('extra', {})
task_th = task.get('treeherder')
if task_th:
extra['treeherderEnv'] = task_th['environments']
treeherder = extra.setdefault('treeherder', {})
machine_platform, collection = task_th['platform'].split('/', 1)
treeherder['machine'] = {'platform': machine_platform}
treeherder['collection'] = {collection: True}
groupSymbol, symbol = split_symbol(task_th['symbol'])
- treeherder['groupSymbol'] = groupSymbol
- if groupSymbol not in GROUP_NAMES:
- raise Exception(UNKNOWN_GROUP_NAME.format(groupSymbol))
- treeherder['groupName'] = GROUP_NAMES[groupSymbol]
+ if groupSymbol != '?':
+ treeherder['groupSymbol'] = groupSymbol
+ if groupSymbol not in GROUP_NAMES:
+ raise Exception(UNKNOWN_GROUP_NAME.format(groupSymbol))
+ treeherder['groupName'] = GROUP_NAMES[groupSymbol]
treeherder['symbol'] = symbol
treeherder['jobKind'] = task_th['kind']
treeherder['tier'] = task_th['tier']
routes.extend([
- '{}.v2.{}.{}.{}'.format(root,
+ '{}.v2.{}.{}.{}'.format(TREEHERDER_ROUTE_ROOTS[env],
config.params['project'],
config.params['head_rev'],
config.params['pushlog_id'])
- for root in 'tc-treeherder', 'tc-treeherder-stage'
+ for env in task_th['environments']
])
+ if 'expires-after' not in task:
+ task['expires-after'] = '14 days' if config.params['project'] == 'try' else '1 year'
+
+ if 'deadline-after' not in task:
+ task['deadline-after'] = '1 day'
+
+ if 'coalesce-name' in task and int(config.params['level']) > 1:
+ key = COALESCE_KEY.format(
+ project=config.params['project'],
+ name=task['coalesce-name'])
+ routes.append('coalesce.v1.' + key)
+
task_def = {
'provisionerId': provisioner_id,
'workerType': worker_type,
'routes': routes,
'created': {'relative-datestamp': '0 seconds'},
'deadline': {'relative-datestamp': task['deadline-after']},
'expires': {'relative-datestamp': task['expires-after']},
'scopes': scopes,
@@ -356,14 +494,18 @@ def build_task(config, tasks):
},
'extra': extra,
'tags': {'createdForUser': config.params['owner']},
}
# add the payload and adjust anything else as required (e.g., scopes)
payload_builders[task['worker']['implementation']](config, task, task_def)
+ attributes = task.get('attributes', {})
+ attributes['run_on_projects'] = task.get('run-on-projects', ['all'])
+
yield {
'label': task['label'],
'task': task_def,
- 'dependencies': task['dependencies'],
- 'attributes': task['attributes'],
+ 'dependencies': task.get('dependencies', {}),
+ 'attributes': attributes,
+ 'when': task.get('when', {}),
}