Bug 1329282 - Documentation for in-tree QEMU images draft
authorJonas Finnemann Jensen <jopsen@gmail.com>
Tue, 22 Aug 2017 19:45:06 -0700
changeset 671394 bcb9b81d50acf46893eda26524d105e8f8fb4d96
parent 671393 7493d10cfabdb4f2bbd70e60277c92952f259760
child 671395 033876cfe3e7833f2beb48604b3fd0079225cf13
push id81953
push userjojensen@mozilla.com
push dateWed, 27 Sep 2017 22:49:28 +0000
bugs1329282
milestone58.0a1
Bug 1329282 - Documentation for in-tree QEMU images * Description of index namespaces * Definition of image building format * Guides for working with |mach qemu| commands. MozReview-Commit-ID: 2IrWy917TCN
taskcluster/docs/index.rst
taskcluster/docs/qemu-images.rst
taskcluster/qemu/README.md
--- a/taskcluster/docs/index.rst
+++ b/taskcluster/docs/index.rst
@@ -22,13 +22,14 @@ check out the :doc:`how-to section <how-
 .. toctree::
 
     taskgraph
     mach
     loading
     transforms
     optimization
     docker-images
+    qemu-images
     cron
     try
     actions
     how-tos
     reference
new file mode 100644
--- /dev/null
+++ b/taskcluster/docs/qemu-images.rst
@@ -0,0 +1,172 @@
+.. taskcluster_qemuimages:
+
+===========
+QEMU Images
+===========
+
+QEMU images for use on taskcluster are defined in-tree. Each folder in
+``taskcluster/qemu/`` defines a QEMU image as a series of steps and scripts that
+are intended to be repeatable. The image definition for building an image can be
+consistently hashed, this allows image building tasks to be indexed and
+automatically triggered on push if the image definition have changed.
+
+Image folders that contains a ``reference.yml`` file are not automatically built
+and must be rebuilt and uploaded with ``mach qemu build <image_name>`` and
+``mach qemu push <image_name>``. This is only used for special images, like the
+image that builds other QEMU images.
+
+To make it easy to work with QEMU images locally a set of ``mach qemu`` commands
+handles downloading/uploading, caching, building and local testing of QEMU
+images. See ``mach help qemu`` for details, or see work-flow examples below.
+
+**Definitions:**
+
+``image_name``:
+  Name of the in-tree folder containing the image definition, ie.
+  ``taskcluster/qemu/<image_name>/``
+``image_hash``:
+  Hash of image definition ``taskcluster/qemu/<image_name>/image.yml`` with
+  all referenced files and folders substituted for their hash.
+
+Image Definition Format
+=======================
+An in-tree QEMU image is in defined in ``taskcluster/qemu/<image_name>/``, where
+``<image_name>`` is the named it is referenced by in task configuration files.
+The image folder must contain an ``image.yml`` file satifying the schema defined
+and documented in ``taskcluster/taskgraph/qemu/imageschema.py``.
+
+In short the ``image.yml`` file defines
+ * Treeherder symbol the image building task,
+ * A boolean indicating if the image is private,
+ * A human-readable markdown description of the image,
+ * The virtual machine configuration as accepted by taskcluster-worker,
+ * Steps for building ISOs for ``cdromA`` and ``cdromB``.
+
+See ``taskcluster/taskgraph/qemu/imageschema.py`` for documentation.
+
+When building the image the steps for construction of ``cdromA`` and ``cdromB``
+are executed, an empty machine is created and booted with ``cdromA`` as
+boot-disk and ``cdromB`` as second (optional) media.
+
+This allows for simple customization of the install media, inclusion of custom
+scripts and binaries, either on the install media or the secondary media.
+For Linux images it is often easiest to extract and repack the install media,
+such that it includes custom scripts, tweaked grub configuration and
+answer files for the installer.
+For Windows images it is possible to the official install media as ``cdromA``,
+and then construct ``cdromB`` with an ``Autounattend.xml``, custom install
+scripts, and binaries.
+
+When hashing an image definition any in-tree files or folders referenced in the
+``image.yml`` files will be replaced with their respective content-hashes, then
+the entire image definition is consistently serialized as JSON and hashed.
+This is the ``image_hash`` and tasks to build QEMU images will be indexed using
+this ``image_hash``, such that images aren't rebuild unless they have changed.
+This works as long as all external resources downloaded are referenced by hash,
+the schema for ``image.yml`` enforces this for any files included on the cdroms,
+but there is no way to enforce this on custom install scripts.
+
+
+Developing QEMU Images Locally
+==============================
+
+When developing QEMU images it can sometimes be faster to test locally than
+pushing to try. This section outlines a local work-flow for developing QEMU
+images using ``mach qemu`` commands.
+
+.. note::
+  The various ``mach qemu`` commands requires a Linux machine with
+  docker and KVM. All binaries like QEMU, etc. are stored in a docker image
+  that will be pulled on demand. Use ``mach qemu setup`` to
+  validate your setup works.
+
+An image is identified by ``<image_name>@sha256:<image_hash>`` where:
+
+ * ``<image_name>`` is the name of the image folder in ``taskcluster/qemu/``, and,
+ * ``<image_hash>`` is the hash of the ``taskcluster/qemu/<image_name>/image.yml``
+   with all referenced file-paths substituted for the hash of the file/folder.
+
+Images will be cached locally in ``$MOZBUILD_STATE_PATH``, which defaults to
+``~/.mozbuild`` if not given. The state of the image cache can be listed with
+``mach qemu images`` which will list images, status and label current images
+with ``*``. A current image, is an image that has an ``<image_hash>`` reflecting
+the current state in the source tree.
+
+Images can be fetched with from S3 or indexed tasks using
+``mach qemu pull <image_name>``, see ``mach qemu help pull`` for additional
+options. By default ``mach qemu pull <image_name>`` will pull an image with
+``<image_hash>`` matching the current state in the source tree, if available.
+
+If an image definition or any file referenced therein, have been modified locally
+it can be rebuilt locally using ``mach qemu build <image_name>``. Obviously,
+it won't be possible to pull the image (as it's never been built before).
+
+When running low on storage cached images can be purged using ``mach qemu purge``.
+
+
+Testing QEMU Images Locally
+===========================
+
+When tweaking and building images locally it is important to test the image, to
+ensure that ``taskcluster-worker qemu-guest-tools`` are installed and started
+correctly. This is obviously also a good way to debug various issues with images.
+
+To run an image use the ``mach qemu run <image_name> -- <command_and_args>``
+command, this will boot the VM and run the given ``<command_and_args>``. It is
+also possible to pass ``--vnc`` to expose the screen as a local VNC server.
+
+The ``mach qemu run <image_name> -- <command_and_args>`` will normally terminate
+the VM and exit as soon as the ``<command_and_args>`` executed inside the VM
+have terminated. Passing ``--keep-alive`` disables this functionality and keeps
+the VM alive until interrupted by a signal (`ctrl + c`). This is useful when
+using the local VNC server, or using interactive shells.
+
+Interactive (ssh-like) shells into the VM can be obtained using
+``mach qemu shell``. This requires the ``taskcluster-worker qemu-guest-tools``
+to be installed and running correctly.
+
+.. note::
+  The ``mach qemu run <image_name>`` command does not support running two VMs
+  at the same time, as it would be too easy to drain the system of resources.
+
+
+Image Index Namespaces
+======================
+
+The decision task will index QEMU image building tasks under the following
+namespaces:
+ * ``gecko.cache.level-{level}.qemu-images.v1.{image_name}.latest``
+ * ``gecko.cache.level-{level}.qemu-images.v1.{image_name}.hash.{image_hash}``
+ * ``gecko.cache.level-{level}.qemu-images.v1.{image_name}.pushdate.{year}.{month}-{day}-{pushtime}``
+
+Where ``level`` is a commit level 1, 2 or 3, ``image_name`` is the name of the
+image and the ``image_hash`` is the image hash.
+
+For details see ``taskcluster/taskgraph/qemu/imagecache.py``.
+
+
+Image Storage in S3
+===================
+
+.. warning::
+  Only images that cannot be automatically built should be uploaded with
+  ``mach qemu push <image_name>``, this is a special case, there will be few
+  such images.
+
+A few images can't be automatically re-built, and must be built locally and
+uploaded to S3. This is notably the case for the image used to build other
+QEMU images.
+
+.. note::
+  At the moment manually built images must be public, private images are not
+  supported.
+
+Once an image is built locally using ``mach qemu build <image_name>`` it can
+be uploaded to S3 using ``mach qemu push <image_name>``. This requires
+taskcluster credentials in environment variables and level 3 commit access,
+in order to have sufficient scopes.
+
+When an image have been pushed using ``mach qemu push <image_name>`` a
+``reference.yml`` file will be written into the image folder. This file must be
+committed and ensures that the decision task will use the image from S3 instead
+of trying to automatically rebuilt it.
new file mode 100644
--- /dev/null
+++ b/taskcluster/qemu/README.md
@@ -0,0 +1,3 @@
+This folder contains definitions for QEMU images used in taskcluster.
+
+See: `taskcluster/docs/qemu-images.rst` for details.