testing: filter docker requirements by what's allowed by virtualenv (
bug 1363509); r?glob
Previously, get_docker_state() would attempt to build all Docker images
required by the set of passed tests. This behavior was fine with a
monolithic test environment. But it is no longer appropriate for
different virtualenvs.
The hgdev virtualenv for example pulls in tests that require BMO,
MozReview, and hg.mo Docker images. However, we don't (yet) wish
to support all these Docker environments in that dev environment.
We add a dict mapping environment name to a set of allowed Docker
requirements. Then we change the function that populates Docker state
to filter the Docker requirements as specified by tests against
what is allowed by the virtualenv. For example, the hgdev virtualenv
will now only build the bmoweb Docker image.
MozReview-Commit-ID: LkFfLwGuDt4
--- a/run-tests
+++ b/run-tests
@@ -197,17 +197,17 @@ if __name__ == '__main__':
os.environ['DOCKER_HOSTNAME'] = docker.docker_hostname
# We build the base BMO images in the test runner because doing it
# from tests would be racey. It is easier to do it here instead of
# complicating code with locks.
#
# But only do this if a test we are running utilizes Docker.
os.environ.update(
- get_docker_state(docker, run_all_tests,
+ get_docker_state(docker, venv_name, run_all_tests,
verbose=verbose,
use_last=options.use_last_images))
# Our custom HGHAVE introduces a check for running Mercurial version.
# This is done by consulting a HGVERSION environment variable.
hg = os.path.join(os.environ['VIRTUAL_ENV'], 'bin', 'hg')
if options.with_hg:
hg = options.with_hg
--- a/testing/vcttesting/testing.py
+++ b/testing/vcttesting/testing.py
@@ -70,16 +70,24 @@ COVERAGE_OMIT = (
'pylib/flake/*',
'pylib/mccabe/*',
'pylib/mercurial-support/*',
'pylib/pycodestyle/*',
'pylib/pyflakes/*',
'pylib/requests/*',
)
+# Maps virtualenv name to allowed Docker requirements.
+# If key not present, all Docker requirements are allowed. A
+# test will be skipped if Docker isn't available or if test
+# requires Docker component not enabled by the virtualenv.
+VIRTUALENV_DOCKER_REQUIREMENTS = {
+ 'hgdev': {'bmo',},
+}
+
def is_test_filename(f):
"""Is a path a test file."""
return f.startswith('test-') and f.endswith(('.py', '.t'))
def get_extensions(extdir):
"""Obtain information about extensions.
@@ -246,28 +254,37 @@ def docker_requirements(tests):
res = set()
for test in tests:
res |= docker_requirements_for_test(test)
return res
-def get_docker_state(docker, tests, verbose=False, use_last=False):
+def get_docker_state(docker, venv_name, tests, verbose=False, use_last=False):
"""Obtain usable Docker images, possibly by building them.
- Given a Docker client and list of .t test paths, determine what
- Docker images are needed to run the tests and then return a dictionary
- of environment variables that define the Docker image IDs.
+ Given a Docker client, name of virtualenv, and list of .t test paths,
+ determine what Docker images are needed/allowed to run the tests and
+ then return a dictionary of environment variables that define the Docker
+ image IDs.
If ``use_last`` is set, existing Docker images will be used. Otherwise,
Docker images are rebuilt to ensure they are up-to-date.
+
+ Only Docker images "allowed" by the specified virtualenv will be built.
+ Not all virtualenvs support all Docker images.
"""
requirements = docker_requirements(tests)
+ # Filter out requirements not specified by the virtualenv.
+ allowed_requirements = VIRTUALENV_DOCKER_REQUIREMENTS.get(venv_name,
+ set(requirements))
+ requirements = requirements & allowed_requirements
+
if not requirements:
return {}
env = {}
print('generating Docker images needed for tests')
t_start = time.time()
mr_images, hgmo_images, bmo_images = docker.build_all_images(
verbose=verbose,