Bug 1417684 - Port client.mk code to run configure to Python; r?build
This is a large patch and there's a lot going on. I'm sorry for anyone
who has to read it.
The build system is effectively a pipeline of stages that complete
one after the other. Here are the stages relevant to this commit:
`mach` -> env config -> configure -> config.status -> build backend
Before this commit, everything after "env config" (finding topsrcdir,
topobjdir, and extracting specific data from an active mozconfig) was
implemented in client.mk. (And until recently pretty much everything
was implemented in client.mk and `mach` effectively ran
`make -f client.mk`.)
The goal of this commit is to move the configure pipeline stage
from client.mk to Python.
The structure of configure is kinda wonky. A lot of that is for
historical reasons.
The "configure" and "js/src/configure" files derive from "configure.in"
files. In the old days, we used autoconf for the derivation. These days,
client.mk just copies the file. (But configure.in is also a valid m4
file, so autoconf can still be used!)
The "configure" files today are very thin shell scripts that invoke
the new Python-based configure: "configure.py."
When configure/configure.py runs, its main output is a "config.status"
file in the objdir. This file - an executable Python script - contains
all the data obtained by running configure.
When "config.status" runs ("configure" will run it automatically
after writing it), it reads moz.build files and generates files
to be consumed by a build backend to build the configured application.
Since we always produce a make backend today, the most visible output
from config.status is a "Makefile" in the objdir.
This commit ports the logic for invoking configure from client.mk
to Python.
When reviewing the code, it may help to look at removals from client.mk
from bottom to top. Here's a summary of what was going on.
client.mk has two primary targets: "build" and "configure."
The recipe for "configure::" is pretty basic: run
$(topsrcdir)/configure from the objdir and touch $(objdir)/Makefile.
Of course, there are a lot of dependencies required to get there. More
on those later.
The relationship between the "build" and "configure" targets is a bit
hard to read (original lines 142-155 in client.mk). But the gist of it
is:
* If Makefile exists, Makefile depends on config.status and
config.status depends on $(CONFIG_STATUS_DEPS).
* If Makefile doesn't exist, Makefile depends on $(CONFIG_STATUS_DEPS).
* Makefile target is produced by running "make -f client.mk configure"
* "build" depends on Makefile and config.status.
Essentially, this is expressing that Makefile is built by
config.status and these both depend on $(CONFIG_STATUS_DEPS).
When "build" is evaluated, both Makefile and config.status must exist
and be up to date. If not, we evaluate the "configure" target of
client.mk in a separate invocation of client.mk (I suspect it isn't
an "internal" target dependency for legacy reasons and to keep things
simple since it avoids issues with phony targets).
The "configure" target itself has a couple of dependencies:
* The configure files themselves (which are copied from configure.in)
* .mozconfig.json (which is derived from the active mozconfig)
The "config.status" target is a bit more interesting. It lists a lot
of random files throughout the repo. Basically things that when changed
have a significant impact on the build configuration.
In addition, the included configure.d file also extended the
dependencies for config.status.
Logically, the "configure" and "config.status" dependencies should
probably be the same. I think the reason they are separate is
historical reasons. If you look at $(EXTRA_CONFIG_DEPS), those are
files related to m4. Those made sense as dependencies when configure
was produced by running autoconf. They should have been removed
when we started producing configure via `cp`. But they weren't.
And, it appears moz.configure isn't tracking these m4 files in its
auto-generated configure.d file. So we need to carry those files
with us in the Python port (I intend to fix this in another commit).
The new Python code is hopefully easier to read.
Because invoking configure represents a distinct stage in the
build pipeline and because building.py was already quite large,
I put the code in a new module.
Because I don't like the class-centric design of the MozbuildObject
base class (I wish I had a time machine to uninvent that "god"
object), I've stepped out of my way to avoid using it explicitly
in the new code.
The new code in configure.py is relatively straightforward. We
have two new high-level APIs: one to ensure configure is up to
date (running it if not) and one to always run configure. There is
definitely some wonkiness. Such as having to set the MAKE
environment variable (we'll fix this later). Otherwise, it is
pretty straightforward with regards to capturing file dependencies
and invoking configure.
There are a few notable changes in behavior.
First, we now pull in all loaded .py modules as dependencies for
configure. This will include python.mozbuild.*, which is desirable.
Less desirable is the inclusion of other modules, like random
mach_commands.py files. I think we'll eventually want to move the
"run configure if necessary" code to a separate process to mitigate
this. We may even want to move the logic to configure itself and
just always invoke configure, having it no-op as appropriate.
Second, since we no longer run the configure process from the context
of client.mk, variables set in client.mk - including from the included
mozconfig-derived make files - are no longer set in the configure
environment. I think this is fine. configure has logic for locating
the mozconfig that matches what mach is doing. So I'm pretty sure
the end result is the same. The only variable we do need to set
though is MAKE. That's because moz.configure's make detection logic
is poor. This will be improved in subsequent commits and the MAKE
hack will go away.
The new code documents a handful of known areas for improvement.
I don't think any of these are show stoppers. Perfect is the
enemy of good. Subsequent commits will improve matters
significantly...
MozReview-Commit-ID: ZCpTdLFur3
--- a/client.mk
+++ b/client.mk
@@ -11,26 +11,16 @@
#
# Options:
# MOZ_OBJDIR - Destination object directory
# MOZ_MAKE_FLAGS - Flags to pass to $(MAKE)
#
#######################################################################
# Defines
-ifdef MACH
-ifndef NO_BUILDSTATUS_MESSAGES
-define BUILDSTATUS
-@echo 'BUILDSTATUS $1'
-
-endef
-endif
-endif
-
-
CWD := $(CURDIR)
ifeq "$(CWD)" "/"
CWD := /.
endif
PYTHON ?= $(shell which python2.7 > /dev/null 2>&1 && echo python2.7 || echo python)
@@ -53,20 +43,16 @@ endif
ifdef MOZ_AUTOMATION
ifeq (4.0,$(firstword $(sort 4.0 $(MAKE_VERSION))))
MOZ_MAKE_FLAGS += --output-sync=line
endif
endif
MOZ_MAKE = $(MAKE) $(MOZ_MAKE_FLAGS) -C $(OBJDIR)
-# 'configure' scripts generated by autoconf.
-CONFIGURES := $(TOPSRCDIR)/configure
-CONFIGURES += $(TOPSRCDIR)/js/src/configure
-
#######################################################################
# Rules
# The default rule is build
build::
ifndef MACH
$(error client.mk must be used via `mach`. Try running \
@@ -81,95 +67,19 @@ build::
-$(MOZBUILD_MANAGE_SCCACHE_DAEMON) --stop-server > /dev/null 2>&1
# Start a new server, ensuring it gets the jobserver file descriptors
# from make (but don't use the + prefix when make -n is used, so that
# the command doesn't run in that case)
$(if $(findstring n,$(filter-out --%, $(MAKEFLAGS))),,+)env RUST_LOG=sccache::compiler=debug SCCACHE_ERROR_LOG=$(OBJDIR)/dist/sccache.log $(MOZBUILD_MANAGE_SCCACHE_DAEMON) --start-server
endif
####################################
-# Configure
-
-MAKEFILE = $(wildcard $(OBJDIR)/Makefile)
-CONFIG_STATUS = $(wildcard $(OBJDIR)/config.status)
-
-EXTRA_CONFIG_DEPS := \
- $(TOPSRCDIR)/aclocal.m4 \
- $(TOPSRCDIR)/old-configure.in \
- $(wildcard $(TOPSRCDIR)/build/autoconf/*.m4) \
- $(TOPSRCDIR)/js/src/aclocal.m4 \
- $(TOPSRCDIR)/js/src/old-configure.in \
- $(NULL)
-
-$(CONFIGURES): %: %.in $(EXTRA_CONFIG_DEPS)
- @echo Generating $@
- cp -f $< $@
- chmod +x $@
-
-CONFIG_STATUS_DEPS := \
- $(wildcard $(TOPSRCDIR)/*/confvars.sh) \
- $(CONFIGURES) \
- $(TOPSRCDIR)/nsprpub/configure \
- $(TOPSRCDIR)/config/milestone.txt \
- $(TOPSRCDIR)/browser/config/version.txt \
- $(TOPSRCDIR)/browser/config/version_display.txt \
- $(TOPSRCDIR)/build/virtualenv_packages.txt \
- $(TOPSRCDIR)/python/mozbuild/mozbuild/virtualenv.py \
- $(TOPSRCDIR)/testing/mozbase/packages.txt \
- $(OBJDIR)/.mozconfig.json \
- $(NULL)
-
-# Include a dep file emitted by configure to track Python files that
-# may influence the result of configure.
--include $(OBJDIR)/configure.d
-
-CONFIGURE_ENV_ARGS += \
- MAKE='$(MAKE)' \
- $(NULL)
-
-# configure uses the program name to determine @srcdir@. Calling it without
-# $(TOPSRCDIR) will set @srcdir@ to "."; otherwise, it is set to the full
-# path of $(TOPSRCDIR).
-ifeq ($(TOPSRCDIR),$(OBJDIR))
- CONFIGURE = ./configure
-else
- CONFIGURE = $(TOPSRCDIR)/configure
-endif
-
-configure-files: $(CONFIGURES)
-
-configure-preqs = \
- configure-files \
- $(OBJDIR)/.mozconfig.json \
- $(NULL)
-
-configure:: $(configure-preqs)
- $(call BUILDSTATUS,TIERS configure)
- $(call BUILDSTATUS,TIER_START configure)
- @echo cd $(OBJDIR);
- @echo $(CONFIGURE) $(CONFIGURE_ARGS)
- @cd $(OBJDIR) && $(CONFIGURE_ENV_ARGS) $(CONFIGURE) $(CONFIGURE_ARGS) \
- || ( echo '*** Fix above errors and then restart with\
- "$(MAKE) -f client.mk build"' && exit 1 )
- @touch $(OBJDIR)/Makefile
- $(call BUILDSTATUS,TIER_FINISH configure)
-
-ifneq (,$(MAKEFILE))
-$(OBJDIR)/Makefile: $(OBJDIR)/config.status
-
-$(OBJDIR)/config.status: $(CONFIG_STATUS_DEPS)
-else
-$(OBJDIR)/Makefile: $(CONFIG_STATUS_DEPS)
-endif
- @$(MAKE) -f $(TOPSRCDIR)/client.mk configure
-
-####################################
# Build it
-build:: $(OBJDIR)/Makefile $(OBJDIR)/config.status
+build::
+$(MOZ_MAKE)
ifdef MOZ_AUTOMATION
build::
+$(MOZ_MAKE) automation/build
endif
ifdef MOZBUILD_MANAGE_SCCACHE_DAEMON
@@ -179,10 +89,9 @@ build::
endif
# This makefile doesn't support parallel execution. It does pass
# MOZ_MAKE_FLAGS to sub-make processes, so they will correctly execute
# in parallel.
.NOTPARALLEL:
.PHONY: \
- build \
- configure
+ build
--- a/python/mozbuild/mozbuild/controller/building.py
+++ b/python/mozbuild/mozbuild/controller/building.py
@@ -32,33 +32,34 @@ except Exception:
from mach.mixin.logging import LoggingMixin
from mozsystemmonitor.resourcemonitor import SystemResourceMonitor
import mozpack.path as mozpath
from .clobber import (
Clobberer,
)
+from .configure import (
+ ensure_configure,
+ run_configure,
+)
from ..base import (
BuildEnvironmentNotFoundException,
MozbuildObject,
)
from ..backend import (
get_backend_class,
)
from ..testing import (
install_test_files,
)
from ..compilation.warnings import (
WarningsCollector,
WarningsDatabase,
)
-from ..shellutil import (
- quote as shell_quote,
-)
from ..util import (
FileAvoidWrite,
mkdir,
resolve_target_to_make,
)
FINDER_SLOW_MESSAGE = '''
@@ -1090,21 +1091,27 @@ class BuildDriver(MozbuildObject):
keep_going=keep_going)
if status != 0:
break
else:
# Try to call the default backend's build() method. This will
# run configure to determine BUILD_BACKENDS if it hasn't run
# yet.
+ # TODO we should probably call ensure_configure() here or
+ # somewhere above.
try:
config = self.config_environment
except Exception:
- config_rc = self.configure(buildstatus_messages=True,
- line_handler=output.on_line)
+ config_rc = run_configure(self.topsrcdir,
+ self.topobjdir,
+ self._make_path(),
+ self.run_process,
+ line_handler=output.on_line)
+
if config_rc != 0:
return config_rc
config = self.config_environment
active_backend = config.substs.get('BUILD_BACKENDS', [None])[0]
if active_backend:
backend_cls = get_backend_class(active_backend)(config)
@@ -1276,26 +1283,23 @@ class BuildDriver(MozbuildObject):
if self._prepare_objdir():
return 1
def on_line(line):
self.log(logging.INFO, 'build_output', {'line': line}, '{line}')
line_handler = line_handler or on_line
- options = ' '.join(shell_quote(o) for o in options or ())
- append_env = {b'CONFIGURE_ARGS': options.encode('utf-8')}
+ status = run_configure(self.topsrcdir, self.topobjdir,
+ self._make_path(), self.run_process,
+ options=options, line_handler=line_handler,
+ buildstatus_messages=buildstatus_messages)
- # Only print build status messages when we have an active
- # monitor.
- if not buildstatus_messages:
- append_env[b'NO_BUILDSTATUS_MESSAGES'] = b'1'
- status = self._run_client_mk(target='configure',
- line_handler=line_handler,
- append_env=append_env)
+ # Clear out cached config.status state.
+ self._config_environment = None
if not status:
print('Configure complete!')
print('Be sure to run |mach build| to pick up any changes');
return status
def install_tests(self, test_objs):
@@ -1388,16 +1392,28 @@ class BuildDriver(MozbuildObject):
def _run_client_mk(self, target=None, line_handler=None, jobs=0,
verbose=None, keep_going=False, append_env=None):
append_env = dict(append_env or {})
append_env['TOPSRCDIR'] = self.topsrcdir
append_env['CONFIG_GUESS'] = self.resolve_config_guess()
append_env['OBJDIR'] = mozpath.normsep(self.topobjdir)
+ res = ensure_configure(self.topsrcdir, self.topobjdir, self.log,
+ self._make_path(), self.run_process,
+ line_handler)
+ configure_ran, configure_exit = res
+
+ # Clear cached config.status state.
+ if configure_ran:
+ self._config_environment = None
+
+ if configure_exit:
+ return configure_exit
+
return self._run_make(srcdir=True,
filename='client.mk',
allow_parallel=False,
ensure_exit_code=False,
print_directory=False,
target=target,
line_handler=line_handler,
log=False,
new file mode 100644
--- /dev/null
+++ b/python/mozbuild/mozbuild/controller/configure.py
@@ -0,0 +1,222 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+# Functionality for evaluating configure.
+
+from __future__ import absolute_import, unicode_literals
+
+import glob
+import logging
+import os
+import stat
+import time
+
+import mozpack.path as mozpath
+
+from ..dependsutils import (
+ is_file_target_current,
+)
+from ..makeutil import (
+ read_dep_makefile,
+)
+from ..pythonutil import (
+ iter_modules_in_path,
+)
+
+
+CONFIGURE_DIRS = (
+ '.',
+ 'js/src',
+)
+
+CONFIGURE_DEPENDENCIES = (
+ 'aclocal.m4',
+ 'old-configure.in',
+ 'build/autoconf/*.m4',
+ 'js/src/aclocal.m4',
+ 'js/src/old-configure.in',
+)
+
+CONFIG_STATUS_DEPENDENCIES = (
+ '*/confvars.sh',
+ 'nsprpub/configure',
+ 'config/milestone.txt',
+ 'browser/config/version.txt',
+ 'browser/config/version_display.txt',
+ 'build/virtualenv_packages.txt',
+ 'python/mozbuild/mozbuild/virtualenv.py',
+ 'testing/mozbase/packages.txt',
+)
+
+CONFIG_STATUS_OBJDIR_DEPENDENCIES = (
+ '.mozconfig.json',
+)
+
+
+def _ensure_configure_files(topsrcdir, log):
+ """Ensures configure files are up to date."""
+
+ # configure files are produced by copying configure.in files. This is a bit
+ # of an historical kludge due to the fact that people may still run
+ # autoconf directly. We should unsupport this someday, possibly by
+ # moving configure.in out of the root directory.
+
+ # We require configure files be newer than some other (mostly m4
+ # related) files in the repo. These file dependencies should be captured
+ # against config.status since moz.configure is the thing consuming these
+ # files. However, the make backend treats them as part of configure's
+ # dependencies. For now, we preserve the legacy behavior.
+ depends = set()
+ for pattern in CONFIGURE_DEPENDENCIES:
+ if '*' in pattern:
+ for p in glob.iglob(mozpath.join(topsrcdir, pattern)):
+ depends.add(mozpath.normpath(p))
+ else:
+ depends.add(mozpath.join(topsrcdir, pattern))
+
+ configure_paths = []
+
+ for d in CONFIGURE_DIRS:
+ out_path = mozpath.normpath(mozpath.join(topsrcdir, d, 'configure'))
+ in_path = '%s.in' % out_path
+
+ if not is_file_target_current([out_path], list(depends) + [in_path]):
+ log(logging.INFO, 'configure',
+ {'msg': 'Generating %s' % out_path}, '{msg}')
+ with open(in_path, 'rb') as ifh, open(out_path, 'wb') as ofh:
+ ofh.write(ifh.read())
+
+ old_mode = os.stat(in_path).st_mode
+ os.chmod(out_path, old_mode | stat.S_IXUSR)
+
+ configure_paths.append(out_path)
+
+ return configure_paths
+
+
+def _configure_needed(topsrcdir, topobjdir, log, configures):
+ """Determine whether configure needs to run.
+
+ Once run, the config.status file is current given the current build
+ configuration.
+ """
+ # The main output of configure is config.status. Running config.status
+ # produces Makefile. So if either is missing, we need to run configure.
+ # Handle this special case as a fast path here.
+ for p in ('config.status', 'Makefile'):
+ if not os.path.exists(os.path.join(topobjdir, p)):
+ log(logging.INFO, 'configure',
+ {'msg': '%s missing; configure needed' % p}, '{msg}')
+ return True
+
+ # Now assemble all the dependencies that influence whether we need
+ # to run configure again.
+ depends = set()
+
+ # Configure depends on the configure scripts themselves.
+ depends |= set(configures)
+
+ # And on various files in the source directory.
+ for pattern in CONFIG_STATUS_DEPENDENCIES:
+ if '*' in pattern:
+ for p in glob.iglob(mozpath.join(topsrcdir, pattern)):
+ depends.add(mozpath.normpath(p))
+ else:
+ depends.add(mozpath.join(topsrcdir, pattern))
+
+ # And on various files in the object directory.
+ for p in CONFIG_STATUS_OBJDIR_DEPENDENCIES:
+ depends.add(mozpath.join(topobjdir, p))
+
+ # Configure is also kind enough to record the set of files it loaded when
+ # running. Pull those in as well.
+ #
+ # We could likely fold the files from above into this auto-generated file...
+ config_status = mozpath.join(topobjdir.encode('utf-8'), 'config.status')
+ with open(mozpath.join(topobjdir, 'configure.d'), 'rb') as fh:
+ target_found = False
+ for rule in read_dep_makefile(fh):
+ for target in rule.targets():
+ target = mozpath.join(target)
+ if target == config_status:
+ depends |= set(mozpath.normpath(p) for p in
+ rule.dependencies())
+ target_found = True
+
+ if not target_found:
+ raise Exception('could not find %s target in configure.d' %
+ config_status)
+
+ # And pull in Python files from this process (since any change to this
+ # code could invalidate configure results). In the context of mach,
+ # this could be a large number of files, many of them irrelevant. So
+ # we may want to split this into its own process to avoid dependency
+ # tainting.
+ depends |= set(iter_modules_in_path(topsrcdir))
+
+ # The build backend must be strictly newer than all of configure's
+ # dependencies.
+ makefile = os.path.join(topobjdir, 'Makefile')
+
+ return not is_file_target_current([makefile], list(depends))
+
+
+def ensure_configure(topsrcdir, topobjdir, log, make, run_process,
+ line_handler):
+ """Ensure configure is up to date, running if necessary.
+
+ Returns a 2-tuple of (bool whether ran, int exit code).
+ """
+ # Process configure.in files.
+ configures = _ensure_configure_files(topsrcdir, log)
+
+ if not _configure_needed(topsrcdir, topobjdir, log, configures):
+ log(logging.INFO, 'configure',
+ {'msg': 'configure and build backend up to date'},
+ '{msg}')
+ return False, None
+
+ return True, run_configure(topsrcdir, topobjdir, make, run_process,
+ line_handler=line_handler)
+
+
+def run_configure(topsrcdir, topobjdir, make, run_process, options=None,
+ line_handler=None, buildstatus_messages=True):
+ """Run configure, even if it doesn't need to run.
+
+ Returns the integer exit code of configure.
+ """
+ log_name = None if line_handler else 'configure'
+
+ options = options or []
+ configure = mozpath.join(topsrcdir, 'configure')
+ command = [configure] + options
+
+ append_env = {
+ # This is a holdover from client.mk. Remove it once configure's
+ # make detection is reasonable.
+ 'MAKE': ' '.join(make),
+ }
+
+ if buildstatus_messages and line_handler:
+ line_handler('BUILDSTATUS TIERS configure')
+ line_handler('BUILDSTATUS TIER_START configure')
+
+ res = run_process(args=command,
+ cwd=topobjdir,
+ append_env=append_env,
+ line_handler=line_handler,
+ log_name=log_name,
+ ensure_exit_code=False)
+
+ if buildstatus_messages and line_handler:
+ line_handler('BUILDSTATUS TIER_FINISH configure')
+
+ # This is a holdover from client.mk. It should be done from within
+ # configure/config.status itself.
+ if not res:
+ now = time.time()
+ os.utime(os.path.join(topobjdir, 'Makefile'), (now, now))
+
+ return res