Bug 1439742 - Allow {AB_CD} and {AB_rCD} in LOCALIZED_GENERATED_FILES. r=ted.mielczarek draft
authorNick Alexander <nalexander@mozilla.com>
Wed, 21 Feb 2018 17:12:17 -0800
changeset 760448 816b6f220758f2bb3bdd3ec81a2cb02269c6de5b
parent 760447 3277fedb43bc3d8007287c223554a085dae2f198
child 760520 4421cd473584660bcad8f513004bd605a102dd96
push id100646
push usernalexander@mozilla.com
push dateTue, 27 Feb 2018 17:06:59 +0000
reviewersted.mielczarek
bugs1439742
milestone60.0a1
Bug 1439742 - Allow {AB_CD} and {AB_rCD} in LOCALIZED_GENERATED_FILES. r=ted.mielczarek There are a lot of choices and moving pieces in this commit. I elected to include the mechanics and the target use case in the same commit so that readers can compare and contrast the implementation and final expression in one review window. - Initially, I wanted to make the {AB_CD} substitutions in LOCALIZED_FILES and not in LOCALIZED_GENERATED_FILES. However, I ran into conceptual blockers doing this. Fundamentally, LOCALIZED_FILES is FINAL_TARGET_FILES, and my use case should _not_ be putting files anywhere near dist/bin. In addition, LOCALIZED_FILES (FINAL_TARGET_FILES) is handled using manifests, which would need to grow locale-aware functionality to handle this. That's not desirable. In addition, if we use manifests, then we lose the powerful locality of |mach build mobile/android{/base}| re-generating changed locale-dependent resources. This is similar to how the build system plumbs dist/idl manifest processing throughout the build: we're repairing local workflows after moving work into a global process. For these reasons, this doesn't support {AB_CD} in LOCALIZED_FILES. - There is even another layer of complexity! There are two axes involved with these files: AB_CD controls localization and the Make target controls destination. For the record, it is: regular builds - AB_CD unset multi-locale builds - AB_CD set single-locale repacks - AB_CD set For the record, the existing logic (before any changes) is: regular builds - Make target is `libs` in mobile/android/base/locales multi-locale builds - Make target is `chrome-%` in mobile/android/base/locales single-locale repacks - Make target is `libs` in mobile/android/base/locales This commit adds targets for both destinations, and uses Make chrome-%:: and libs:: magic to control what is invoked in the various situations. Tricky! - I added MERGE_RELATIVE_FILES in order to be able to follow-up this patch with more patches that will get rid of m/a/base/locales/{moz.build,Makefile.in} altogether, and fold this work into m/a/base. As it stands, we're already reaching from m/a/base/locales all the way out to mobile/locales/.../region.properties, so the existing code doesn't follow the layout expected between mozilla-central and l10n-central/$(AB_CD). But that'll impedance will get worse as we improve the build system dependencies, not better, so we should grow support for localized resources that aren't exactly as expected. - I chose to follow Python's syntax for string substitutions. I would have preferred to mark files that should be localized with a leading '%'... but I took that for filesystem absolute paths in moz.build files already. I also considered @AB_CD@ to echo the preprocessor, but didn't want to open the door to an expecation that _all_ preprocessor DEFINEs will work in the way {AB_CD} does. - The generate_*py script changes required a bit of a hack to "turn off" locale dependent resources. This would have been nicer if we had marked localized resources with '%'... but we didn't. See the --fallback flag. The real reason this is needed is that we're doing work which is more like the work of compare-locales (merging locale-dependent resources) at build-time rather than repack time. I don't know why that's the case -- probably when we (I) implemented it, compare-locales and the whole l10n process was entirely opaque. It's not worth changing it now, so we use this --fallback flag approach. - I didn't get to tup support. This should gently fail without breaking tup builds: any {AB_CD} substitutions just won't be expanded. I haven't a clue how this should work in tup in the future (or, more generally, how to make any sense of repacks without declaring the full set of expected locales at configure time.) - strings.xml can't be a LOCALIZED_PP_FILES, since we need to customize the output location based on AB_rCD, and since we need a little more flexibility than PP_FILES gives for our inputs. MozReview-Commit-ID: MyfIkNSEzt
config/config.mk
mobile/android/base/locales/Makefile.in
mobile/android/base/locales/moz.build
python/mozbuild/mozbuild/action/generate_browsersearch.py
python/mozbuild/mozbuild/action/generate_strings_xml.py
python/mozbuild/mozbuild/action/generate_suggestedsites.py
python/mozbuild/mozbuild/backend/recursivemake.py
python/mozbuild/mozbuild/frontend/context.py
python/mozbuild/mozbuild/frontend/emitter.py
python/mozbuild/mozbuild/test/action/test_generate_browsersearch.py
python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/en-US/localized-input
python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/foo-data
python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/generate-foo.py
python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/inner/locales/en-US/localized-input
python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/moz.build
python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/non-localized-input
python/mozbuild/mozbuild/test/backend/test_recursivemake.py
python/mozbuild/mozbuild/test/frontend/data/localized-files-no-en-us/inner/locales/en-US/bar.ini
python/mozbuild/mozbuild/test/frontend/data/localized-files-no-en-us/moz.build
python/mozbuild/mozbuild/test/frontend/test_emitter.py
--- a/config/config.mk
+++ b/config/config.mk
@@ -374,18 +374,30 @@ else
 MAKE_JARS_FLAGS += -c $(LOCALE_SRCDIR)
 endif # ! relativesrcdir
 
 ifdef IS_LANGUAGE_REPACK
 MERGE_FILE = $(firstword \
   $(wildcard $(REAL_LOCALE_MERGEDIR)/$(subst /locales,,$(LOCALE_RELATIVEDIR))/$(1)) \
   $(wildcard $(LOCALE_SRCDIR)/$(1)) \
   $(srcdir)/en-US/$(1) )
+# Like MERGE_FILE, but with the specified relative source directory
+# $(2) replacing $(srcdir).  It's expected that $(2) will include
+# '/locales' but not '/locales/en-US'.
+#
+# MERGE_RELATIVE_FILE and MERGE_FILE could be -- ahem -- merged by
+# making the second argument optional, but that expression makes for
+# difficult to read Make.
+MERGE_RELATIVE_FILE = $(firstword \
+  $(wildcard $(REAL_LOCALE_MERGEDIR)/$(subst /locales,,$(2))/$(1)) \
+  $(wildcard $(call EXPAND_LOCALE_SRCDIR,$(2))/$(1)) \
+  $(topsrcdir)/$(2)/en-US/$(1) )
 else
 MERGE_FILE = $(LOCALE_SRCDIR)/$(1)
+MERGE_RELATIVE_FILE = $(call EXPAND_LOCALE_SRCDIR,$(2))/$(1)
 endif
 
 ifneq (WINNT,$(OS_ARCH))
 RUN_TEST_PROGRAM = $(DIST)/bin/run-mozilla.sh
 endif # ! WINNT
 
 #
 # Java macros
--- a/mobile/android/base/locales/Makefile.in
+++ b/mobile/android/base/locales/Makefile.in
@@ -1,100 +1,13 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 include $(topsrcdir)/config/AB_rCD.mk
 
-SYNCSTRINGSPATH = $(abspath $(call MERGE_FILE,sync_strings.dtd))
-STRINGSPATH = $(abspath $(call MERGE_FILE,android_strings.dtd))
-# Fennec branding is en-US only: see $(MOZ_BRANDING_DIRECTORY)/locales/jar.mn.
-BRANDPATH := $(topsrcdir)/$(MOZ_BRANDING_DIRECTORY)/locales/en-US/brand.dtd
-$(warnIfEmpty,AB_CD) # todo: $(errorIfEmpty )
-
-dir-res-values := ../res/values
-strings-xml    := $(dir-res-values)/strings.xml
-strings-xml-in := $(srcdir)/../strings.xml.in
-
-GARBAGE += $(strings-xml)
-
-dir-res-raw := ../res/raw
-suggestedsites := $(dir-res-raw)/suggestedsites.json
-browsersearch := $(dir-res-raw)/browsersearch.json
-
-libs realchrome:: \
-  $(strings-xml) \
-  $(NULL)
-
 chrome-%:: AB_CD=$*
 chrome-%::
 	@$(MAKE) \
 	  ../res/values$(AB_rCD)/strings.xml \
 	  ../res/raw$(AB_rCD)/suggestedsites.json \
 	  ../res/raw$(AB_rCD)/browsersearch.json \
 	  AB_CD=$*
-
-# Determine the ../res/values[-*]/ path
-strings-xml-bypath  = $(filter %/strings.xml,$(MAKECMDGOALS))
-ifeq (,$(strip $(strings-xml-bypath)))
-  strings-xml-bypath = $(strings-xml)
-endif
-dir-strings-xml = $(patsubst %/,%,$(dir $(strings-xml-bypath)))
-
-strings-xml-preqs =\
-  $(strings-xml-in) \
-  $(BRANDPATH) \
-  $(STRINGSPATH) \
-  $(SYNCSTRINGSPATH) \
-  $(if $(IS_LANGUAGE_REPACK),FORCE) \
-  $(NULL)
-
-$(dir-strings-xml)/strings.xml: $(strings-xml-preqs)
-	$(call py_action,preprocessor, \
-      $(DEFINES) \
-      $(ACDEFINES) \
-	  -DANDROID_PACKAGE_NAME=$(ANDROID_PACKAGE_NAME) \
-	  -DBRANDPATH='$(BRANDPATH)' \
-	  -DMOZ_APP_DISPLAYNAME='@MOZ_APP_DISPLAYNAME@' \
-	  -DSTRINGSPATH='$(STRINGSPATH)' \
-	  -DSYNCSTRINGSPATH='$(SYNCSTRINGSPATH)' \
-      $< \
-	  -o $@)
-
-# Arg 1: Valid Make identifier, like suggestedsites.
-# Arg 2: File name, like suggestedsites.json.
-define generated_file_template
-
-# Determine the ../res/raw[-*] path.  This can be ../res/raw when no
-# locale is explicitly specified.
-$(1)-bypath = $(filter %/$(2),$(MAKECMDGOALS))
-ifeq (,$$(strip $$($(1)-bypath)))
-  $(1)-bypath = $($(1))
-endif
-$(1)-dstdir-raw = $$(patsubst %/,%,$$(dir $$($(1)-bypath)))
-
-GARBAGE += $($(1))
-
-libs realchrome:: $($(1))
-endef
-
-# L10NBASEDIR is not defined for en-US.
-l10n-srcdir := $(if $(filter en-US,$(AB_CD)),,$(or $(realpath $(L10NBASEDIR)),$(abspath $(L10NBASEDIR)))/$(AB_CD)/mobile/chrome)
-
-$(eval $(call generated_file_template,suggestedsites,suggestedsites.json))
-
-$(suggestedsites-dstdir-raw)/suggestedsites.json: FORCE
-	$(call py_action,generate_suggestedsites, \
-		--verbose \
-		--android-package-name=$(ANDROID_PACKAGE_NAME) \
-		--resources=$(topsrcdir)/mobile/android/app/src/photon/res \
-		$(if $(filter en-US,$(AB_CD)),,--srcdir=$(l10n-srcdir)) \
-		--srcdir=$(topsrcdir)/mobile/locales/en-US/chrome \
-		$@)
-
-$(eval $(call generated_file_template,browsersearch,browsersearch.json))
-
-$(browsersearch-dstdir-raw)/browsersearch.json: FORCE
-	$(call py_action,generate_browsersearch, \
-		--verbose \
-		$(if $(filter en-US,$(AB_CD)),,--srcdir=$(l10n-srcdir)) \
-		--srcdir=$(topsrcdir)/mobile/locales/en-US/chrome \
-		$@)
--- a/mobile/android/base/locales/moz.build
+++ b/mobile/android/base/locales/moz.build
@@ -1,5 +1,68 @@
 # -*- Mode: python; indent-tabs-mode: nil; tab-width: 40 -*-
 # vim: set filetype=python:
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+# Regular builds invoke `libs` targets that localize files with no AB_CD set
+# into the default resources (res/{values,raw}).
+#
+# Multi-locale builds invoke `chrome-%` targets that localize files into
+# locale-specific resources (res/{values,raw}-AB-rCD).  Single-locale repacks
+# invoke `libs AB_CD=$*` targets that localize files into the default resources
+# (res/{values,raw}).
+#
+# Therefore, every localized generated file needs to be written into both the
+# default resources (res/{values,raw}) and the locale-specific resources
+# (res/{values,raw}-AB-rCD), depending on Make target magic controlled in
+# mobile/android/base (for regular builds) and in mobile/android/locales (for
+# multi-locale builds and single-locale repacks).  Make target magic ensures
+# that res/{values,raw}-AB-rCD is not written for regular en-US builds.
+
+for f in ['../res/raw/browsersearch.json',
+          '../res/raw{AB_rCD}/browsersearch.json']:
+    LOCALIZED_GENERATED_FILES += [f]
+    browsersearch = LOCALIZED_GENERATED_FILES[f]
+    browsersearch.script = '/python/mozbuild/mozbuild/action/generate_browsersearch.py'
+    browsersearch.inputs = [
+        # The `locales/en-US/` in this path will be rewritten to the
+        # locale-specific path.
+        '/mobile/locales/en-US/chrome/region.properties',
+    ]
+    browsersearch.flags += [
+        '--verbose',
+        '--fallback',
+        # The `locales/en-US` in this path will not be rewritten.
+        TOPSRCDIR + '/mobile/locales/en-US/chrome/region.properties',
+    ]
+
+for f in ['../res/raw/suggestedsites.json',
+          '../res/raw{AB_rCD}/suggestedsites.json']:
+    LOCALIZED_GENERATED_FILES += [f]
+    suggestedsites = LOCALIZED_GENERATED_FILES[f]
+    suggestedsites.script = '/python/mozbuild/mozbuild/action/generate_suggestedsites.py'
+    suggestedsites.inputs = [
+        # The `locales/en-US/` in this path will be rewritten to the
+        # locale-specific path.
+        '/mobile/locales/en-US/chrome/region.properties',
+    ]
+    suggestedsites.flags += [
+        '--verbose',
+        '--android-package-name', CONFIG['ANDROID_PACKAGE_NAME'],
+        '--resources', TOPSRCDIR + '/mobile/android/app/src/photon/res',
+        '--fallback',
+        # The `locales/en-US` in this path will not be rewritten.
+        TOPSRCDIR + '/mobile/locales/en-US/chrome/region.properties',
+    ]
+
+for f in ['../res/values/strings.xml',
+          '../res/values{AB_rCD}/strings.xml']:
+    LOCALIZED_GENERATED_FILES += [f]
+    strings = LOCALIZED_GENERATED_FILES[f]
+    strings.script = '/python/mozbuild/mozbuild/action/generate_strings_xml.py'
+    strings.inputs = [
+        '../strings.xml.in',
+        # The `en-US/` will be rewritten to the locale-specific path.
+        'en-US/android_strings.dtd',
+        'en-US/sync_strings.dtd',
+    ]
--- a/python/mozbuild/mozbuild/action/generate_browsersearch.py
+++ b/python/mozbuild/mozbuild/action/generate_browsersearch.py
@@ -3,19 +3,19 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 '''
 Script to generate the browsersearch.json file for Fennec.
 
 This script follows these steps:
 
-1. Read the region.properties file in all the given source directories (see
-srcdir option). Merge all properties into a single dict accounting for the
-priority of source directories.
+1. Read all the given region.properties files (see inputs and --fallback
+options). Merge all properties into a single dict accounting for the priority
+of inputs.
 
 2. Read the default search plugin from 'browser.search.defaultenginename'.
 
 3. Read the list of search plugins from the 'browser.search.order.INDEX'
 properties with values identifying particular search plugins by name.
 
 4. Read each region-specific default search plugin from each property named like
 'browser.search.defaultenginename.REGION'.
@@ -33,62 +33,76 @@ e.g. raw/browsersearch.json, raw-pt-rBR/
 from __future__ import (
     absolute_import,
     print_function,
     unicode_literals,
 )
 
 import argparse
 import codecs
+import errno
 import json
 import sys
 import os
 
 from mozbuild.dotproperties import (
     DotProperties,
 )
 from mozbuild.util import (
     FileAvoidWrite,
 )
 import mozpack.path as mozpath
 
 
-def merge_properties(filename, srcdirs):
-    """Merges properties from the given file in the given source directories."""
+def merge_properties(paths):
+    """Merges properties from the given paths."""
     properties = DotProperties()
-    for srcdir in srcdirs:
-        path = mozpath.join(srcdir, filename)
+    for path in paths:
         try:
             properties.update(path)
-        except IOError:
-            # Ignore non-existing files
-            continue
+        except IOError as e:
+            if e.errno != errno.ENOENT:
+                raise e
     return properties
 
 
-def main(args):
+def main(output, *args, **kwargs):
     parser = argparse.ArgumentParser()
     parser.add_argument('--verbose', '-v', default=False, action='store_true',
                         help='be verbose')
     parser.add_argument('--silent', '-s', default=False, action='store_true',
                         help='be silent')
-    parser.add_argument('--srcdir', metavar='SRCDIR',
-                        action='append', required=True,
-                        help='directories to read inputs from, in order of priority')
-    parser.add_argument('output', metavar='OUTPUT',
-                        help='output')
+    parser.add_argument('inputs', metavar='INPUT', nargs='*',
+                        help='inputs, in order of priority')
+    # This awkward "extra input" is an expedient work-around for an issue with
+    # the build system.  It allows to specify the in-tree unlocalized version
+    # of a resource where a regular input will always be the localized version.
+    parser.add_argument('--fallback',
+                        required=True,
+                        help='fallback input')
     opts = parser.parse_args(args)
 
-    # Use reversed order so that the first srcdir has higher priority to override keys.
-    properties = merge_properties('region.properties', reversed(opts.srcdir))
+    # Ensure the fallback is valid.
+    if not os.path.isfile(opts.fallback):
+        print('Fallback path {fallback} is not a file!'.format(fallback=opts.fallback))
+        return 1
+
+    # Use reversed order so that the first input has higher priority to override keys.
+    sources = [opts.fallback] + list(reversed(opts.inputs))
+    properties = merge_properties(sources)
 
     # Default, not region-specific.
     default = properties.get('browser.search.defaultenginename')
     engines = properties.get_list('browser.search.order')
 
+    # We must define at least one engine -- that's what the fallback is for.
+    if not engines:
+        print('No engines defined: searched in {}!'.format(sources))
+        return 1
+
     writer = codecs.getwriter('utf-8')(sys.stdout)
     if opts.verbose:
         print('Read {len} engines: {engines}'.format(len=len(engines), engines=engines), file=writer)
         print("Default engine is '{default}'.".format(default=default), file=writer)
 
     browsersearch = {}
     browsersearch['default'] = default
     browsersearch['engines'] = engines
@@ -107,25 +121,18 @@ def main(args):
             print("Region '{region}': Default engine is '{region_default}'.".format(
                 region=region, region_default=region_default), file=writer)
 
         browsersearch['regions'][region] = {
             'default': region_default,
             'engines': region_engines,
         }
 
-    # FileAvoidWrite creates its parent directories.
-    output = os.path.abspath(opts.output)
-    fh = FileAvoidWrite(output)
-    json.dump(browsersearch, fh)
-    existed, updated = fh.close()
+    json.dump(browsersearch, output)
+    existed, updated = output.close()  # It's safe to close a FileAvoidWrite multiple times.
 
     if not opts.silent:
         if updated:
-            print('{output} updated'.format(output=output))
+            print('{output} updated'.format(output=os.path.abspath(output.name)))
         else:
-            print('{output} already up-to-date'.format(output=output))
+            print('{output} already up-to-date'.format(output=os.path.abspath(output.name)))
 
-    return 0
-
-
-if __name__ == '__main__':
-    sys.exit(main(sys.argv[1:]))
+    return set([opts.fallback])
new file mode 100644
--- /dev/null
+++ b/python/mozbuild/mozbuild/action/generate_strings_xml.py
@@ -0,0 +1,38 @@
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+from __future__ import absolute_import
+
+import sys
+
+import buildconfig
+
+from mozbuild import preprocessor
+
+
+def main(output, strings_xml, android_strings_dtd, sync_strings_dtd, locale=None):
+    if not locale:
+        raise ValueError('locale must be specified!')
+
+    CONFIG = buildconfig.substs
+
+    defines = {}
+    defines['ANDROID_PACKAGE_NAME'] = CONFIG['ANDROID_PACKAGE_NAME']
+    defines['MOZ_APP_DISPLAYNAME'] = CONFIG['MOZ_APP_DISPLAYNAME']
+    # Includes.
+    defines['STRINGSPATH'] = android_strings_dtd
+    defines['SYNCSTRINGSPATH'] = sync_strings_dtd
+    # Fennec branding is en-US only: see
+    # $(MOZ_BRANDING_DIRECTORY)/locales/jar.mn.
+    defines['BRANDPATH'] = '{}/{}/locales/en-US/brand.dtd'.format(
+        buildconfig.topsrcdir, CONFIG['MOZ_BRANDING_DIRECTORY'])
+
+    includes = preprocessor.preprocess(includes=[strings_xml],
+                                       defines=defines,
+                                       output=output)
+    return includes
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])
--- a/python/mozbuild/mozbuild/action/generate_suggestedsites.py
+++ b/python/mozbuild/mozbuild/action/generate_suggestedsites.py
@@ -2,19 +2,19 @@
 # This Source Code Form is subject to the terms of the Mozilla Public
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 ''' Script to generate the suggestedsites.json file for Fennec.
 
 This script follows these steps:
 
-1. Read the region.properties file in all the given source directories
-(see srcdir option). Merge all properties into a single dict accounting for
-the priority of source directories.
+1. Read all the given region.properties files (see inputs and --fallback
+options). Merge all properties into a single dict accounting for the priority
+of inputs.
 
 2. Read the list of sites from the list 'browser.suggestedsites.list.INDEX' and
 'browser.suggestedsites.restricted.list.INDEX' properties with value of these keys
 being an identifier for each suggested site e.g. browser.suggestedsites.list.0=mozilla,
 browser.suggestedsites.list.1=fxmarketplace.
 
 3. For each site identifier defined by the list keys, look for matching branches
 containing the respective properties i.e. url, title, etc. For example,
@@ -25,66 +25,76 @@ 4. Generate a JSON representation of eac
 write the result to suggestedsites.json on the locale-specific raw resource
 directory e.g. raw/suggestedsites.json, raw-pt-rBR/suggestedsites.json.
 '''
 
 from __future__ import absolute_import, print_function
 
 import argparse
 import copy
+import errno
 import json
 import sys
 import os
 
 from mozbuild.dotproperties import (
     DotProperties,
 )
 from mozbuild.util import (
     FileAvoidWrite,
 )
 from mozpack.files import (
     FileFinder,
 )
 import mozpack.path as mozpath
 
 
-def merge_properties(filename, srcdirs):
-    """Merges properties from the given file in the given source directories."""
+def merge_properties(paths):
+    """Merges properties from the given paths."""
     properties = DotProperties()
-    for srcdir in srcdirs:
-        path = mozpath.join(srcdir, filename)
+    for path in paths:
         try:
             properties.update(path)
-        except IOError:
-            # Ignore non-existing files
-            continue
+        except IOError as e:
+            if e.errno == errno.ENOENT:
+                # Ignore non-existant files.
+                continue
     return properties
 
 
-def main(args):
+def main(output, *args, **kwargs):
     parser = argparse.ArgumentParser()
     parser.add_argument('--verbose', '-v', default=False, action='store_true',
                         help='be verbose')
     parser.add_argument('--silent', '-s', default=False, action='store_true',
                         help='be silent')
     parser.add_argument('--android-package-name', metavar='NAME',
                         required=True,
                         help='Android package name')
     parser.add_argument('--resources', metavar='RESOURCES',
                         default=None,
                         help='optional Android resource directory to find drawables in')
-    parser.add_argument('--srcdir', metavar='SRCDIR',
-                        action='append', required=True,
-                        help='directories to read inputs from, in order of priority')
-    parser.add_argument('output', metavar='OUTPUT',
-                        help='output')
+    parser.add_argument('inputs', metavar='INPUT', nargs='*',
+                        help='inputs, in order of priority')
+    # This awkward "extra input" is an expedient work-around for an issue with
+    # the build system.  It allows to specify the in-tree unlocalized version
+    # of a resource where a regular input will always be the localized version.
+    parser.add_argument('--fallback',
+                        required=True,
+                        help='fallback input')
     opts = parser.parse_args(args)
 
+    # Ensure the fallback is valid.
+    if not os.path.isfile(opts.fallback):
+        print('Fallback path {fallback} is not a file!'.format(fallback=opts.fallback))
+        return 1
+
     # Use reversed order so that the first srcdir has higher priority to override keys.
-    properties = merge_properties('region.properties', reversed(opts.srcdir))
+    sources = [opts.fallback] + list(reversed(opts.inputs))
+    properties = merge_properties(sources)
 
     # Keep these two in sync.
     image_url_template = 'android.resource://%s/drawable/suggestedsites_{name}' % opts.android_package_name
     drawables_template = 'drawable*/suggestedsites_{name}.*'
 
     # Load properties corresponding to each site name and define their
     # respective image URL.
     sites = []
@@ -122,26 +132,27 @@ def main(args):
         print('Reading {len} suggested site lists: {lists}'.format(len=len(lists), lists=[list_name for list_name, _ in lists]))
 
     for (list_name, list_item_defaults) in lists:
         names = properties.get_list(list_name)
         if opts.verbose:
             print('Reading {len} suggested sites from {list}: {names}'.format(len=len(names), list=list_name, names=names))
         add_names(names, list_item_defaults)
 
+    # We must define at least one site -- that's what the fallback is for.
+    if not sites:
+        print('No sites defined: searched in {}!'.format(sources))
+        return 1
 
-    # FileAvoidWrite creates its parent directories.
-    output = os.path.abspath(opts.output)
-    fh = FileAvoidWrite(output)
-    json.dump(sites, fh)
-    existed, updated = fh.close()
+    json.dump(sites, output)
+    existed, updated = output.close()  # It's safe to close a FileAvoidWrite multiple times.
 
     if not opts.silent:
         if updated:
-            print('{output} updated'.format(output=output))
+            print('{output} updated'.format(output=os.path.abspath(output.name)))
         else:
-            print('{output} already up-to-date'.format(output=output))
+            print('{output} already up-to-date'.format(output=os.path.abspath(output.name)))
 
-    return 0
+    return set([opts.fallback])
 
 
 if __name__ == '__main__':
     sys.exit(main(sys.argv[1:]))
--- a/python/mozbuild/mozbuild/backend/recursivemake.py
+++ b/python/mozbuild/mozbuild/backend/recursivemake.py
@@ -514,43 +514,70 @@ class RecursiveMakeBackend(CommonBackend
         elif isinstance(obj, GeneratedFile):
             if obj.required_for_compile:
                 tier = 'export'
             elif obj.localized:
                 tier = 'libs'
             else:
                 tier = 'misc'
             self._no_skip[tier].add(backend_file.relobjdir)
-            first_output = obj.outputs[0]
+
+            # Localized generated files can use {AB_CD} and {AB_rCD} in their
+            # output paths.
+            if obj.localized:
+                substs = {'AB_CD': '$(AB_CD)', 'AB_rCD': '$(AB_rCD)'}
+            else:
+                substs = {}
+            outputs = []
+
+            needs_AB_rCD = False
+            for o in obj.outputs:
+                needs_AB_rCD = needs_AB_rCD or ('AB_rCD' in o)
+                try:
+                    outputs.append(o.format(**substs))
+                except KeyError as e:
+                    raise ValueError('%s not in %s is not a valid substitution in %s'
+                                     % (e.args[0], ', '.join(sorted(substs.keys())), o))
+
+            first_output = outputs[0]
             dep_file = "%s.pp" % first_output
 
             if obj.inputs:
                 if obj.localized:
                     # Localized generated files can have locale-specific inputs, which are
-                    # indicated by paths starting with `en-US/`.
+                    # indicated by paths starting with `en-US/` or containing `/locales/en-US/`.
                     def srcpath(p):
-                        bits = p.split('en-US/', 1)
-                        if len(bits) == 2:
-                            e, f = bits
+                        if '/locales/en-US' in p:
+                            e, f = p.split('/locales/en-US/', 1)
+                            assert(f)
+                            return '$(call MERGE_RELATIVE_FILE,%s,/locales/%s)' % (f, e)
+                        elif p.startswith('en-US/'):
+                            e, f = p.split('en-US/', 1)
                             assert(not e)
                             return '$(call MERGE_FILE,%s)' % f
                         return self._pretty_path(p, backend_file)
                     inputs = [srcpath(f) for f in obj.inputs]
                 else:
                     inputs = [self._pretty_path(f, backend_file) for f in obj.inputs]
             else:
                 inputs = []
 
+            if needs_AB_rCD:
+                backend_file.write_once('include $(topsrcdir)/config/AB_rCD.mk\n')
+
             # If we're doing this during export that means we need it during
             # compile, but if we have an artifact build we don't run compile,
             # so we can skip it altogether or let the rule run as the result of
             # something depending on it.
             if tier != 'export' or not self.environment.is_artifact_build:
-                backend_file.write('%s:: %s\n' % (tier, first_output))
-            for output in obj.outputs:
+                if not needs_AB_rCD:
+                    # Android localized resources have special Makefile
+                    # handling.
+                    backend_file.write('%s:: %s\n' % (tier, first_output))
+            for output in outputs:
                 if output != first_output:
                     backend_file.write('%s: %s ;\n' % (output, first_output))
                 backend_file.write('GARBAGE += %s\n' % output)
             backend_file.write('EXTRA_MDDEPEND_FILES += %s\n' % dep_file)
             if obj.script:
                 backend_file.write("""{output}: {script}{inputs}{backend}{repack_force}
 \t$(REPORT_BUILD)
 \t$(call py_action,file_generate,{locale}{script} {method} {output} $(MDDEPDIR)/{dep_file}{inputs}{flags})
--- a/python/mozbuild/mozbuild/frontend/context.py
+++ b/python/mozbuild/mozbuild/frontend/context.py
@@ -1456,20 +1456,27 @@ VARIABLES = {
         For simple cases of text substitution, prefer ``LOCALIZED_PP_FILES``.
 
         Refer to the documentation of ``GENERATED_FILES``; for the most part things work the same.
         The two major differences are:
         1. The function in the Python script will be passed an additional keyword argument `locale`
            which provides the locale in use, i.e. ``en-US``.
         2. The ``inputs`` list may contain paths to files that will be taken from the locale
            source directory (see ``LOCALIZED_FILES`` for a discussion of the specifics). Paths
-           in ``inputs`` starting with ``en-US/`` are considered localized files.
+           in ``inputs`` starting with ``en-US/`` or containing ``/locales/en-US/`` are considered
+           localized files.
 
         To place the generated output file in a specific location, list its objdir path in
         ``LOCALIZED_FILES``.
+
+        In addition, ``LOCALIZED_GENERATED_FILES`` can use the special substitutions ``{AB_CD}``
+        and ``{AB_rCD}`` in their output paths.  ``{AB_CD}`` expands to the current locale during
+        multi-locale builds and single-locale repacks and ``{AB_rCD}`` expands to an
+        Android-specific encoding of the current locale.  Both expand to the empty string when the
+        current locale is ``en-US``.
         """),
 
     'OBJDIR_FILES': (ContextDerivedTypedHierarchicalStringList(Path), list,
         """List of files to be installed anywhere in the objdir. Use sparingly.
 
         ``OBJDIR_FILES`` is similar to FINAL_TARGET_FILES, but it allows copying
         anywhere in the object directory. This is intended for various one-off
         cases, not for general use. If you wish to add entries to OBJDIR_FILES,
--- a/python/mozbuild/mozbuild/frontend/emitter.py
+++ b/python/mozbuild/mozbuild/frontend/emitter.py
@@ -1230,20 +1230,27 @@ class TreeMetadataEmitter(LoggingMixin):
                                 'OBJDIR_PP_FILES',
                                 'LOCALIZED_PP_FILES') and
                         not isinstance(f, SourcePath)):
                         raise SandboxValidationError(
                                 ('Only source directory paths allowed in ' +
                                  '%s: %s')
                                 % (var, f,), context)
                     if var.startswith('LOCALIZED_'):
-                        if isinstance(f, SourcePath) and not f.startswith('en-US/'):
-                            raise SandboxValidationError(
-                                    '%s paths must start with `en-US/`: %s'
-                                    % (var, f,), context)
+                        if isinstance(f, SourcePath):
+                            if f.startswith('en-US/'):
+                                pass
+                            elif '/locales/en-US/' in f:
+                                pass
+                            else:
+                                raise SandboxValidationError(
+                                        '%s paths must start with `en-US/` or '
+                                        'contain `/locales/en-US/`: %s'
+                                        % (var, f,), context)
+
                     if not isinstance(f, ObjDirPath):
                         path = f.full_path
                         if '*' not in path and not os.path.exists(path):
                             raise SandboxValidationError(
                                 'File listed in %s does not exist: %s'
                                 % (var, path), context)
                     else:
                         # TODO: Bug 1254682 - The '/' check is to allow
--- a/python/mozbuild/mozbuild/test/action/test_generate_browsersearch.py
+++ b/python/mozbuild/mozbuild/test/action/test_generate_browsersearch.py
@@ -8,16 +8,19 @@ from __future__ import unicode_literals
 import json
 import os
 import unittest
 
 import mozunit
 
 import mozbuild.action.generate_browsersearch as generate_browsersearch
 
+from mozbuild.util import (
+    FileAvoidWrite,
+)
 from mozfile.mozfile import (
     NamedTemporaryFile,
     TemporaryDirectory,
 )
 
 import mozpack.path as mozpath
 
 
@@ -30,20 +33,23 @@ class TestGenerateBrowserSearch(unittest
     Unit tests for generate_browsersearch.py.
     """
 
     def _test_one(self, name):
         with TemporaryDirectory() as tmpdir:
             with NamedTemporaryFile(mode='r+') as temp:
                 srcdir = os.path.join(test_data_path, name)
 
-                generate_browsersearch.main([
-                    '--silent',
-                    '--srcdir', srcdir,
-                    temp.name])
+                with FileAvoidWrite(temp.name) as faw:
+                    generate_browsersearch.main(faw,
+                        '--silent',
+                        '--fallback',
+                        mozpath.join(srcdir, 'region.properties'),
+                        )
+
                 return json.load(temp)
 
     def test_valid_unicode(self):
         o = self._test_one('valid-zh-CN')
         self.assertEquals(o['default'], '百度')
         self.assertEquals(o['engines'], ['百度', 'Google'])
 
     def test_invalid_unicode(self):
copy from python/mozbuild/mozbuild/test/backend/data/localized-generated-files/en-US/localized-input
copy to python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/en-US/localized-input
copy from python/mozbuild/mozbuild/test/backend/data/localized-generated-files/foo-data
copy to python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/foo-data
copy from python/mozbuild/mozbuild/test/backend/data/localized-generated-files/generate-foo.py
copy to python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/generate-foo.py
copy from python/mozbuild/mozbuild/test/backend/data/localized-generated-files/en-US/localized-input
copy to python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/inner/locales/en-US/localized-input
copy from python/mozbuild/mozbuild/test/backend/data/localized-generated-files/moz.build
copy to python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/moz.build
--- a/python/mozbuild/mozbuild/test/backend/data/localized-generated-files/moz.build
+++ b/python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/moz.build
@@ -1,15 +1,21 @@
 # -*- Mode: python; indent-tabs-mode: nil; tab-width: 40 -*-
 # Any copyright is dedicated to the Public Domain.
 # http://creativecommons.org/publicdomain/zero/1.0/
 
-LOCALIZED_GENERATED_FILES += [ 'foo.xyz' ]
+LOCALIZED_GENERATED_FILES += [ 'foo{AB_CD}.xyz' ]
 
-foo = LOCALIZED_GENERATED_FILES['foo.xyz']
+foo = LOCALIZED_GENERATED_FILES['foo{AB_CD}.xyz']
 foo.script = 'generate-foo.py'
 foo.inputs = [
     'en-US/localized-input',
     'non-localized-input',
 ]
 
-# Also check that using it in LOCALIZED_FILES does the right thing.
-LOCALIZED_FILES += [ '!foo.xyz' ]
+LOCALIZED_GENERATED_FILES += [ 'bar{AB_rCD}.xyz' ]
+
+bar = LOCALIZED_GENERATED_FILES['bar{AB_rCD}.xyz']
+bar.script = 'generate-foo.py'
+bar.inputs = [
+    'inner/locales/en-US/localized-input',
+    'non-localized-input',
+]
copy from python/mozbuild/mozbuild/test/backend/data/localized-generated-files/non-localized-input
copy to python/mozbuild/mozbuild/test/backend/data/localized-generated-files-AB_CD/non-localized-input
--- a/python/mozbuild/mozbuild/test/backend/test_recursivemake.py
+++ b/python/mozbuild/mozbuild/test/backend/test_recursivemake.py
@@ -437,16 +437,44 @@ class TestRecursiveMakeBackend(BackendTe
             'LOCALIZED_FILES_0_DEST = $(FINAL_TARGET)/',
             'LOCALIZED_FILES_0_TARGET := libs',
             'INSTALL_TARGETS += LOCALIZED_FILES_0',
         ]
 
         self.maxDiff = None
         self.assertEqual(lines, expected)
 
+    def test_localized_generated_files_AB_CD(self):
+        """Ensure LOCALIZED_GENERATED_FILES is handled properly
+        when {AB_CD} and {AB_rCD} are used."""
+        env = self._consume('localized-generated-files-AB_CD', RecursiveMakeBackend)
+
+        backend_path = mozpath.join(env.topobjdir, 'backend.mk')
+        lines = [l.strip() for l in open(backend_path, 'rt').readlines()[2:]]
+
+        expected = [
+            'libs:: foo$(AB_CD).xyz',
+            'GARBAGE += foo$(AB_CD).xyz',
+            'EXTRA_MDDEPEND_FILES += foo$(AB_CD).xyz.pp',
+            'foo$(AB_CD).xyz: %s/generate-foo.py $(call MERGE_FILE,localized-input) $(srcdir)/non-localized-input $(if $(IS_LANGUAGE_REPACK),FORCE)' % env.topsrcdir,
+            '$(REPORT_BUILD)',
+            '$(call py_action,file_generate,--locale=$(AB_CD) %s/generate-foo.py main foo$(AB_CD).xyz $(MDDEPDIR)/foo$(AB_CD).xyz.pp $(call MERGE_FILE,localized-input) $(srcdir)/non-localized-input)' % env.topsrcdir,
+            '',
+            'include $(topsrcdir)/config/AB_rCD.mk',
+            'GARBAGE += bar$(AB_rCD).xyz',
+            'EXTRA_MDDEPEND_FILES += bar$(AB_rCD).xyz.pp',
+            'bar$(AB_rCD).xyz: %s/generate-foo.py $(call MERGE_RELATIVE_FILE,localized-input,/locales/inner) $(srcdir)/non-localized-input $(if $(IS_LANGUAGE_REPACK),FORCE)' % env.topsrcdir,
+            '$(REPORT_BUILD)',
+            '$(call py_action,file_generate,--locale=$(AB_CD) %s/generate-foo.py main bar$(AB_rCD).xyz $(MDDEPDIR)/bar$(AB_rCD).xyz.pp $(call MERGE_RELATIVE_FILE,localized-input,/locales/inner) $(srcdir)/non-localized-input)' % env.topsrcdir,
+            '',
+        ]
+
+        self.maxDiff = None
+        self.assertEqual(lines, expected)
+
     def test_exports_generated(self):
         """Ensure EXPORTS that are listed in GENERATED_FILES
         are handled properly."""
         env = self._consume('exports-generated', RecursiveMakeBackend)
 
         # EXPORTS files should appear in the dist_include install manifest.
         m = InstallManifest(path=mozpath.join(env.topobjdir,
             '_build_manifests', 'install', 'dist_include'))
new file mode 100644
--- a/python/mozbuild/mozbuild/test/frontend/data/localized-files-no-en-us/moz.build
+++ b/python/mozbuild/mozbuild/test/frontend/data/localized-files-no-en-us/moz.build
@@ -1,8 +1,9 @@
 # -*- Mode: python; indent-tabs-mode: nil; tab-width: 40 -*-
 # Any copyright is dedicated to the Public Domain.
 # http://creativecommons.org/publicdomain/zero/1.0/
 
 LOCALIZED_FILES.foo += [
     'en-US/bar.ini',
     'foo.js',
+    'inner/locales/en-US/bar.ini',
 ]
--- a/python/mozbuild/mozbuild/test/frontend/test_emitter.py
+++ b/python/mozbuild/mozbuild/test/frontend/test_emitter.py
@@ -1367,20 +1367,20 @@ class TestEmitterBasic(unittest.TestCase
             self.assertEqual(len(files), 3)
 
             expected = {'en-US/bar.ini', 'en-US/code/*.js', 'en-US/foo.js'}
             for f in files:
                 self.assertTrue(unicode(f) in expected)
 
     def test_localized_files_no_en_us(self):
         """Test that LOCALIZED_FILES errors if a path does not start with
-        `en-US/`."""
+        `en-US/` or contain `/locales/en-US/`."""
         reader = self.reader('localized-files-no-en-us')
         with self.assertRaisesRegexp(SandboxValidationError,
-             'LOCALIZED_FILES paths must start with `en-US/`:'):
+             'LOCALIZED_FILES paths must start with `en-US/` or contain `/locales/en-US/`: foo.js'):
             objs = self.read_topsrcdir(reader)
 
     def test_localized_pp_files(self):
         """Test that LOCALIZED_PP_FILES works properly."""
         reader = self.reader('localized-pp-files')
         objs = self.read_topsrcdir(reader)
 
         self.assertEqual(len(objs), 1)