Bug 1438655 - Add Introduction to Fluent for Firefox Developers. r?flod draft
authorZibi Braniecki <zbraniecki@mozilla.com>
Tue, 27 Feb 2018 23:52:26 -0800
changeset 763399 506d529208350eb219ceaa12ee4507b2c9512a9c
parent 763311 0ef34a9ec4fbfccd03ee0cfb26b182c03e28133a
push id101453
push userbmo:gandalf@aviary.pl
push dateMon, 05 Mar 2018 23:18:59 +0000
reviewersflod
bugs1438655
milestone60.0a1
Bug 1438655 - Add Introduction to Fluent for Firefox Developers. r?flod MozReview-Commit-ID: 33v3OTFYRHi
intl/l10n/docs/fluent_tutorial.rst
intl/l10n/docs/index.rst
intl/l10n/moz.build
new file mode 100644
--- /dev/null
+++ b/intl/l10n/docs/fluent_tutorial.rst
@@ -0,0 +1,683 @@
+.. role:: html(code)
+   :language: html
+
+.. role:: js(code)
+   :language: javascript
+
+=============================
+Fluent for Firefox Developers
+=============================
+
+
+This tutorial is intended for Firefox engineers already familiar with the previous
+localization systems offered by Gecko - `DTD`_ and  `StringBundle`_ - and assumes
+prior experience with those systems.
+
+
+Using Fluent in Gecko
+=====================
+
+`Fluent`_ is a modern localization system currently being progressively introduced into
+the Gecko platform with a focus on quality, performance, maintenance and completeness.
+
+In order to ensure that Fluent is ready for engineers to work with, the initial
+migrations are performed manually with a lot of oversight from the involved
+stakeholders.
+
+In this initial phase, `Firefox Preferences`_ is being migrated as the first target
+and as a result, the first bindings to be stabilized are for chrome-privileged
+XUL context.
+
+From there we plan to focus on two areas:
+
+ - `Unprivileged Contexts`_
+ - `System Add-ons`_
+
+The end goal is replacing all uses of DTD and StringBundle within Firefox's codebase.
+
+If you want to use Fluent and your code involves one of the areas currently unsupported,
+we'd like to work with you on getting Fluent ready for your code.
+
+
+Getting a Review
+----------------
+
+If you end up working on any patch which touches FTL files, we have a temporary
+hook in place that will reject your patch unless you get an r+ from one of the following
+L10n Drivers:
+
+  - Francesco Lodolo (:flod)
+  - Zibi Braniecki (:gandalf)
+  - Axel Hecht (:pike)
+  - Stas Malolepszy (:stas)
+
+
+Major Benefits
+==============
+
+Not only was the previous system designed over 20 years ago using file formats
+never intended for localization, but also the Web stack which Fluent ties into has
+completely changed over the same period, and the domain of internationalization
+got a powerful foundation in the form of `Unicode`_, `CLDR`_ and `ICU`_ which Fluent tightly
+`interoperates with`__.
+
+__ https://github.com/projectfluent/fluent/wiki/Fluent-and-Standards
+
+While it is beyond the scope of this document to cover all the benefits of Fluent in detail,
+below is an attempt to select some most observable changes for each group of consumers.
+
+
+Developers
+----------
+
+ - Support for XUL, XHTML, HTML, Web Components, React, JS, Python and Rust
+ - Strings are available in a single, unified localization context available for both DOM and runtime code
+ - Full internationalization (i18n) support: date and time formatting, number formatting, plurals, genders etc.
+ - Strong focus on `declarative API via DOM attributes`__
+ - Extensible with custom formatters, Mozilla-specific APIs etc.
+ - `Separation of concerns`__: localization details, and the added complexity of some languages, don't leak onto the source code and are no concern for developers
+ - Compound messages link a single translation unit to a single UI element
+ - `DOM Overlays`__ allow for localization of DOM fragments
+ - Simplified build system model
+ - No need for pre-processing instructions
+
+__ https://github.com/projectfluent/fluent/wiki/Get-Started
+__ https://github.com/projectfluent/fluent/wiki/Design-Principles
+__ https://github.com/projectfluent/fluent.js/wiki/DOM-Overlays
+
+
+Product Quality
+------------------
+
+ - A robust, multilevel, `error fallback system`__ prevents XML errors and runtime errors
+ - Simplified l10n API reduces the amount of l10n specific code and resulting bugs
+ - Runtime localization allows for dynamic language changes and updates over-the-air
+ - DOM Overlays increase localization security
+
+Many other smaller improvements will be noticed by the users of the system over time
+and, with the new foundation, the Fluent team is `currently working`__ on multiple highly
+requested features which will further improve the experience of developing
+localizable UIs.
+
+__ https://github.com/projectfluent/fluent/wiki/Error-Handling
+__ https://github.com/projectfluent/fluent/wiki/Roadmap
+
+
+Fluent Translation List - FTL
+=============================
+
+Fluent introduces a new localization format designed specifically for easy readability
+and localization features offered by the system.
+
+At first glance the format resembles `.properties` file. It may look like this:
+
+.. code-block:: properties
+
+  home-page-header = Home Page
+
+  # The label of a button opening a new tab
+  new-tab-open = Open New Tab
+
+But the FTL file format is significantly more powerful and the additional features
+quickly add up. In order to familiarize yourself with the basic features,
+consider reading through the `Fluent Syntax Guide`_ to understand
+a more complex example like:
+
+.. code-block:: properties
+
+  ### These messages correspond to security and privacy user interface.
+  ###
+  ### Please, choose simple and non-threatening language when localizing
+  ### to help user feel in control when interacting with the UI.
+
+  ## General Section
+
+  -brand-short-name = Firefox
+      .gender = masculine
+  
+  pref-pane =
+      .title =
+          { PLATFORM() ->
+              [windows] Options
+             *[other] Preferences
+          }
+      .accesskey = C
+  
+  # Variables:
+  #   $tabCount (Number) - number of container tabs to be closed
+  containers-disable-alert-ok-button =
+      { $tabCount ->
+          [one] Close { $tabCount } Container Tab
+         *[other] Close { $tabCount } Container Tabs
+      }
+  
+  update-application-info =
+      You are using { -brand-short-name } Version: { $version }.
+      <span>Please, read the <a>privacy policy</a>.</span>
+
+The above, of course, is a particular selection of complex strings intended to exemplify
+the new features and concepts introduced by Fluent.
+
+In order to ensure the quality of the output, a lot of new checks and tooling
+has been added to the build system.
+`Pontoon`_, the main localization tool used to translate Firefox, has been rebuilding
+its user experience to support localizers in their work.
+
+
+Social Contract
+===============
+
+Fluent uses the concept of a `social contract` between developer and localizers.
+This contract is established by the selection of a unique identifier, called :js:`l10n-id`,
+which carries a promise of being used in a particular place to carry a particular meaning.
+
+The use of unique identifiers is not new for Firefox engineers, but it is important
+to recognize that Fluent formalizes this relationship.
+
+.. important::
+
+  An important part of the contract is that the developer commits to treat the
+  localization output as `opaque`. That means that no concatenations, replacements
+  or splitting should happen after the translation is completed to generate the
+  desired output.
+
+In return, localizers enter the social contract by promising to provide an accurate
+and clean translation of the messages that match the request.
+
+In previous localization systems, developers were responsible for differentiating
+string variant based on a platform via pre-processing instructions, or
+selecting which strings should be formatted using `PluralForms.jsm`.
+
+In Fluent, the developer is not to be bothered with inner logic and complexity that the
+localization will use to construct the response. Whether `declensions`__ or other
+variant selection techniques are used is up to a localizer and their particular translation.
+From the developer perspective, Fluent returns a final string to be presented to
+the user, with no l10n logic required in the running code.
+
+__ https://en.wikipedia.org/wiki/Declension
+
+
+Markup Localization
+===================
+
+Fluent fully replaces the use of `DTD`_ in localization.
+
+To localize an element in Fluent, the developer adds a new message to
+an FTL file and then has to associate an :js:`l10n-id` with the element
+by defining a :js:`data-l10n-id` attribute:
+
+.. code-block:: html
+
+  <h1 data-l10n-id="home-page-header" />
+
+  <button data-l10n-id="pref-pane" />
+
+Fluent will take care of the rest, populating the element with the message value
+in its content and all localizable attributes if defined.
+
+The difference compared to the use of DTD is that the developer provides only a single
+message to localize the whole element, rather than a separate entity for
+the value and each of the attributes.
+
+The other change is that the developer can localize a whole fragment of DOM:
+
+.. code-block:: html
+
+  <p data-l10n-id="update-application-info" data-l10n-args="{'version': '60.0'}">
+    <span class="bold">
+      <a href="http://www.mozilla.org/privacy" />
+    </span>
+  </p>
+
+.. code-block:: properties
+
+  -brand-short-name = Firefox
+  update-application-info =
+      You are using { -brand-short-name } Version: { $version }.
+      <span>Please, read the <a>privacy policy</a>.</span>
+
+
+Fluent will overlay the translation onto the source fragment preserving attributes like
+:code:`class` and :code:`href` from the source and adding translations for the elements
+inside. The end result will look like this:
+
+.. code-block:: html
+
+  <p data-l10n-id="update-application-info" data-l10n-args="{'version': '60.0'}">
+    You are using Firefox Version: 60.0.
+    <span class="bold">
+      Please, read the <a href="http://www.mozilla.org/privacy">privacy policy</a>.
+    </span>
+  </p>
+
+
+This operation is sanitized, and Fluent takes care of selecting which elements and
+attributes can be safely provided by the localization.
+The list of allowed elements and attributes is `maintained by the W3C`__, and if
+the developer needs to allow for localization of additional attributes, they can
+whitelist them using :code:`data-l10n-attrs` list:
+
+.. code-block:: html
+
+  <label data-l10n-id="search-input" data-l10n-attrs="style" />
+
+The above example adds an attribute :code:`style` to be allowed on this
+particular :code:`label` element.
+
+
+External Arguments
+------------------
+
+Notice in the previous example the attribute :code:`data-l10n-args`, which is
+a JSON object storing variables exposed by the developer to the localizer.
+
+This is the main channel for the developer to provide additional variables
+to be used in the localization.
+
+It is very rare that the arguments are needed for localizations which previously
+used DTD, because such variables will usually have to be computed from the runtime code,
+but it is worth understanding that when the :code:`l10n-args` are set in
+the runtime code, they are in fact encoded via JSON and stored together with
+:code:`l10n-id` as an attribute on the element.
+
+__ https://www.w3.org/TR/2011/WD-html5-20110525/text-level-semantics.html
+
+
+Runtime Localization
+====================
+
+Fluent fully replaces the use of `StringBundle`_ in localization.
+
+In almost every case the JS runtime code will operate on a particular document, either
+XUL, XHTML or HTML.
+
+If the document has its markup already localized, then Fluent exposes a new
+attribute on the :js:`document` element - :js:`document.l10n`.
+
+This property is an object of type :js:`DOMLocalization` which maintains the main
+localization context for this document and exposes it to runtime code as well.
+
+With a focus on `declarative localization`__, the primary method of localization is
+to alter the localization attributes in the DOM. Fluent provides a method to facilitate this:
+
+.. code-block:: javascript
+
+  document.l10n.setAttributes(element, "new-panel-header");
+
+This will set the :code:`data-l10n-id` on the element and translate it before the next
+animation frame.
+
+The reason to use this API over manually setting the attribute is that it also
+facilitates encoding l10n arguments as JSON:
+
+.. code-block:: javascript
+
+  document.l10n.setAttributes(element "containers-disable-alert-ok-button", {
+    tabCount: 5
+  }
+
+__ https://github.com/projectfluent/fluent/wiki/Good-Practices-for-Developers
+
+
+Non-Markup Localization
+-----------------------
+
+In rare cases, when the runtime code needs to retrieve the translation and not
+apply it onto the DOM, Fluent provides an API to retrieve it:
+
+.. code-block:: javascript
+
+  let [ msg ] = await document.l10n.formatValues([
+    ["remove-containers-description"]
+  ]);
+
+  alert(msg);
+
+This model is heavily discouraged and should be used only in cases where the
+DOM annotation is not possible.
+
+.. note::
+
+  This API is currently only available as asynchronous. In case of Firefox,
+  the only non-DOM localizable calls are used where the output goes to
+  a third-party like Bluetooth, Notifications etc.
+  All those cases should already be asynchronous.
+
+
+Internationalization
+====================
+
+The majority of internationalization issues are implicitly handled by Fluent without
+any additional requirement. Full Unicode support, `bidirectionality`__, and
+correct number formatting work without any action required from either
+developer or localizer.
+
+__ https://github.com/projectfluent/fluent/wiki/BiDi-in-Fluent
+
+.. code-block:: javascript
+
+  document.l10n.setAttributes(element, "welcome-message", {
+    userName: "اليسع",
+    count: 5
+  });
+
+A message like this localized to American English will correctly wrap the user
+name in directionality marks allowing the layout engine to determine how to
+display the bidirectional text.
+
+On the other hand, the same message localized to Arabic will use the Eastern Arabic
+numeral for number "5".
+
+
+Plural Rules
+------------
+
+The most common localization feature is the ability to provide different variants
+of the same string depending on plural categories.
+
+Fluent replaces the use of the proprietary :code:`PluralForms.jsm` with a Unicode CLDR
+standard called `Plural Rules`_.
+
+In order to allow localizers to use it, all the developer has to do is to pass
+an external argument number:
+
+.. code-block:: javascript
+
+  document.l10n.setAttributes(element, "unread-warning", { unreadCount: 5 });
+
+Localizers can use the argument to build a multi variant message if their
+language requires that:
+
+.. code-block:: properties
+
+  unread-warning =
+      { $unreadCount ->
+          [one] You have { $unreadCount } unread message
+         *[other] You have { $unreadCount } unread messages
+      }
+
+Fluent guesses that since the variant selection is performed based on a number,
+its `plural category`__ should be retrieved.
+
+If the given translation doesn't need pluralization for the string (for example
+Japanese often will not), the localizer can replace it with:
+
+.. code-block:: properties
+
+  unread-warning = You have { $unreadCount } unread messages
+
+and the message will preserve the social contract.
+
+One additional feature is that the localizer can further improve the message by
+specifying variants for particular values:
+
+.. code-block:: properties
+
+  unread-warning =
+      { $unreadCount ->
+          [0] You have no unread messages
+          [1] You have one unread message
+         *[other] You have { $unreadCount } unread messages
+      }
+
+The advantage here is that per-locale choices don't leak onto the source code
+and the developer is not affected.
+
+
+.. note::
+
+  There is an important distinction between a variant keyed on plural category
+  `one` and digit `1`. Although in English the two are synonymous, in other
+  languages category `one` may be used for other numbers.
+  For example in `Bosnian`__, category `one` is used for numbers like `1`, `21`, `31`
+  and so on, and also for fractional numbers like `0.1`.
+
+__ https://unicode.org/cldr/charts/latest/supplemental/language_plural_rules.html
+__ https://unicode.org/cldr/charts/latest/supplemental/language_plural_rules.html#bs
+
+Partial Arguments
+-----------------
+
+When it comes to formatting data, Fluent allows the developer to provide
+a set of parameters for the formatter, and the localizer can fine tune some of them.
+This technique is called `partial arguments`__.
+
+For example, when formatting a date, the developer can just pass a JS :js:`Date` object,
+but its default formatting will be pretty expressive. In most cases, the developer
+may want to use some of the :js:`Intl.DateTimeFormat` options to select the default
+representation of the date in string:
+
+.. code-block:: javascript
+
+  document.l10n.setAttributes(element, "welcome-message", {
+  startDate: FluentDateTime(new Date(), {
+      year: "numeric",
+      month: "long",
+      day: "numeric"
+    })
+  });
+
+.. code-block:: properties
+
+  welcome-message = Your session will start date: { $startDate }
+
+In most cases, that will be enough and the date would get formatted in the current
+Firefox as `February 28, 2018`.
+
+But if in some other locale the string would get too long, the localizer can fine
+tune the options as well:
+
+.. code-block:: properties
+
+  welcome-message = Początek Twojej sesji: { DATETIME($startDate, month="short") }
+
+This will adjust the length of the month token in the message to short and get formatted
+in Polish as `28 lut 2018`.
+
+At the moment Fluent supports two formatters that match JS Intl API counterparts:
+
+ * **NUMBER**: `Intl.NumberFormat`__
+ * **DATETIME**: `Intl.DateTimeFormat`__
+
+With time more formatters will be added.
+
+__ http://projectfluent.org/fluent/guide/functions.html#partial-arguments
+__ https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/NumberFormat
+__ https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DateTimeFormat
+
+Registering New L10n Files
+==========================
+
+In the previous system, a new localization file had to be registered in order to
+add it in the `jar.mn` file for packaging.
+
+Fluent uses a wildcard statement packaging all localization resources into
+their component's `/localization/` directory.
+
+That means that, if a new file is added to a component of Firefox already
+covered by Fluent like `browser`, it's enough to add the new file to the
+repository in a path like `browser/locales/en-US/browser/component/file.ftl` and
+the toolchain will package it into `browser/localization/browser/component/file.ftl`.
+
+At runtime Firefox uses a special registry for all localization data. It will
+register the browser's `/localization/` directory and make all files inside it
+available to be references.
+
+To make the document localized using Fluent, all the developer has to do is add
+a single polyfill for the Fluent API to the source and list the resources
+that will be used:
+
+.. code-block:: html
+
+  <link rel="localization" href="branding/brand.ftl"/>
+  <link rel="localization" href="browser/preferences/preferences.ftl"/>
+  <script src="chrome://global/content/l10n.js"></script>
+
+For performance reasons the :html:`<link/>` elements have to be specified above the
+:html:`<script/>` and the :html:`<script/>` itself has to be synchronous in order to ensure
+that the localization happens before first paint.
+
+This allows Fluent to trigger asynchronous resource loading early enough to
+perform the initial DOM translation before the initial layout.
+
+The URI provided to the :html:`<link/>` element are relative paths within the localization
+system.
+
+Notice that only the registration of the script is synchronous. All the I/O and
+translation happen asynchronously.
+
+
+Custom Contexts
+===============
+
+The above method creates a single localization context per document.
+In almost all scenarios that's sufficient.
+
+In rare edge cases where the developer needs to fetch additional resources, or
+the same resources in another language, it is possible to create additional
+contexts manually using `Localization` class:
+
+.. code-block:: javascript
+
+	const { Localization } =
+		ChromeUtils.import("resource://gre/modules/Localization.jsm", {});
+
+
+	const myL10n = new Localization([
+		"branding/brand.ftl",
+		"browser/preferences/preferences.ftl"
+	]);
+
+
+	let [isDefaultMsg, isNotDefaultMsg] =
+		myL10n.formatValues(["is-default", "is-not-default"]);
+
+
+.. admonition:: Example
+
+  An example of a use case is the Preferences UI in Firefox which uses the
+  main context to localize the UI but also to build a search index.
+
+  It is common to build such search index both in a current langauge and additionally
+  in English since a lot of documentation and online help exists in that language.
+
+  A developer may create manually a new context with the same resources as the main one
+  uses, but hardcode it to `en-US` and then build the search index using both contexts.
+
+Designing Localizable APIs
+==========================
+
+When designing localizable APIs, the most important rule is to resolve localization as
+late as possible. That means that instead of resolving strings somewhere deep in the
+codebase and then passing them on or even caching, it is highly recommended to pass
+around :code:`l10n-id` or :code:`[l10n-id, l10n-args]` pairs until the top-most code
+resolves them or applies them onto the DOM element.
+
+
+Testing
+=======
+
+When writing tests that involve both I18n and L10n, the general rule is that
+result strings are opaque. That means that the developer should not assume any particular
+value and should never test against it.
+
+In case of raw i18n the :js:`resolvedOptions` method on all :js:`Intl.*` formatters
+makes it relatively easy. In case of localization, the recommended way is to test that
+the code sets the right :code:`l10n-id`/:code:`l10n-args` attributes like this:
+
+.. code-block:: javascript
+  
+  testedFunction();
+  
+  const l10nAttrs = document.l10n.getAttributes(element);
+  
+  deepEquals(l10nAttrs, {
+		id: "my-expected-id",
+		args: {
+			unreadCount: 5
+		}
+  });
+
+If the code really has to test for particular values in the localized UI, it is
+always better to scan for a variable:
+
+.. code-block:: javascript
+
+  testedFunction();
+  
+  equals(element.textContent.contains("John"));
+
+.. important::
+
+  Testing against whole values is brittle and will break when we insert Unicode
+  bidirectionality marks into the result string or adapt the output in other ways.
+
+
+Inner Structure of Fluent
+=========================
+
+The inner structure of Fluent in Gecko is out of scope of this tutorial, but
+since the class and file names may show up during debugging or profiling,
+below is a list of major components, each with a corresponding file in `/intl/l10n`
+modules in Gecko.
+
+
+MessageContext
+--------------
+
+MessageContext is the lowest level API. It's fully synchronous, contains a parser for the
+FTL file format and a resolver for the logic. It is not meant to be used by
+consumers directly.
+
+In the future we intend to offer this layer for standardization and it may become
+part of the :js:`mozIntl.*` or even :js:`Intl.*` API sets.
+
+That part of the codebase is also the first that we'll be looking to port to Rust.
+
+
+Localization
+------------
+
+Localization is a higher level API which uses :js:`MessageContext` internally but
+provides a full layer of compound message formatting and robust error fall-backing.
+
+It is intended for use in runtime code and contains all fundamental localization
+methods.
+
+
+DOMLocalization
+---------------
+
+DOMLocalization extends :js:`Localization` with functionality to operate on HTML, XUL
+and the DOM directly including DOM Overlays and Mutation Observers.
+
+
+l10n.js
+-------
+
+l10n.js is a small runtime code which fetches the :html:`<link>` elements specified
+in the document and initializes the main :js:`DOMLocalization` context
+on :js:`document.l10n`.
+
+
+L10nRegistry
+------------
+
+L10nRegistry is our resource management service. It replaces :js:`ChromeRegistry` and
+maintains the state of resources packaged into the build and language packs,
+providing an asynchronous iterator of :js:`MessageContext` objects for a given locale set
+and resources that the :js:`Localization` class uses.
+
+
+.. _Fluent: http://projectfluent.org/
+.. _DTD: https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XUL/Tutorial/Localization
+.. _StringBundle: https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XUL/Tutorial/Property_Files
+.. _Firefox Preferences: https://bugzilla.mozilla.org/show_bug.cgi?id=1415730
+.. _Unprivileged Contexts: https://bugzilla.mozilla.org/show_bug.cgi?id=1407418
+.. _System Add-ons: https://bugzilla.mozilla.org/show_bug.cgi?id=1425104
+.. _CLDR: http://cldr.unicode.org/
+.. _ICU: http://site.icu-project.org/
+.. _Unicode: https://www.unicode.org/
+.. _Fluent Syntax Guide: http://projectfluent.org/fluent/guide/
+.. _Pontoon: https://pontoon.mozilla.org/
+.. _Plural Rules: http://cldr.unicode.org/index/cldr-spec/plural-rules
new file mode 100644
--- /dev/null
+++ b/intl/l10n/docs/index.rst
@@ -0,0 +1,23 @@
+======
+Fluent
+======
+
+`Fluent`_ is a new localization system, developed by Mozilla, which aims to replace
+all existing localization models currently used at Mozilla.
+
+In case of Firefox it directly superseeds DTD and StringBundle systems providing
+a large number of new features and improvements over them both, for developers
+and localizers.
+
+.. toctree::
+   :maxdepth: 2
+
+   fluent_tutorial
+
+Other resources:
+
+ * `Fluent Syntax Guide <http://projectfluent.org/fluent/guide/>`_
+ * `Fluent Wiki <https://github.com/projectfluent/fluent/wiki>`_
+ * `Fluent.js Wiki <https://github.com/projectfluent/fluent.js/wiki>`_
+
+.. _Fluent: http://projectfluent.org/
--- a/intl/l10n/moz.build
+++ b/intl/l10n/moz.build
@@ -12,9 +12,11 @@ EXTRA_JS_MODULES += [
 ]
 
 XPCSHELL_TESTS_MANIFESTS += ['test/xpcshell.ini']
 
 MOCHITEST_CHROME_MANIFESTS += ['test/chrome.ini']
 
 JAR_MANIFESTS += ['jar.mn']
 
+SPHINX_TREES['l10n'] = 'docs'
+
 FINAL_LIBRARY = 'xul'