bug 1460595 - Document the event ping r?Dexter draft
authorChris H-C <chutten@mozilla.com>
Thu, 10 May 2018 13:30:11 -0400
changeset 810752 074294ee69e69566079af454d2968a353fc78d1a
parent 810712 ca98b6f47b4e33673291a4dc2a4ebca28ce6db00
child 810753 18534bd69dd81e06d72a389cb9f31cd5bf608d56
push id114089
push userbmo:chutten@mozilla.com
push dateTue, 26 Jun 2018 12:37:14 +0000
reviewersDexter
bugs1460595
milestone63.0a1
bug 1460595 - Document the event ping r?Dexter MozReview-Commit-ID: ES8jW1xOQUd
toolkit/components/telemetry/docs/collection/events.rst
toolkit/components/telemetry/docs/data/event-ping.rst
toolkit/components/telemetry/docs/data/main-ping.rst
toolkit/components/telemetry/docs/internals/preferences.rst
--- a/toolkit/components/telemetry/docs/collection/events.rst
+++ b/toolkit/components/telemetry/docs/collection/events.rst
@@ -9,16 +9,18 @@ Across the different Firefox initiatives
 Adding events is supported in artifact builds and build faster workflows.
 
 For events recorded into Firefox Telemetry we also provide an API that opaquely handles storage and submission to our servers.
 
 .. important::
 
     Every new data collection in Firefox needs a `data collection review <https://wiki.mozilla.org/Firefox/Data_Collection#Requesting_Approval>`_ from a data collection peer. Just set the feedback? flag for one of the data peers. We try to reply within a business day.
 
+.. _events.serializationformat:
+
 Serialization format
 ====================
 
 Events are submitted as an array, e.g.:
 
 .. code-block:: js
 
   [
@@ -230,21 +232,25 @@ Example:
   // Now events can be recorded.
   Services.telemetry.recordEvent("myAddon.interaction", "click", "red_button");
 
 Internal API
 ------------
 
 .. code-block:: js
 
-  Services.telemetry.snapshotEvents(dataset, clear);
+  Services.telemetry.snapshotEvents(dataset, clear, eventLimit);
   Services.telemetry.clearEvents();
 
 These functions are only supposed to be used by Telemetry internally or in tests.
 
+Also, the ``event-telemetry-storage-limit-reached`` topic is notified when the event ping event
+limit is reached (configurable via the ``toolkit.telemetry.eventping.eventLimit`` preference).
+This is intended only for use internally or in tests.
+
 .. _events.event-summary:
 
 Event Summary
 =============
 
 Calling ``recordEvent`` on any non-expired registered event will accumulate to a
 :doc:`Scalar <scalars>` for ease of analysing uptake and usage patterns. Even if the event category
 isn't enabled.
new file mode 100644
--- /dev/null
+++ b/toolkit/components/telemetry/docs/data/event-ping.rst
@@ -0,0 +1,92 @@
+
+"event" ping
+=============
+
+This ping transmits :ref:`Telemetry Event <eventtelemetry>` records.
+
+The client id is submitted with this ping.
+The :doc:`Telemetry Environment <../data/environment>` is submitted in this ping.
+
+.. code-block:: js
+
+    {
+      "type": "event",
+      ... common ping data
+      "clientId": <UUID>,
+      "environment": { ... },
+      "payload": {
+        "reason": {periodic, max, shutdown}, // Why the ping was submitted
+        "processCreationTimestamp": <UNIX Timestamp>, // Minute precision, for calculating absolute time across pings
+        "sessionId": <UUID>, // For linking to "main" pings
+        "subsessionId": <UUID>, // For linking to "main" pings
+        "lostEventsCount": <number>, // How many events we had to drop. Valid only for reasons "max" and "shutdown"
+        "events": {
+          "parent": [ // process name, one of the keys from Processes.yaml
+            [timestamp, category, method, object, value, extra],
+            ... // At most 1000
+          ]
+        }
+      }
+    }
+
+Send behavior
+-------------
+
+The ping is submitted at most once per ten minute interval, and at least once per hour in
+which an event was recorded. Upon reaching 1000 events, the ping is sent immediately
+unless it would be within ten minutes of the previous ping, in which case some event
+records may be lost. A count of these lost records is included in the ping.
+to avoid losing collected data.
+
+On shutdown, during profile-before-change, a final ping is sent with any remaining event
+records, regardless of frequency but obeying the event record limit.
+
+The 1000-record limit and one-hour and ten-minute frequencies are controlled by
+:doc:`preferences <../internals/preferences>`.
+
+Field details
+-------------
+
+reason
+~~~~~~
+The ``reason`` field contains the information about why the "event" ping was submitted:
+
+* ``periodic``: The event ping was submitted because at least one event happened in the past hour.
+* ``max``: The event ping was submitted because the 1000-record limit was reached.
+* ``shutdown``: The event ping was submitted because Firefox is shutting down and some events
+  have yet to be submitted.
+
+processCreationTimestamp
+~~~~~~~~~~~~~~~~~~~~~~~~
+The minute the user's Firefox main process was created. Event record timestamps are recorded
+relative to Firefox's main process start. This provides the basis for reconstructing a user's full
+session of events in order, as well as offering a mechanism for grouping event pings.
+
+sessionId
+~~~~~~~~~~~~
+The id of the session that was current when the ping was sent.
+
+subsessionId
+~~~~~~~~~~~~
+The id of the subsession that was current when the ping was sent.
+
+.. note::
+
+  This may not be the same subsession that the events occurred in if a
+  :ref:`session split <sessionsplit>` happened in between.
+
+lostEventsCount
+~~~~~~~~~~~~~~~
+The number of events we had to discard because we reached the 1000-per-ping limit before
+we were able to send the ping. Should only have a non-zero value on "event" pings with
+reason set to "max" or "shutdown". Which events are missing should be calculable via the
+client's "main" pings using :ref:`Event Summary <events.event-summary>`.
+
+events
+~~~~~~
+A map from process names to arrays of event records that have been :ref:`serialized <events.serializationformat>`.
+
+Version History
+===============
+
+- Firefox 62: Started sending the "event" ping (`bug 1460595 <https://bugzilla.mozilla.org/show_bug.cgi?id=1460595>`_).
--- a/toolkit/components/telemetry/docs/data/main-ping.rst
+++ b/toolkit/components/telemetry/docs/data/main-ping.rst
@@ -11,16 +11,18 @@ It includes histograms and other perform
 This ping may be triggered for one of many reasons documented by the ``reason`` field:
 
 * ``aborted-session`` - this ping is regularly saved to disk (every 5 minutes), overwriting itself, and deleted at shutdown. If a previous aborted session ping is found at startup, it gets sent to the server. The first aborted-session ping is generated as soon as Telemetry starts
 * ``environment-change`` - the :doc:`environment` changed, so the session measurements got reset and a new subsession starts
 * ``shutdown`` - triggered when the browser session ends. For the first browsing session, this ping is saved to disk and sent on the next browser restart. From the second browsing session on, this ping is sent immediately on shutdown using the :doc:`../internals/pingsender`, unless the OS is shutting down
 * ``daily`` - a session split triggered in 24h hour intervals at local midnight. If an ``environment-change`` ping is generated by the time it should be sent, the daily ping is rescheduled for the next midnight
 * ``saved-session`` - the *"classic"* Telemetry payload with measurements covering the whole browser session (only submitted on Android)
 
+.. _sessionsplit:
+
 Most reasons lead to a session split, initiating a new *subsession*. We reset important measurements for those subsessions.
 
 After a new subsession split, the ``internal-telemetry-after-subsession-split`` topic is notified to all the observers. *This is an internal topic and is only meant for internal Telemetry usage.*
 
 .. note::
 
   ``saved-session`` is sent with a different ping type (``saved-session``, not ``main``), but otherwise has the same format as discussed here. As of Firefox 61 this is sent on Android only.
 
@@ -111,47 +113,39 @@ Structure:
 
 .. code-block:: js
 
     "processes" : {
       // ... other processes ...
       "parent": {
         scalars: {...},
         keyedScalars: {...},
-        events: {...},
         // parent process histograms and keyedHistograms are in main payload
       },
       "content": {
         scalars: {...},
         keyedScalars: {...},
         histograms: {...},
         keyedHistograms: {...},
-        events: {...},
       },
       "gpu": {
         // ...
       }
     }
 
 histograms and keyedHistograms
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 This section contains histograms and keyed histograms accumulated on content processes. Histograms recorded on a content child process have different character than parent histograms. For instance, ``GC_MS`` will be much different in ``processes.content`` as it has to contend with web content, whereas the instance in ``payload.histograms`` has only to contend with browser JS. Also, some histograms may be absent if never recorded on a content child process (``EVENTLOOP_UI_ACTIVITY`` is parent-process-only).
 
 This format was adopted in Firefox 51 via bug 1218576.
 
 scalars and keyedScalars
 ~~~~~~~~~~~~~~~~~~~~~~~~
 This section contains the :doc:`../collection/scalars` that are valid for the current platform. Scalars are only submitted if data was added to them, and are only reported with subsession pings. The recorded scalars are described in the `Scalars.yaml <https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Scalars.yaml>`_ file. The ``info.revision`` field indicates the revision of the file that describes the reported scalars.
 
-events
-~~~~~~
-This section contains the :ref:`eventtelemetry` that are recorded for the current subsession. Events are not always recorded, recording has to be enabled first for the Firefox session.
-
-The recorded events are defined in the `Events.yaml <https://dxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/Events.yaml>`_. The ``info.revision`` field indicates the revision of the file that describes the reported events.
-
 simpleMeasurements
 ------------------
 This section contains a list of simple measurements, or counters. In addition to the ones highlighted below, Telemetry timestamps (see `here <https://dxr.mozilla.org/mozilla-central/search?q=%22TelemetryTimestamps.add%22&redirect=false&case=true>`__ and `here <https://dxr.mozilla.org/mozilla-central/search?q=%22recordTimestamp%22&redirect=false&case=true>`__) can be reported.
 
 totalTime
 ~~~~~~~~~
 A non-monotonic integer representing the number of seconds the session has been alive.
 
@@ -695,8 +689,12 @@ Version History
 ===============
 
 - Firefox 61:
 
   - Stopped reporting ``childPayloads`` (`bug 1443599 <https://bugzilla.mozilla.org/show_bug.cgi?id=1443599>`_).
   - Stopped reporting ``saved-session`` pings on Firefox Desktop (`bug 1443603 <https://bugzilla.mozilla.org/show_bug.cgi?id=1443603>`_).
   - Stopped reporting ``simpleMeasurements.js`` (`bug 1278920 <https://bugzilla.mozilla.org/show_bug.cgi?id=1278920>`_).
   - Stopped reporting ``UITelemetry`` (`bug 1443605 <https://bugzilla.mozilla.org/show_bug.cgi?id=1443605>`_)
+
+- Firefox 62:
+
+  - ``events`` are now reported via the :doc:`../data/event-ping` (`bug 1460595 <https://bugzilla.mozilla.org/show_bug.cgi?id=1460595>`_).
--- a/toolkit/components/telemetry/docs/internals/preferences.rst
+++ b/toolkit/components/telemetry/docs/internals/preferences.rst
@@ -137,16 +137,31 @@ Preferences
 
   Enable the :doc:`../data/update-ping` on browser updates.
 
 ``toolkit.telemetry.maxEventSummaryKeys``
 
   Set the maximum number of keys per process of the :ref:`Event Summary <events.event-summary>`
   :ref:`keyed scalars <scalars.keyed-scalars>`. Default is 500. Change requires restart.
 
+``toolkit.telemetry.eventping.eventLimit``
+
+  The maximum number of event records permitted in the :doc:`../data/event-ping`.
+  Default is 1000.
+
+``toolkit.telemetry.eventping.minimumFrequency``
+
+  The minimum frequency at which an :doc:`../data/event-ping` will be sent.
+  Default is 60 (minutes).
+
+``toolkit.telemetry.eventping.maximumFrequency``
+
+  The maximum frequency at which an :doc:`../data/event-ping` will be sent.
+  Default is 10 (minutes).
+
 Data-choices notification
 -------------------------
 
 ``toolkit.telemetry.reportingpolicy.firstRun``
 
   This preference is not present until the first run. After, its value is set to false. This is used to show the infobar with a more aggressive timeout if it wasn't shown yet.
 
 ``datareporting.policy.firstRunURL``