Bug 1432416 - Add comments and documentation about adding new DAMP tests;r=ochameau draft
authorJulian Descottes <jdescottes@mozilla.com>
Thu, 22 Feb 2018 10:54:28 +0100
changeset 759863 2b78c3a617f54f46d9583a8f99c8b919fb3381d7
parent 759862 e9710d484d67c5649b19043840719862daab748c
child 759864 b7bb449b855a2e9bbdf5ea167fbd0d145f002bbd
push id100497
push userjdescottes@mozilla.com
push dateMon, 26 Feb 2018 19:26:03 +0000
reviewersochameau
bugs1432416
milestone60.0a1
Bug 1432416 - Add comments and documentation about adding new DAMP tests;r=ochameau MozReview-Commit-ID: LxFbX0AzJfX
devtools/docs/tests/writing-perf-tests.md
testing/talos/talos/tests/devtools/addon/content/damp.js
--- a/devtools/docs/tests/writing-perf-tests.md
+++ b/devtools/docs/tests/writing-perf-tests.md
@@ -1,13 +1,23 @@
 # Writing new performance test
 
 See [Performance tests (DAMP)](performance-tests.md) for an overall description of our performance tests.
 Here, we will describe how to write a new test with an example: track the performance of clicking inside the inspector panel.
 
+## Consider modifying existing tests first
+
+If a `custom` page already exists for the tool you are testing, try to modify the existing `custom` test rather than adding a new individual test.
+
+New individual tests run separately, in new tabs, and make DAMP slower than just modifying existing tests. Complexifying `custom` test pages should also help cover more scenarios and catch more regressions. For those reasons, modifying existing tests should be the preferred way of extending DAMP coverage.
+
+`custom` tests are using complex documents that should stress a particular tool in various ways. They are all named `custom.${tool}` (for instance `custom.inspector`). The test pages for those tests can be found in [pages/custom](https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/devtools/addon/content/pages/custom).
+
+If your test case requires a dedicated document or can't run next to the other tests in the current `custom` test, follow the instructions below to add a new individual test.
+
 ## Where is the code for the test?
 
 For now, all the tests live in a single file [damp.js](https://searchfox.org/mozilla-central/source/testing/talos/talos/tests/devtools/addon/content/damp.js).
 
 There are two kinds of tests:
 * The first kind is being run against two documents:
   * "Simple", an empty webpage. This one helps highlighting the load time of panels,
   * "Complicated", a copy of bild.be, a German newspaper website. This allows us to examine the performance of the tools when inspecting complicated, big websites.
--- a/testing/talos/talos/tests/devtools/addon/content/damp.js
+++ b/testing/talos/talos/tests/devtools/addon/content/damp.js
@@ -1127,23 +1127,26 @@ async _consoleOpenWithCachedMessagesTest
       expectedText: "Math;function ga(a,b){return a.name=b}"
     }));
 
     // Run all tests against a document specific to each tool
     tests["custom.inspector"] = this.customInspector;
     tests["custom.debugger"] = this.customDebugger;
     tests["custom.webconsole"] = this.customConsole;
 
-    // Run individual tests covering a very precise tool feature
+    // Run individual tests covering a very precise tool feature.
     tests["console.bulklog"] = this._consoleBulkLoggingTest;
     tests["console.streamlog"] = this._consoleStreamLoggingTest;
     tests["console.objectexpand"] = this._consoleObjectExpansionTest;
     tests["console.openwithcache"] = this._consoleOpenWithCachedMessagesTest;
     tests["inspector.mutations"] = this._inspectorMutationsTest;
     tests["inspector.layout"] = this._inspectorLayoutTest;
+    // ⚠  Adding new individual tests slows down DAMP execution ⚠
+    // ⚠  Consider contributing to custom.${tool} rather than adding isolated tests ⚠
+    // ⚠  See http://docs.firefox-dev.tools/tests/writing-perf-tests.html ⚠
 
     // Filter tests via `./mach --subtests filter` command line argument
     let filter = Services.prefs.getCharPref("talos.subtests", "");
     if (filter) {
       for (let name in tests) {
         if (!name.includes(filter)) {
           delete tests[name];
         }