Bug 1444049 - [marionette] Enhance documentation for Testing.md. draft
authorHenrik Skupin <mail@hskupin.info>
Thu, 08 Mar 2018 14:33:03 +0100
changeset 765188 1e2b9ee3caca84c038178ce3a4e9cb066b356e0d
parent 764770 a6a32fb286fa9e5d5f6d5b3b77423ab6b96c9502
child 765189 41f5fd476a545df1b2550c8c8afe16e390722da8
push id101996
push userbmo:hskupin@gmail.com
push dateFri, 09 Mar 2018 08:41:30 +0000
bugs1444049
milestone60.0a1
Bug 1444049 - [marionette] Enhance documentation for Testing.md. This fixes spelling mistakes, and adds sections for the setup of tests, and running Marionette harness unit tests. MozReview-Commit-ID: DO11Si67Zkp
testing/marionette/doc/Debugging.md
testing/marionette/doc/Patches.md
testing/marionette/doc/Testing.md
--- a/testing/marionette/doc/Debugging.md
+++ b/testing/marionette/doc/Debugging.md
@@ -12,16 +12,24 @@ working directory.  With Fennec it can b
 `mach marionette test` takes a `--gecko-log` option which lets
 you redirect this output stream.  This is convenient if you want to
 “merge” the test harness output with the stdout from the browser.
 Per Unix conventions you can use `-` (dash) to have Firefox write
 its log to stdout instead of file:
 
 	% ./mach marionette test --gecko-log -
 
+It is common to use this in conjunction with an option to increase
+the Marionette log level:
+
+  % ./mach test --gecko-log - -vv TEST
+
+A single `-v` enables debug logging, and a double `-vv` enables
+trace logging.
+
 This debugging technique can be particularly effective when combined
 with using [pdb] in the Python client or the JS remote debugger
 that is described below.
 
 [pdb]: https://docs.python.org/2/library/pdb.html
 
 
 JavaScript debugger
--- a/testing/marionette/doc/Patches.md
+++ b/testing/marionette/doc/Patches.md
@@ -4,11 +4,15 @@ Submitting patches
 You can submit patches by uploading .diff files to Bugzilla or by
 sending them to [MozReview].
 
 Once you have contributed a couple of patches, we are happy to
 sponsor you in [becoming a Mozilla committer].  When you have been
 granted commit access level 1 you will have permission to use the
 [Firefox CI] to trigger your own “try runs” to test your changes.
 
+This is a good try syntax to use when testing Marionette changes:
+
+    -b do -p linux,linux64,macosx64,win64,android-api-16 -u marionette-e10s,marionette-headless-e10s,xpcshell,web-platform-tests,firefox-ui-functional -t none
+
 [MozReview]: http://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview.html
 [becoming a Mozilla committer]: https://www.mozilla.org/en-US/about/governance/policies/commit/
 [Firefox CI]: https://treeherder.mozilla.org/
--- a/testing/marionette/doc/Testing.md
+++ b/testing/marionette/doc/Testing.md
@@ -1,68 +1,108 @@
-Running tests
-=============
+# Running Tests
 
 We verify and test Marionette in a couple of different ways.
-Marionette has a set of [xpcshell] unit tests located in
-_testing/marionette/test_*.js_.  These can be run this way:
+While for the server component XPCShell tests are used, the
+client and harness packages both use Python.
+
+## Setting up the tests
+
+Marionette-based tests can be run in-tree with a locally built
+or downloaded Firefox through `mach`, as well as with out-of-tree
+tests with `marionette`.
+
+Running in-tree tests, `mach` will automatically manage the Python
+virtual environment in which your tests are run.  The Marionette
+client that is picked up is the one that is in-tree at
+_testing/marionette/client_.
 
-	% ./mach test testing/marionette/test_*.js
+If you want to run tests from a downloaded test archive, you will
+need to download the `target.common.tests.zip` artifact as attached to
+Treeherder [build jobs] `B` for your system.  Extract that file and set up
+the Python Marionette client and harness by executing the following
+command:
+
+    % pip install -r config/marionette_requirements.txt
 
-Because tests are run in parallell and xpcshell itself is quite
+The tests can then be found under
+*marionette/tests/testing/marionette/harness/marionette_harness/tests* and
+can be executed with the command `marionette`.  It supports the same options as
+described below for `mach`.
+
+[build jobs]: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=build
+
+
+## XPCShell tests
+
+Marionette has a set of [xpcshell] unit tests located in
+_testing/marionette/test*.js_.  These can be run this way:
+
+    % ./mach test testing/marionette/test_*.js
+
+Because tests are run in parallel and xpcshell itself is quite
 chatty, it can sometimes be useful to run the tests sequentially:
 
-	% ./mach test --sequential testing/marionette/test_error.js
+    % ./mach test --sequential testing/marionette/test_error.js
+
+These unit tests run as part of the `X` jobs on Treeherder.
 
-These unit tests run as part of the _X_ jobs on Treeherder.
+[xpcshell]: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests
+
+
+## Python functional tests
 
 We also have a set of functional tests that make use of the Marionette
 Python client.  These start a Firefox process and tests the Marionette
 protocol input and output.  The following command will run all tests:
 
-	% ./mach test mn
+    % ./mach marionette-test
 
 But you can also run individual tests:
 
-	% ./mach test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py
+    % ./mach marionette-test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py
+
+In case you want to run the tests with eg. a Nightly build of Firefox,
+mach let you do this with the --binary option:
+
+    % ./mach marionette-test --binary=/path/to/firefox-executable TEST
 
 When working on Marionette code it is often useful to surface the
-stdout from Firefox:
-
-	% ./mach test --gecko-log - TEST
-
-It is common to use this in conjunction with an option to increase
-the Marionette log level:
-
-	% ./mach test --gecko-log - -vv TEST
-
-A single `-v` enables debug logging, and a double `-vv` enables
-trace logging.
+stdout from Firefox, which can be achived with the `--gecko-log` option.
+See [Debugging](Debugging.html) for usage instructions.
 
 As these are functional integration tests and pop up Firefox windows
 sporadically, a helpful tip is to surpress the window whilst you
 are running them by using Firefox’ [headless mode]:
 
-	% ./mach test -z TEST
+    % ./mach marionette-test -z TEST
 
 `-z` is an alias for `--headless` and equivalent to setting the
 `MOZ_HEADLESS` output variable.  In addition to `MOZ_HEADLESS`
 there is also `MOZ_HEADLESS_WIDTH` and `MOZ_HEADLESS_HEIGHT` for
 controlling the dimensions of the no-op virtual display.  This is
 similar to using xvfb(1) which you may know from the X windowing system,
 but has the additional benefit of also working on macOS and Windows.
 
-We have a separate page documenting how to write good Python tests in
-<doc/PythonTests.md>.  These tests will run as part of the _Mn_
-job on Treeherder.
+These functional tests will run as part of the `Mn` job on Treeherder.
 
 In addition to these two test types that specifically test the
 Marionette protocol, Marionette is used as the backend for the
 [geckodriver] WebDriver implementation.  It is served by a WPT test
 suite which effectively tests conformance to the W3C specification.
 
-This is a good try syntax to use when testing Marionette changes:
-
-	-b do -p linux,linux64,macosx64,win64,android-api-16 -u marionette-e10s,marionette-headless-e10s,xpcshell,web-platform-tests,firefox-ui-functional-local-e10s,firefox-ui-functional-remote-e10s -t none
-
-[xpcshell]: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests
 [headless mode]: https://developer.mozilla.org/en-US/Firefox/Headless_mode
 [geckodriver]: ../geckodriver/README.md
+
+
+## Python harness tests
+
+The Marionette harness Python package has a set of unit tests, which
+are written by using the [pytest] framework. The following command will
+run all tests:
+
+    % ./mach python-test testing/marionette/
+
+To run a specific test specify the full path to the module:
+
+    % ./mach python-test testing/marionette/harness/marionette_harness/tests/harness_unit/test_serve.py
+
+[pytest]: https://docs.pytest.org/en/latest/