Bug 1444049 - [marionette] Enhance documentation for Testing.md.
This fixes spelling mistakes, and adds sections for the
setup of tests, and running Marionette harness unit tests.
MozReview-Commit-ID: DO11Si67Zkp
--- a/testing/marionette/doc/Debugging.md
+++ b/testing/marionette/doc/Debugging.md
@@ -12,16 +12,24 @@ working directory. With Fennec it can b
`mach marionette test` takes a `--gecko-log` option which lets
you redirect this output stream. This is convenient if you want to
“merge” the test harness output with the stdout from the browser.
Per Unix conventions you can use `-` (dash) to have Firefox write
its log to stdout instead of file:
% ./mach marionette test --gecko-log -
+It is common to use this in conjunction with an option to increase
+the Marionette log level:
+
+ % ./mach test --gecko-log - -vv TEST
+
+A single `-v` enables debug logging, and a double `-vv` enables
+trace logging.
+
This debugging technique can be particularly effective when combined
with using [pdb] in the Python client or the JS remote debugger
that is described below.
[pdb]: https://docs.python.org/2/library/pdb.html
JavaScript debugger
--- a/testing/marionette/doc/Patches.md
+++ b/testing/marionette/doc/Patches.md
@@ -4,11 +4,15 @@ Submitting patches
You can submit patches by uploading .diff files to Bugzilla or by
sending them to [MozReview].
Once you have contributed a couple of patches, we are happy to
sponsor you in [becoming a Mozilla committer]. When you have been
granted commit access level 1 you will have permission to use the
[Firefox CI] to trigger your own “try runs” to test your changes.
+This is a good try syntax to use when testing Marionette changes:
+
+ -b do -p linux,linux64,macosx64,win64,android-api-16 -u marionette-e10s,marionette-headless-e10s,xpcshell,web-platform-tests,firefox-ui-functional -t none
+
[MozReview]: http://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview.html
[becoming a Mozilla committer]: https://www.mozilla.org/en-US/about/governance/policies/commit/
[Firefox CI]: https://treeherder.mozilla.org/
--- a/testing/marionette/doc/Testing.md
+++ b/testing/marionette/doc/Testing.md
@@ -1,68 +1,108 @@
-Running tests
-=============
+# Running Tests
We verify and test Marionette in a couple of different ways.
-Marionette has a set of [xpcshell] unit tests located in
-_testing/marionette/test_*.js_. These can be run this way:
+While for the server component XPCShell tests are used, the
+client and harness packages both use Python.
+
+## Setting up the tests
+
+Marionette-based tests can be run in-tree with a locally built
+or downloaded Firefox through `mach`, as well as with out-of-tree
+tests with `marionette`.
+
+Running in-tree tests, `mach` will automatically manage the Python
+virtual environment in which your tests are run. The Marionette
+client that is picked up is the one that is in-tree at
+_testing/marionette/client_.
- % ./mach test testing/marionette/test_*.js
+If you want to run tests from a downloaded test archive, you will
+need to download the `target.common.tests.zip` artifact as attached to
+Treeherder [build jobs] `B` for your system. Extract that file and set up
+the Python Marionette client and harness by executing the following
+command:
+
+ % pip install -r config/marionette_requirements.txt
-Because tests are run in parallell and xpcshell itself is quite
+The tests can then be found under
+*marionette/tests/testing/marionette/harness/marionette_harness/tests* and
+can be executed with the command `marionette`. It supports the same options as
+described below for `mach`.
+
+[build jobs]: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=build
+
+
+## XPCShell tests
+
+Marionette has a set of [xpcshell] unit tests located in
+_testing/marionette/test*.js_. These can be run this way:
+
+ % ./mach test testing/marionette/test_*.js
+
+Because tests are run in parallel and xpcshell itself is quite
chatty, it can sometimes be useful to run the tests sequentially:
- % ./mach test --sequential testing/marionette/test_error.js
+ % ./mach test --sequential testing/marionette/test_error.js
+
+These unit tests run as part of the `X` jobs on Treeherder.
-These unit tests run as part of the _X_ jobs on Treeherder.
+[xpcshell]: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests
+
+
+## Python functional tests
We also have a set of functional tests that make use of the Marionette
Python client. These start a Firefox process and tests the Marionette
protocol input and output. The following command will run all tests:
- % ./mach test mn
+ % ./mach marionette-test
But you can also run individual tests:
- % ./mach test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py
+ % ./mach marionette-test testing/marionette/harness/marionette_harness/tests/unit/test_navigation.py
+
+In case you want to run the tests with eg. a Nightly build of Firefox,
+mach let you do this with the --binary option:
+
+ % ./mach marionette-test --binary=/path/to/firefox-executable TEST
When working on Marionette code it is often useful to surface the
-stdout from Firefox:
-
- % ./mach test --gecko-log - TEST
-
-It is common to use this in conjunction with an option to increase
-the Marionette log level:
-
- % ./mach test --gecko-log - -vv TEST
-
-A single `-v` enables debug logging, and a double `-vv` enables
-trace logging.
+stdout from Firefox, which can be achived with the `--gecko-log` option.
+See [Debugging](Debugging.html) for usage instructions.
As these are functional integration tests and pop up Firefox windows
sporadically, a helpful tip is to surpress the window whilst you
are running them by using Firefox’ [headless mode]:
- % ./mach test -z TEST
+ % ./mach marionette-test -z TEST
`-z` is an alias for `--headless` and equivalent to setting the
`MOZ_HEADLESS` output variable. In addition to `MOZ_HEADLESS`
there is also `MOZ_HEADLESS_WIDTH` and `MOZ_HEADLESS_HEIGHT` for
controlling the dimensions of the no-op virtual display. This is
similar to using xvfb(1) which you may know from the X windowing system,
but has the additional benefit of also working on macOS and Windows.
-We have a separate page documenting how to write good Python tests in
-<doc/PythonTests.md>. These tests will run as part of the _Mn_
-job on Treeherder.
+These functional tests will run as part of the `Mn` job on Treeherder.
In addition to these two test types that specifically test the
Marionette protocol, Marionette is used as the backend for the
[geckodriver] WebDriver implementation. It is served by a WPT test
suite which effectively tests conformance to the W3C specification.
-This is a good try syntax to use when testing Marionette changes:
-
- -b do -p linux,linux64,macosx64,win64,android-api-16 -u marionette-e10s,marionette-headless-e10s,xpcshell,web-platform-tests,firefox-ui-functional-local-e10s,firefox-ui-functional-remote-e10s -t none
-
-[xpcshell]: https://developer.mozilla.org/en-US/docs/Mozilla/QA/Writing_xpcshell-based_unit_tests
[headless mode]: https://developer.mozilla.org/en-US/Firefox/Headless_mode
[geckodriver]: ../geckodriver/README.md
+
+
+## Python harness tests
+
+The Marionette harness Python package has a set of unit tests, which
+are written by using the [pytest] framework. The following command will
+run all tests:
+
+ % ./mach python-test testing/marionette/
+
+To run a specific test specify the full path to the module:
+
+ % ./mach python-test testing/marionette/harness/marionette_harness/tests/harness_unit/test_serve.py
+
+[pytest]: https://docs.pytest.org/en/latest/