Merge pull request #17 from matrix-org/bwindels/flagsandreadme

structure flags better and document them

README.md

@@ -2,48 +2,36 @@
 
 This repository contains tests for the matrix-react-sdk web app. The tests fire up a headless chrome and simulate user interaction (end-to-end). Note that end-to-end has little to do with the end-to-end encryption matrix supports, just that we test the full stack, going from user interaction to expected DOM in the browser.
 
-## Current tests
- - test riot loads (check title)
- - signup with custom homeserver
- - join preexisting room
-
-## Roadmap
-- get rid of jest, as a test framework won't be helpful to have a continuous flow going from one use case to another (think: do login, create a room, invite a user, ...). a test framework usually assumes the tests are semi-indepedent.
-- better error reporting (show console.log, XHR requests, partial DOM, screenshot) on error
-- cleanup helper methods
-- add more css id's/classes to riot web to make css selectors in test less brittle.
-- avoid delay when waiting for location.hash to change
-- more tests!
-- setup installing & running riot and synapse as part of the tests.
-   - Run 2 synapse instances to test federation use cases.
-   - start synapse with clean config/database on every test run
-- look into CI(Travis) integration
-- create interactive mode, where window is opened, and browser is kept open until Ctrl^C, for easy test debugging.
-
-## It's broken! How do I see what's happening in the browser?
-
-Look for this line:
-```
-puppeteer.launch();
-```
-Now change it to:
-```
-puppeteer.launch({headless: false});
-```
-
-## How to run
-
-### Setup
+## Setup
 
 Run `./install.sh`. This will:
  - install synapse, fetches the master branch at the moment. If anything fails here, please refer to the synapse README to see if you're missing one of the prerequisites.
  - install riot, this fetches the master branch at the moment.
  - install dependencies (will download copy of chrome)
 
-### Run tests
- 
+## Running the tests
+
 Run tests with `./run.sh`.
 
+### Debug tests locally.
+
+`./run.sh` will run the tests against the riot copy present in `riot/riot-web` served by a static python http server. You can symlink your `riot-web` develop copy here but that doesn't work well with webpack recompiling. You can run the test runner directly and specify parameters to get more insight into a failure or run the tests against your local webpack server.
+
+```
+./synapse/stop.sh && \
+./synapse/start.sh && \
+node start.js <parameters>
+```
+It's important to always stop and start synapse before each run of the tests to clear the in-memory sqlite database it uses, as the tests assume a blank slate.
+
+start.js accepts the following parameters that can help running the tests locally:
+
+ - `--no-logs` dont show the excessive logging show by default (meant for CI), just where the test failed.
+ - `--riot-url <url>` don't use the riot copy and static server provided by the tests, but use a running server like the webpack watch server to run the tests against. Make sure to have `welcomeUserId` disabled in your config as the tests assume there is no riot-bot currently.
+ - `--slow-mo` run the tests a bit slower, so it's easier to follow along with `--windowed`.
+ - `--windowed` run the tests in an actual browser window Try to limit interacting with the windows while the tests are running. Hovering over the window tends to fail the tests, dragging the title bar should be fine though.
+ - `--dev-tools` open the devtools in the browser window, only applies if `--windowed` is set as well.
+
 Developer Guide
 ===============
 

start.js

@@ -21,19 +21,20 @@ const scenario = require('./src/scenario');
 const program = require('commander');
 program
     .option('--no-logs', "don't output logs, document html on error", false)
-    .option('--debug', "open browser window and slow down interactions", false)
     .option('--riot-url [url]', "riot url to test", "http://localhost:5000")
+    .option('--windowed', "dont run tests headless", false)
+    .option('--slow-mo', "run tests slower to follow whats going on", false)
+    .option('--dev-tools', "open chrome devtools in browser window", false)
     .parse(process.argv);
 
 async function runTests() {
     let sessions = [];
     console.log("running tests ...");
-    const options = {};
-    if (program.debug) {
-        options.slowMo = 20;
-        options.devtools = true;
-        options.headless = false;
-    }
+    const options = {
+        slowMo: program.slowMo ? 20 : undefined,
+        devtools: program.devTools,
+        headless: !program.windowed,
+    };
     if (process.env.CHROME_PATH) {
         const path = process.env.CHROME_PATH;
         console.log(`(using external chrome/chromium at ${path}, make sure it's compatible with puppeteer)`);