diff --git a/.github/workflows/end-to-end-tests.yaml b/.github/workflows/end-to-end-tests.yaml index 51b2ba89f8..d315f22f73 100644 --- a/.github/workflows/end-to-end-tests.yaml +++ b/.github/workflows/end-to-end-tests.yaml @@ -51,9 +51,6 @@ jobs: # Run 4 instances in Parallel runner: [1, 2, 3, 4] steps: - - uses: browser-actions/setup-chrome@803ef6dfb4fdf22089c9563225d95e4a515820a0 # v1 - - run: echo "BROWSER_PATH=$(which chrome)" >> $GITHUB_ENV - # There's a 'download artifact' action, but it hasn't been updated for the workflow_run action # (https://github.com/actions/download-artifact/issues/60) so instead we get this mess: - name: 📥 Download artifact @@ -95,7 +92,19 @@ jobs: working-directory: matrix-react-sdk run: yarn install --frozen-lockfile + - name: Get installed Playwright version + run: echo "PLAYWRIGHT_VERSION=$(yarn info @playwright/test -A --json | jq -r .data.version)" >> $GITHUB_ENV + + - name: Cache playwright binaries + uses: actions/cache@v3 + id: playwright-cache + with: + path: | + ~/.cache/ms-playwright + key: ${{ runner.os }}-playwright-${{ env.PLAYWRIGHT_VERSION }} + - name: Install Playwright browsers + if: steps.playwright-cache.outputs.cache-hit != 'true' working-directory: matrix-react-sdk run: yarn playwright install --with-deps diff --git a/docs/cypress.md b/docs/cypress.md index d786e1b5c0..d71ebf0ff7 100644 --- a/docs/cypress.md +++ b/docs/cypress.md @@ -1,5 +1,9 @@ # Cypress in Element Web +# 🚨 We are moving away from Cypress in favour of Playwright. + +Please do not write any new tests in Cypress and check out [the Playwright docs](playwright.md). + ## Contents - How to run the tests diff --git a/docs/playwright.md b/docs/playwright.md new file mode 100644 index 0000000000..d6f37f147e --- /dev/null +++ b/docs/playwright.md @@ -0,0 +1,219 @@ +# Playwright in Element Web + +## Contents + +- How to run the tests +- How the tests work +- How to write great Playwright tests +- Visual testing + +## Running the Tests + +Our Playwright tests run automatically as part of our CI along with our other tests, +on every pull request and on every merge to develop & master. + +You may need to follow instructions to set up your development environment for running +Playwright by following and +. + +However the Playwright tests are run, an element-web instance must be running on +http://localhost:8080 (this is configured in `playwright.config.ts`) - this is what will +be tested. When running Playwright tests yourself, the standard `yarn start` from the +element-web project is fine: leave it running it a different terminal as you would +when developing. Alternatively if you followed the development set up from element-web then +Playwright will be capable of running the webserver on its own if it isn't already running. + +The tests use Docker to launch Homeserver (Synapse or Dendrite) instances to test against, so you'll also +need to have Docker installed and working in order to run the Playwright tests. + +There are a few different ways to run the tests yourself. The simplest is to run: + +```shell +docker pull matrixdotorg/synapse:develop +yarn run test:playwright +``` + +This will run the Playwright tests once, non-interactively. + +Note: you don't need to run the `docker pull` command every time, but you should +do it regularly to ensure you are running against an up-to-date Synapse. + +You can also run individual tests this way too, as you'd expect: + +```shell +yarn run test:playwright --spec playwright/e2e/register/register.spec.ts +``` + +Playwright also has its own UI that you can use to run and debug the tests. +To launch it: + +```shell +yarn run test:playwright:open --headed --debug +``` + +See more command line options at . + +### Running with Rust cryptography + +`matrix-js-sdk` is currently in the +[process](https://github.com/vector-im/element-web/issues/21972) of being +updated to replace its end-to-end encryption implementation to use the [Matrix +Rust SDK](https://github.com/matrix-org/matrix-rust-sdk). This is not currently +enabled by default, but it is possible to have Playwright configure Element to use +the Rust crypto implementation by passing `--project="Rust Crypto"` or using +the top left options in open mode. + +## How the Tests Work + +Everything Playwright-related lives in the `playwright/` subdirectory of react-sdk +as is typical for Playwright tests. Likewise, tests live in `playwright/e2e`. + +`playwright/plugins/homeservers` contains Playwright plugins that starts instances +of Synapse/Dendrite in Docker containers. These servers are what Element-web runs +against in the tests. + +Synapse can be launched with different configurations in order to test element +in different configurations. `playwright/plugins/homeserver/synapse/templates` +contains template configuration files for each different configuration. + +Each test suite can then launch whatever Synapse instances it needs in whatever +configurations. + +Note that although tests should stop the Homeserver instances after running and the +plugin also stop any remaining instances after all tests have run, it is possible +to be left with some stray containers if, for example, you terminate a test such +that the `after()` does not run and also exit Playwright uncleanly. All the containers +it starts are prefixed, so they are easy to recognise. They can be removed safely. + +After each test run, logs from the Synapse instances are saved in `playwright/synapselogs` +with each instance in a separate directory named after its ID. These logs are removed +at the start of each test run. + +## Writing Tests + +Mostly this is the same advice as for writing any other Playwright test: the Playwright +docs are well worth a read if you're not already familiar with Playwright testing, eg. +https://playwright.dev/docs/best-practices. To avoid your tests being flaky it is also +recommended to use [auto-retrying assertions](https://playwright.dev/docs/test-assertions#auto-retrying-assertions). + +### Getting a Synapse + +We heavily leverage the magic of [Playwright fixtures](https://playwright.dev/docs/test-fixtures). +To acquire a homeserver within a test just add the `homeserver` fixture to the test: + +```typescript +test("should do something", async ({ homeserver }) => { + // homeserver is a Synapse/Dendrite instance +}); +``` + +This returns an object with information about the Homeserver instance, including what port +it was started on and the ID that needs to be passed to shut it down again. It also +returns the registration shared secret (`registrationSecret`) that can be used to +register users via the REST API. The Homeserver has been ensured ready to go by awaiting +its internal health-check. + +Homeserver instances should be reasonably cheap to start (you may see the first one take a +while as it pulls the Docker image). +You do not need to explicitly clean up the instance as it will be cleaned up by the fixture. + +### Synapse Config Templates + +When a Synapse instance is started, it's given a config generated from one of the config +templates in `playwright/plugins/homeserver/synapse/templates`. There are a couple of special files +in these templates: + +- `homeserver.yaml`: + Template substitution happens in this file. Template variables are: + - `REGISTRATION_SECRET`: The secret used to register users via the REST API. + - `MACAROON_SECRET_KEY`: Generated each time for security + - `FORM_SECRET`: Generated each time for security + - `PUBLIC_BASEURL`: The localhost url + port combination the synapse is accessible at +- `localhost.signing.key`: A signing key is auto-generated and saved to this file. + Config templates should not contain a signing key and instead assume that one will exist + in this file. + +All other files in the template are copied recursively to `/data/`, so the file `foo.html` +in a template can be referenced in the config as `/data/foo.html`. + +### Logging In + +We again heavily leverage the magic of [Playwright fixtures](https://playwright.dev/docs/test-fixtures). +To acquire a logged-in user within a test just add the `user` fixture to the test: + +```typescript +test("should do something", async ({ user }) => { + // user is a logged in user +}); +``` + +You can specify a display name for the user via `test.use` `displayName`, +otherwise a random one will be generated. +This will register a random userId using the registrationSecret with a random password +and the given display name. The user fixture will contain details about the credentials for if +they are needed for User-Interactive Auth or similar but localStorage will already be seeded with them +and the app loaded (path `/`). + +### Joining a Room + +Many tests will also want to start with the client in a room, ready to send & receive messages. Best +way to do this may be to get an access token for the user and use this to create a room with the REST +API before logging the user in. +You can make use of the bot fixture and the `client` field on the app fixture to do this. + +### Try to write tests from the users' perspective + +Like for instance a user will not look for a button by querying a CSS selector. +Instead, you should work with roles / labels etc, see https://playwright.dev/docs/locators. + +### Using matrix-js-sdk + +Due to the way we run the Playwright tests in CI, at this time you can only use the matrix-js-sdk module +exposed on `window.matrixcs`. This has the limitation that it is only accessible with the app loaded. +This may be revisited in the future. + +## Good Test Hygiene + +This section mostly summarises general good Playwright testing practice, and should not be news to anyone +already familiar with Playwright. + +1. Test a well-isolated unit of functionality. The more specific, the easier it will be to tell what's + wrong when they fail. +1. Don't depend on state from other tests: any given test should be able to run in isolation. +1. Try to avoid driving the UI for anything other than the UI you're trying to test. e.g. if you're + testing that the user can send a reaction to a message, it's best to send a message using a REST + API, then react to it using the UI, rather than using the element-web UI to send the message. +1. Avoid explicit waits. Playwright locators & assertions will implicitly wait for the specified + element to appear and all assertions are retried until they either pass or time out, so you should + never need to manually wait for an element. + - For example, for asserting about editing an already-edited message, you can't wait for the + 'edited' element to appear as there was already one there, but you can assert that the body + of the message is what is should be after the second edit and this assertion will pass once + it becomes true. You can then assert that the 'edited' element is still in the DOM. + - You can also wait for other things like network requests in the + browser to complete (https://playwright.dev/docs/api/class-page#page-wait-for-response). + Needing to wait for things can also be because of race conditions in the app itself, which ideally + shouldn't be there! + +This is a small selection - the Playwright best practices guide, linked above, has more good advice, and we +should generally try to adhere to them. + +## Screenshot testing + +When we previously used Cypress we also dabbled with Percy, and whilst powerful it did not +lend itself well to being executed on all PRs without needing to budget it substantially. + +Playwright has built-in support for [visual comparison testing](https://playwright.dev/docs/test-snapshots). +Screenshots are saved in `playwright/snapshots` and are rendered in a Linux Docker environment for stability. + +One must be careful to exclude any dynamic content from the screenshot, such as timestamps, avatars, etc, +via the `mask` option. See the [Playwright docs](https://playwright.dev/docs/test-snapshots#masking). + +Some UI elements render differently between test runs, such as BaseAvatar when +there is no avatar set, choosing a colour from the theme palette based on the +hash of the user/room's Matrix ID. To avoid this creating flaky tests we inject +some custom CSS, for this to happen we use the custom assertion `toMatchScreenshot` +instead of the native `toHaveScreenshot`. + +If you are running Linux and are unfortunate that the screenshots are not rendering identically, +you may wish to specify `--ignore-snapshots` and rely on Docker to render them for you. diff --git a/package.json b/package.json index 8f08c2afc9..1ade36cf58 100644 --- a/package.json +++ b/package.json @@ -55,6 +55,9 @@ "test:cypress:open": "cypress open", "test:playwright": "playwright test", "test:playwright:open": "yarn test:playwright --ui", + "test:playwright:screenshots": "yarn test:playwright:screenshots:build && yarn test:playwright:screenshots:run", + "test:playwright:screenshots:build": "docker build playwright -t matrix-react-sdk-playwright", + "test:playwright:screenshots:run": "docker run --rm --network host -v $(pwd)/../:/work/ -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/:/tmp/ -it matrix-react-sdk-playwright", "coverage": "yarn test --coverage", "lint:workflows": "find .github/workflows -type f \\( -iname '*.yaml' -o -iname '*.yml' \\) | xargs -I {} sh -c 'echo \"Linting {}\"; action-validator \"{}\"'" }, diff --git a/playwright/README.md b/playwright/README.md deleted file mode 100644 index ee110a1135..0000000000 --- a/playwright/README.md +++ /dev/null @@ -1,13 +0,0 @@ -To update snapshots you will need to run Playwright on a Linux machine. -If you have access to docker then you can use the following: - -```shell -docker build playwright -t matrix-react-sdk-playwright -docker run \ - --rm \ - --network host \ - -v $(pwd)/../:/work/ \ - -v /var/run/docker.sock:/var/run/docker.sock \ - -v /tmp/:/tmp/ \ - -it matrix-react-sdk-playwright -```