2022-05-03 13:46:36 +02:00
|
|
|
# Cypress in Element Web
|
|
|
|
|
2023-09-05 13:55:53 +02:00
|
|
|
## Contents
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2023-09-05 13:55:53 +02:00
|
|
|
- How to run the tests
|
2022-12-12 12:24:14 +01:00
|
|
|
- How the tests work
|
|
|
|
- How to write great Cypress tests
|
|
|
|
- Visual testing
|
2022-05-03 13:46:36 +02:00
|
|
|
|
|
|
|
## Running the Tests
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2022-05-03 13:46:36 +02:00
|
|
|
Our Cypress tests run automatically as part of our CI along with our other tests,
|
2022-05-04 16:11:33 +02:00
|
|
|
on every pull request and on every merge to develop & master.
|
2022-05-03 13:46:36 +02:00
|
|
|
|
|
|
|
However the Cypress tests are run, an element-web must be running on
|
|
|
|
http://localhost:8080 (this is configured in `cypress.json`) - this is what will
|
|
|
|
be tested. When running Cypress tests yourself, the standard `yarn start` from the
|
|
|
|
element-web project is fine: leave it running it a different terminal as you would
|
|
|
|
when developing.
|
|
|
|
|
2023-01-11 00:29:56 +01:00
|
|
|
The tests use Docker to launch Homeserver (Synapse or Dendrite) instances to test against, so you'll also
|
2022-05-03 13:46:36 +02:00
|
|
|
need to have Docker installed and working in order to run the Cypress tests.
|
|
|
|
|
|
|
|
There are a few different ways to run the tests yourself. The simplest is to run:
|
|
|
|
|
|
|
|
```
|
|
|
|
yarn run test:cypress
|
|
|
|
```
|
|
|
|
|
|
|
|
This will run the Cypress tests once, non-interactively.
|
|
|
|
|
|
|
|
You can also run individual tests this way too, as you'd expect:
|
|
|
|
|
|
|
|
```
|
2022-07-11 13:23:06 +02:00
|
|
|
yarn run test:cypress --spec cypress/e2e/1-register/register.spec.ts
|
2022-05-03 13:46:36 +02:00
|
|
|
```
|
|
|
|
|
|
|
|
Cypress also has its own UI that you can use to run and debug the tests.
|
|
|
|
To launch it:
|
|
|
|
|
|
|
|
```
|
|
|
|
yarn run test:cypress:open
|
|
|
|
```
|
|
|
|
|
2023-09-05 13:55:53 +02:00
|
|
|
### Matching the CI environment
|
|
|
|
|
|
|
|
In our Continuous Integration environment, we run the Cypress tests in the
|
|
|
|
Chrome browser.
|
|
|
|
|
|
|
|
In some rare cases, tests behave differently between different browsers, so if
|
|
|
|
you see CI failures for the Cypress tests, but those tests work OK on your local
|
|
|
|
machine, try running them in Chrome like this:
|
|
|
|
|
|
|
|
```bash
|
|
|
|
yarn run test:cypress --browser=chrome
|
|
|
|
```
|
|
|
|
|
|
|
|
(Use `--browser=chromium` if you'd prefer to use Chromium.)
|
|
|
|
|
|
|
|
If you launch the interactive UI you can choose the browser you want to use. To
|
|
|
|
match the CI setup, choose Chrome.
|
|
|
|
|
|
|
|
Note that you will need to have Chrome installed on your system to run the tests
|
|
|
|
inside those browsers, whereas the default is to use Electron, which is included
|
|
|
|
within the Cypress dependency.
|
|
|
|
|
2023-05-19 15:55:13 +02:00
|
|
|
### Running with Rust cryptography
|
|
|
|
|
|
|
|
`matrix-js-sdk` is currently in the
|
|
|
|
[process](https://github.com/vector-im/element-web/issues/21972) of being
|
|
|
|
updated to replace its end-to-end encryption implementation to use the [Matrix
|
|
|
|
Rust SDK](https://github.com/matrix-org/matrix-rust-sdk). This is not currently
|
|
|
|
enabled by default, but it is possible to have Cypress configure Element to use
|
|
|
|
the Rust crypto implementation by setting the environment variable
|
|
|
|
`CYPRESS_RUST_CRYPTO=1`.
|
|
|
|
|
2022-05-03 13:46:36 +02:00
|
|
|
## How the Tests Work
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2022-05-03 13:46:36 +02:00
|
|
|
Everything Cypress-related lives in the `cypress/` subdirectory of react-sdk
|
2022-07-11 13:23:06 +02:00
|
|
|
as is typical for Cypress tests. Likewise, tests live in `cypress/e2e`.
|
2022-05-03 13:46:36 +02:00
|
|
|
|
|
|
|
`cypress/plugins/synapsedocker` contains a Cypress plugin that starts instances
|
|
|
|
of Synapse in Docker containers. These synapses are what Element-web runs against
|
|
|
|
in the Cypress tests.
|
|
|
|
|
|
|
|
Synapse can be launched with different configurations in order to test element
|
|
|
|
in different configurations. `cypress/plugins/synapsedocker/templates` contains
|
|
|
|
template configuration files for each different configuration.
|
|
|
|
|
2023-01-11 00:29:56 +01:00
|
|
|
Each test suite can then launch whatever Synapse instances it needs in whatever
|
2022-05-03 13:46:36 +02:00
|
|
|
configurations.
|
|
|
|
|
2023-01-11 00:29:56 +01:00
|
|
|
Note that although tests should stop the Homeserver instances after running and the
|
2022-05-03 13:46:36 +02:00
|
|
|
plugin also stop any remaining instances after all tests have run, it is possible
|
|
|
|
to be left with some stray containers if, for example, you terminate a test such
|
|
|
|
that the `after()` does not run and also exit Cypress uncleanly. All the containers
|
2022-05-04 16:11:33 +02:00
|
|
|
it starts are prefixed, so they are easy to recognise. They can be removed safely.
|
2022-05-03 13:46:36 +02:00
|
|
|
|
2022-05-04 16:11:33 +02:00
|
|
|
After each test run, logs from the Synapse instances are saved in `cypress/synapselogs`
|
|
|
|
with each instance in a separate directory named after its ID. These logs are removed
|
2022-05-03 13:46:36 +02:00
|
|
|
at the start of each test run.
|
|
|
|
|
|
|
|
## Writing Tests
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2022-05-03 13:46:36 +02:00
|
|
|
Mostly this is the same advice as for writing any other Cypress test: the Cypress
|
|
|
|
docs are well worth a read if you're not already familiar with Cypress testing, eg.
|
2022-11-16 18:51:27 +01:00
|
|
|
https://docs.cypress.io/guides/references/best-practices. To avoid your tests being
|
|
|
|
flaky it is also recommended to give https://docs.cypress.io/guides/core-concepts/retry-ability
|
|
|
|
a read.
|
2022-05-03 13:46:36 +02:00
|
|
|
|
|
|
|
### Getting a Synapse
|
2022-12-12 12:24:14 +01:00
|
|
|
|
|
|
|
The key difference is in starting Synapse instances. Tests use this plugin via
|
2023-01-11 00:29:56 +01:00
|
|
|
`cy.startHomeserver()` to provide a Homeserver instance to log into:
|
2022-05-03 13:46:36 +02:00
|
|
|
|
2022-05-04 16:11:33 +02:00
|
|
|
```javascript
|
2023-01-11 00:29:56 +01:00
|
|
|
cy.startHomeserver("consent").then((result) => {
|
|
|
|
homeserver = result;
|
2022-05-03 13:46:36 +02:00
|
|
|
});
|
|
|
|
```
|
|
|
|
|
2023-01-11 00:29:56 +01:00
|
|
|
This returns an object with information about the Homeserver instance, including what port
|
2022-05-03 13:46:36 +02:00
|
|
|
it was started on and the ID that needs to be passed to shut it down again. It also
|
|
|
|
returns the registration shared secret (`registrationSecret`) that can be used to
|
2023-01-11 00:29:56 +01:00
|
|
|
register users via the REST API. The Homeserver has been ensured ready to go by awaiting
|
2022-05-04 16:11:33 +02:00
|
|
|
its internal health-check.
|
2022-05-03 13:46:36 +02:00
|
|
|
|
2023-01-11 00:29:56 +01:00
|
|
|
Homeserver instances should be reasonably cheap to start (you may see the first one take a
|
2022-05-03 13:46:36 +02:00
|
|
|
while as it pulls the Docker image), so it's generally expected that tests will start a
|
2023-01-11 00:29:56 +01:00
|
|
|
Homeserver instance for each test suite, i.e. in `before()`, and then tear it down in `after()`.
|
2022-05-04 16:11:33 +02:00
|
|
|
|
2023-01-11 00:29:56 +01:00
|
|
|
To later destroy your Homeserver you should call `stopHomeserver`, passing the HomeserverInstance
|
2022-05-04 16:11:33 +02:00
|
|
|
object you received when starting it.
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2022-05-04 16:11:33 +02:00
|
|
|
```javascript
|
2023-01-11 00:29:56 +01:00
|
|
|
cy.stopHomeserver(homeserver);
|
2022-05-04 16:11:33 +02:00
|
|
|
```
|
2022-05-03 13:46:36 +02:00
|
|
|
|
|
|
|
### Synapse Config Templates
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2022-05-03 13:46:36 +02:00
|
|
|
When a Synapse instance is started, it's given a config generated from one of the config
|
|
|
|
templates in `cypress/plugins/synapsedocker/templates`. There are a couple of special files
|
|
|
|
in these templates:
|
2022-12-12 12:24:14 +01:00
|
|
|
|
|
|
|
- `homeserver.yaml`:
|
|
|
|
Template substitution happens in this file. Template variables are:
|
|
|
|
- `REGISTRATION_SECRET`: The secret used to register users via the REST API.
|
|
|
|
- `MACAROON_SECRET_KEY`: Generated each time for security
|
|
|
|
- `FORM_SECRET`: Generated each time for security
|
|
|
|
- `PUBLIC_BASEURL`: The localhost url + port combination the synapse is accessible at
|
|
|
|
- `localhost.signing.key`: A signing key is auto-generated and saved to this file.
|
|
|
|
Config templates should not contain a signing key and instead assume that one will exist
|
|
|
|
in this file.
|
2022-05-03 13:46:36 +02:00
|
|
|
|
|
|
|
All other files in the template are copied recursively to `/data/`, so the file `foo.html`
|
|
|
|
in a template can be referenced in the config as `/data/foo.html`.
|
|
|
|
|
|
|
|
### Logging In
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2022-05-04 16:11:33 +02:00
|
|
|
There exists a basic utility to start the app with a random user already logged in:
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2022-05-04 16:11:33 +02:00
|
|
|
```javascript
|
2023-01-11 00:29:56 +01:00
|
|
|
cy.initTestUser(homeserver, "Jeff");
|
2022-05-04 16:11:33 +02:00
|
|
|
```
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2023-01-11 00:29:56 +01:00
|
|
|
It takes the HomeserverInstance you received from `startHomeserver` and a display name for your test user.
|
2022-05-04 16:11:33 +02:00
|
|
|
This custom command will register a random userId using the registrationSecret with a random password
|
|
|
|
and the given display name. The returned Chainable will contain details about the credentials for if
|
|
|
|
they are needed for User-Interactive Auth or similar but localStorage will already be seeded with them
|
|
|
|
and the app loaded (path `/`).
|
|
|
|
|
|
|
|
The internals of how this custom command run may be swapped out later,
|
|
|
|
but the signature can be maintained for simpler maintenance.
|
2022-05-03 13:46:36 +02:00
|
|
|
|
|
|
|
### Joining a Room
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2022-05-03 13:46:36 +02:00
|
|
|
Many tests will also want to start with the client in a room, ready to send & receive messages. Best
|
|
|
|
way to do this may be to get an access token for the user and use this to create a room with the REST
|
2023-01-11 00:29:56 +01:00
|
|
|
API before logging the user in. You can make use of `cy.getBot(homeserver)` and `cy.getClient()` to do this.
|
2022-05-03 13:46:36 +02:00
|
|
|
|
|
|
|
### Convenience APIs
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2023-01-11 00:29:56 +01:00
|
|
|
We should probably end up with convenience APIs that wrap the homeserver creation, logging in and room
|
2022-05-03 13:46:36 +02:00
|
|
|
creation that can be called to set up tests.
|
|
|
|
|
2023-03-29 08:47:05 +02:00
|
|
|
### Try to write tests from the users's perspective
|
|
|
|
|
|
|
|
Like for instance a user will not look for a button by querying a CSS selector. Instead you should work
|
|
|
|
with roles / labels etc.. You can make use of `cy.findBy…` queries provided by
|
2023-04-12 09:26:12 +02:00
|
|
|
[Cypress Testing Library](https://github.com/testing-library/cypress-testing-library).
|
2023-03-29 08:47:05 +02:00
|
|
|
|
2022-05-10 19:09:31 +02:00
|
|
|
### Using matrix-js-sdk
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2022-05-10 19:09:31 +02:00
|
|
|
Due to the way we run the Cypress tests in CI, at this time you can only use the matrix-js-sdk module
|
|
|
|
exposed on `window.matrixcs`. This has the limitation that it is only accessible with the app loaded.
|
|
|
|
This may be revisited in the future.
|
|
|
|
|
2022-05-03 13:46:36 +02:00
|
|
|
## Good Test Hygiene
|
2022-12-12 12:24:14 +01:00
|
|
|
|
2022-05-03 13:46:36 +02:00
|
|
|
This section mostly summarises general good Cypress testing practice, and should not be news to anyone
|
|
|
|
already familiar with Cypress.
|
|
|
|
|
|
|
|
1. Test a well-isolated unit of functionality. The more specific, the easier it will be to tell what's
|
2022-05-04 16:11:33 +02:00
|
|
|
wrong when they fail.
|
2022-05-03 13:46:36 +02:00
|
|
|
1. Don't depend on state from other tests: any given test should be able to run in isolation.
|
2022-05-04 16:11:33 +02:00
|
|
|
1. Try to avoid driving the UI for anything other than the UI you're trying to test. e.g. if you're
|
2022-05-03 13:46:36 +02:00
|
|
|
testing that the user can send a reaction to a message, it's best to send a message using a REST
|
|
|
|
API, then react to it using the UI, rather than using the element-web UI to send the message.
|
|
|
|
1. Avoid explicit waits. `cy.get()` will implicitly wait for the specified element to appear and
|
|
|
|
all assertions are retired until they either pass or time out, so you should never need to
|
|
|
|
manually wait for an element.
|
2022-12-12 12:24:14 +01:00
|
|
|
- For example, for asserting about editing an already-edited message, you can't wait for the
|
2022-05-03 13:46:36 +02:00
|
|
|
'edited' element to appear as there was already one there, but you can assert that the body
|
|
|
|
of the message is what is should be after the second edit and this assertion will pass once
|
|
|
|
it becomes true. You can then assert that the 'edited' element is still in the DOM.
|
2022-12-12 12:24:14 +01:00
|
|
|
- You can also wait for other things like network requests in the
|
2022-05-03 13:46:36 +02:00
|
|
|
browser to complete (https://docs.cypress.io/guides/guides/network-requests#Waiting).
|
|
|
|
Needing to wait for things can also be because of race conditions in the app itself, which ideally
|
|
|
|
shouldn't be there!
|
|
|
|
|
|
|
|
This is a small selection - the Cypress best practices guide, linked above, has more good advice, and we
|
|
|
|
should generally try to adhere to them.
|
2022-05-17 17:16:14 +02:00
|
|
|
|
2023-03-27 09:18:11 +02:00
|
|
|
## Screenshot testing with Percy
|
|
|
|
|
|
|
|
We also support visual testing via [Percy](https://percy.io). Within many of our
|
|
|
|
Cypress tests you can see lines calling `cy.percySnapshot()`. This creates a
|
|
|
|
screenshot and uses Percy to check whether it has changed from the last time
|
|
|
|
this test was run.
|
|
|
|
|
|
|
|
It can help to pass `percyCSS` in as the 2nd argument to `percySnapshot` to hide
|
|
|
|
elements that vary (e.g. timestamps). See the existing code for examples of
|
|
|
|
this. (Note: it is also possible for team members to mark certain parts of a
|
|
|
|
screenshot to be ignored. This is done within the Percy UI.)
|
|
|
|
|
|
|
|
Percy screenshots are created using custom renderers based on Safari, Firefox,
|
|
|
|
Chrome and Edge. Each `percySnapshot` actually creates 8 screenshots (4
|
|
|
|
browsers, 2 sizes). Since we have a limited budget for Percy screenshots, by
|
|
|
|
default we only run Percy once per day against the `develop` branch, based on a
|
|
|
|
nightly build at approximately 04:00 UTC every day. (The schedule is defined in
|
|
|
|
[element-web.yaml](../.github/workflows/element-web.yaml) and the Percy tests are
|
|
|
|
enabled/disabled in [cypress.yaml](../.github/workflows/cypress.yaml).)
|
|
|
|
|
|
|
|
If your pull request makes visual changes, you are encouraged to request Percy
|
2023-04-06 16:30:56 +02:00
|
|
|
to run by adding the label `X-Needs-Percy` to the PR, these will only run in
|
|
|
|
the merge queue to save snapshots. This will help us find any
|
2023-03-27 09:18:11 +02:00
|
|
|
visual bugs or validate visual changes at the time they are made, instead of
|
|
|
|
having to figure it out later after the nightly build. If you don't have
|
|
|
|
permission to add a label, please ask your reviewer to do it. Note: it's best to
|
|
|
|
add this label when the change is nearly ready, because the screenshots will be
|
|
|
|
re-created every time you make a change to your PR.
|
2023-07-12 16:57:01 +02:00
|
|
|
|
|
|
|
Some UI elements render differently between test runs, such as BaseAvatar when
|
|
|
|
there is no avatar set, choosing a colour from the theme palette based on the
|
|
|
|
hash of the user/room's Matrix ID. To avoid this creating flaky tests we can use
|
|
|
|
the `@media only percy` CSS query to override the variable colour into a fixed one
|
|
|
|
for tests where it is not feasible to fix the underlying identifiers issued by the
|
|
|
|
server. See https://docs.percy.io/docs/percy-specific-css#percy-css-media-query.
|