Commit Graph

182 Commits (bb39fc4366eb59464838d99a676b1620b879c2b8)

Author SHA1 Message Date
Erik Johnston c85c5ace52
Add rust to CI (#13763) 2022-09-09 11:29:04 +01:00
Erik Johnston 1cc729c177
Fix latest deps (#13743) 2022-09-08 13:58:31 +01:00
Erik Johnston 8d7fcf9b76
Fix latest deps CI (#13734) 2022-09-07 14:07:06 +00:00
Erik Johnston dc0e896b68
Add some rust caching to CI (#13735) 2022-09-07 13:56:59 +00:00
Erik Johnston c9b7e97355
Add a stub Rust crate (#12595) 2022-09-06 19:01:37 +01:00
Erik Johnston b5effc7201
Update trial old deps CI to use poetry 1.2.0 (#13707) 2022-09-06 11:43:04 +00:00
Patrick Cloke 32fc3b7ba4
Remove configuration options for direct TCP replication. (#13647)
Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.
2022-09-06 07:50:02 +00:00
Erik Johnston 8edf3f66d5
Reduce number of CI jobs run on PRs (#13713)
* Reduce number of CI jobs run on PRs

* Newsfile

* Also limit sytest jobs

* Fix typo

* Fix up

* Fixup
2022-09-05 13:31:42 +01:00
Erik Johnston 0fdb685c2b
Only run trial CI on all python versions on non-PRs (#13698) 2022-09-02 13:12:47 +01:00
Erik Johnston 18e4092801
Bump docker image to use stable poetry version (#13688) 2022-09-01 13:46:47 +01:00
Richard van der Hoff 4f6de33f41
Print complement failure results last (#13639)
Since github always scrolls to the bottom of any test output, let's put the
failed tests last and hide any successful packages.
2022-08-28 20:05:30 +00:00
Kat Gerasimova a282446502
Update automation for incoming issues (#13629)
GitHub appears to be deprecating addProjectNextItem by not allowing it to be used alongside projectV2 to get the project ID, so switching to using addProjectV2ItemById instead.
2022-08-25 12:09:23 +01:00
Andy Balaam 371db86a86
First draft of triage_labelled action (#13612) 2022-08-24 13:59:33 +01:00
Kat Gerasimova 2e2040c93e
Add GitHub automation for new issues (#13610)
Set up automation to move newly opened issues in GitHub to the issue triage board.
2022-08-24 12:10:32 +01:00
David Robertson ba8938b090
Reject non-strict types in Pydantic models (#13502) 2022-08-17 11:17:04 +01:00
reivilibre 05aeeb3a80
Enable Complement CI tests in the 'latest deps' test run. (#13213)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-08-01 10:55:31 +00:00
reivilibre b817574be7
Re-enable running Complement tests against Synapse with workers. (#13420) 2022-08-01 11:51:44 +01:00
David Robertson e9ce4d089b
Use and recommend poetry 1.1.14, up from 1.1.12 (#13285) 2022-07-15 16:18:47 +01:00
Patrick Cloke 90e9b4fa1e
Do not fail build if complement with workers fails. (#13266) 2022-07-13 08:30:42 -04:00
reivilibre 68695d8007
Factor out some common Complement CI setup commands to a script. (#13157) 2022-07-05 14:24:42 +01:00
reivilibre c04e25789e
Enable Complement testing in the 'Twisted Trunk' CI runs. (#13079) 2022-07-01 15:42:49 +00:00
reivilibre 9667bad55d
Improve startup times in Complement test runs against workers, particularly in CPU-constrained environments. (#13127)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-06-30 11:58:12 +00:00
reivilibre e12ff697a4
Sort failing jobs in Complement CI to the top of the logs to make them easier to read. (#13057) 2022-06-15 16:13:36 +00:00
Erik Johnston de334ac183
Add a CI job to check that schema deltas are in the correct folder. (#13063) 2022-06-15 16:27:18 +01:00
Erik Johnston 9ad2197fa7
Rename complement-developonly (#13046) 2022-06-15 15:11:42 +00:00
reivilibre 538044ac01
Collapse Docker build commands in Complement CI runs to make the logs easier to read. (#13058) 2022-06-15 14:42:27 +00:00
reivilibre 09a3c5ce0b
Fix Complement runs always being Postgres (#13034)
* Fix Complement runs always being Postgres

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-06-14 13:13:35 +01:00
reivilibre 3d1d510fa9
Enable testing against PostgreSQL databases in Complement CI. (#12965) 2022-06-09 14:16:34 +01:00
reivilibre 3c8f1290b8
Use a GitHub form for issues rather than a hard-to-read, easy-to-ignore template. (#12982)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-06-08 11:11:02 +00:00
Michael Telatynski 782cb7420a
Fix complement tests using the wrong path (#12933) 2022-06-01 15:57:09 +01:00
reivilibre bf01e51554
Test Synapse against Complement with workers. (#12810)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-05-31 13:02:00 +00:00
Henry b8fa24b022
Use `docker/metadata-action` to generate docker image tags (#12573)
Update the "Build docker images" GitHub Actions workflow to use
`docker/metadata-action` to generate docker image tags, instead of a
custom shell script.

Signed-off-by: Henry <97804910+henryclw@users.noreply.github.com>
2022-05-05 12:36:42 +00:00
David Robertson 57fac2a234
Allow unused ignores in "bleeding edge" CI (#12576)
* Allow unused ignores in "bleeding edge" CI

Where "bleeding edge" means the Twisted Trunk and Latest Deps jobs.

Follow up from #12531.
Resolves #12574.

* Use `--extras all` in latest deps mypy CI

Twisted trunk job already does this.

Missed in #12531.

* changelog
2022-04-29 17:57:23 +01:00
David Robertson 5d3509dfda
Revert accidental direct-to-develop commits.
This reverts commit 5a320baa45.
This reverts commit f282d5fc11.
This reverts commit ce6ecdd4b4.
2022-04-28 11:33:05 +01:00
David Robertson f282d5fc11
Use `--extras all` in latest deps mypy CI
Twisted trunk job already does this.

Missed in #12531.
2022-04-28 11:29:13 +01:00
David Robertson ce6ecdd4b4
Allow unused ignores in "bleeding edge" CI
Where "bleeding edge" means the Twisted Trunk and Latest Deps jobs.

Follow up from #12531.
Resolves #12574.
2022-04-28 11:28:22 +01:00
David Robertson 6463244375
Remove unused `# type: ignore`s (#12531)
Over time we've begun to use newer versions of mypy, typeshed, stub
packages---and of course we've improved our own annotations. This makes
some type ignore comments no longer necessary. I have removed them.

There was one exception: a module that imports `select.epoll`. The
ignore is redundant on Linux, but I've kept it ignored for those of us
who work on the source tree using not-Linux. (#11771)

I'm more interested in the config line which enforces this. I want
unused ignores to be reported, because I think it's useful feedback when
annotating to know when you've fixed a problem you had to previously
ignore.

* Installing extras before typechecking

Lacking an easy way to install all extras generically, let's bite the bullet and
make install the hand-maintained `all` extra before typechecking.

Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to
the release/v1 branch.
2022-04-27 14:03:44 +01:00
David Robertson e5a76ec00b
Dump setuptools; correct pyproject version number (#12478) 2022-04-20 17:33:20 +01:00
David Robertson eed38c5027
Add CI job to act as a canary for testing against latest dependencies (#12472)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-04-20 12:48:44 +01:00
David Robertson d24cd17820
Use poetry lockfile in twisted trunk CI job (#12425)
Fixes #12458

Co-authored-by: Sean Quah <seanq@element.io>
Co-authored-by: Dan Callahan <danc@element.io>
2022-04-13 11:26:53 +01:00
David Robertson 5a275a2377
Run "main" trial tests under poetry (#12438)
* Run "main" trial tests under poetry

Olddeps and twisted trunk tests are handled in separate PRs.

The PyPy config is a best-effort only; it's completely untested.

Pulled out from #12337.

* Changelog
2022-04-12 17:41:21 +01:00
David Robertson 58c657322a
Run lints under poetry in CI; remove lint tox jobs (#12434)
Co-authored-by: Dan Callahan <danc@element.io>
2022-04-12 17:35:48 +01:00
David Robertson 4e13743738
Poetry: select olddeps using `poetry` (#12407)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-04-12 10:50:11 +00:00
Richard van der Hoff 5f72ea1bde
Move complement setup stuff into the Synapse repo (#12404)
Fixes matrix-org/complement#330 (or it will, once we remove the old files).

It's not quite a lift-and-shift: I've also taken the opportunity to get rid of the custom CA that we used to use to sign the TLS certs, which has been superceded by the CA exposed by Complement.
2022-04-11 11:39:28 +01:00
David Robertson 98ec375b26
CI: Fix the export-data job to run under poetry (#12418)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-04-08 19:18:54 +00:00
David Robertson 7851a2c62f
CI: Get the portdb job to run under poetry (#12412) 2022-04-08 12:00:05 +01:00
Andrew Morgan 163fd686b5
Update mdbook to 0.4.17 (#12339)
Update mdbook (the tool used to render the documentation website) to version 0.4.17.
2022-04-05 17:21:58 +01:00
David Robertson 9a3f1f5383
Nuke the tox `packaging` job (#12334) 2022-03-31 11:01:26 +01:00
Andrew Morgan 19a1d6a42a
Remove references to "msc2403" (#12165) 2022-03-30 11:15:47 +00:00
kegsay 8b7b371ff6
Re-enable test concurrency on Complement (#12283) 2022-03-24 15:59:06 -04:00
reivilibre bfa7d6b035
Fix CI not attaching source distributions and wheels to the GitHub releases. (#12131) 2022-03-08 15:11:50 +00:00
Richard van der Hoff 0211f18d65
Switch the `tests-done` job to an Action (#12161)
I've factored it out for easier use in other workflows.
2022-03-07 12:24:06 +00:00
David Robertson 36071d39f7
Changelog (#12153) 2022-03-04 12:01:51 +00:00
Eric Eastwood a511a890d7
Enable MSC2716 Complement tests in Synapse (#12145)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-03-03 11:19:20 +00:00
Eric Eastwood 31b125ccec
Enable MSC3030 Complement tests in Synapse (#12144)
The Complement tests for MSC3030 are now merged, https://github.com/matrix-org/complement/pull/178

Synapse implmentation: https://github.com/matrix-org/synapse/pull/9445
2022-03-03 11:45:23 +01:00
David Robertson 1fbe0316a9
Add suffices to scripts in scripts-dev (#12137)
* Rename scripts-dev to have suffices

* Update references to `scripts-dev`

* Changelog

* These scripts don't pass mypy
2022-03-02 18:00:26 +00:00
David Robertson 5f62a094de
Detox, part 1 of N (#12119)
* Don't use `tox` for `check-sampleconfig`
* Don't use `tox` for check-newsfragment
2022-03-01 19:47:02 +00:00
David Robertson af2c1e3d2a
Tidy the building of sdists and wheels (#12051)
* Don't build distribution pkgs in tests.yml
* Run `release-artifacts` on release branches
* Use backend-meta workflow for packaging
2022-02-22 11:33:37 +00:00
Richard van der Hoff bb98c593a5
Prepare for rename of default complement branch (#11971)
use `HEAD` rather than hardcoding `master`
2022-02-11 14:43:34 +00:00
kegsay a3865ed525
Run Complement tests sequentially (#11910)
Since #11811 there has been general Complement flakiness around networking.
It seems like tests are hitting the wrong containers. In an effort to diagnose
the cause of this, as well as reduce its impact on this project, set the
parallelsim to 1 (no parallelism) when running tests.

If this fixes the flakiness then this indicates the cause and I can diagnose
this further. If this doesn't fix the flakiness then that implies some kind
of test pollution which also helps to diagnose this further.
2022-02-04 13:04:57 +00:00
kegsay 0938f32e93
CI: run Complement on the VM, not inside Docker (#11811)
* CI: run Complement on the VM, not inside Docker

This requires https://github.com/matrix-org/complement/pull/289

We now run Complement on the VM instead of inside a Docker container.
This is to allow Complement to bind to any high-numbered port when it
starts up its own federation servers. We want to do this to allow for
more concurrency when running complement tests. Previously, Complement
only ever bound to `:8448` when running its own federation server. This
prevented multiple federation tests running at the same time as they would
fight each other on the port. This did however allow Complement to run
in Docker, as the host could just port forward `:8448` to allow homeserver
containers to communicate to Complement. Now that we are using random
ports however, we cannot use Docker to run Complement. This ends up
being a good thing because:
 - Running Complement tests locally is closer to how they run in CI.
 - Allows the `CI` env var to be removed in Complement.
 - Slightly speeds up runs as we don't need to pull down the Complement
   image prior to running tests. This assumes GHA caches actions sensibly.

* Changelog

* Full stop

* Update .github/workflows/tests.yml

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

* Review comments

* Update .github/workflows/tests.yml

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-01-25 15:05:22 +00:00
Richard van der Hoff 2d327d25bf
Skip the initial amd64-only Docker build (#11810)
PyNaCl's recent 1.5.0 release on PyPi includes arm64 wheels, which means our
arm64 docker images now build in a sensible amount of time, so we can skip the
amd64-only build.
2022-01-24 18:31:23 +00:00
Shay 9006ee36d1
Drop support for and remove references to EOL Python 3.6 (#11683)
* remove reference in comments to python3.6

* upgrade tox python env in script

* bump python version in example for completeness

* upgrade python version requirement in setup doc

* upgrade necessary python version in __init__.py

* upgrade python version in setup.py

* newsfragment

* drops refs to bionic and replace with focal

* bump refs to postgres 9.6 to 10

* fix hanging ci

* try installing tzdata first

* revert change made in b979f336

* ignore new random mypy error while debugging other error

* fix lint error for temporary workaround

* revert change to install list

* try passing env var

* export debian frontend var?

* move line and add comment

* bump pillow dependency

* bump lxml depenency

* install libjpeg-dev for pillow

* bump automat version to one compatible with py3.8

* add libwebp for pillow

* bump twisted trunk python version

* change suffix of newsfragment

* remove redundant python 3.7 checks

* lint
2022-01-21 14:23:26 -08:00
David Robertson 4ca8fcdd5a
Invite PR submitters to credit themselves (#11744)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-01-13 18:12:59 +00:00
Richard van der Hoff d41c4654db
Use buildkit's cache feature to speed up docker builds (#11691)
Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
2022-01-12 10:37:57 +00:00
kegsay 3be63654e4
Prettier complement logs (#11707)
* Prettier complement logs

* Changelog
2022-01-10 11:46:40 +00:00
Shay 87da37374a
Drop EOL python 3.6 from CI (#11595)
* remove python 3.6 and postgres 9.6 from github workflow

* remove python 3.6 env from tox

* newsfragment

* correct postgres version

* add py310 to tox env list
2021-12-21 11:37:04 -08:00
Patrick Cloke a4521ce0a8
Support the stable /hierarchy endpoint from MSC2946 (#11329)
This also makes additional updates where the implementation
had drifted from the approved MSC.

Unstable endpoints will be removed at a later data.
2021-11-29 14:32:20 -05:00
Patrick Cloke 9d1971a5c4
Return the stable `event` field from `/send_join` per MSC3083. (#11413)
This does not remove the unstable field and still parses both.
Handling of the unstable field will need to be removed in the
future.
2021-11-29 15:43:20 +00:00
Richard van der Hoff e2e9bea1ce
Publish a `develop` docker image (#11380)
I'd find it helpful to have a docker image corresponding to current develop,
without having to build my own.
2021-11-19 10:56:59 +00:00
Dirk Klimpel 93aa670642
Update outdated links in `PULL_REQUEST_TEMPLATE.md` (#11225) 2021-11-01 17:29:51 +00:00
Hillery Shay 7d70582eb0
Fix broken export-data admin command and add a test for it to CI (#11078)
Fix broken export-data admin command and add a test for it to CI
2021-10-18 08:14:12 -07:00
Patrick Cloke 8711e15734
Always dump logs from trial during CI. (#11068)
Instead of only dumping them if trial passes.
2021-10-12 20:09:49 +00:00
Erik Johnston 0b4d5ce5e3
Fix CI to run the unit tests without optional deps (#11017)
This also turns off calculating code coverage, as we didn't use it and it was a lot of noise
2021-10-08 10:05:48 +01:00
Brendan Abolivier b0460936c8
Add the synapse-core team as code owners (#10994)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-10-06 16:03:17 +02:00
David Robertson 6f6e956338
Run CI with Python 3.10 and Postgres 14 (#10992) 2021-10-05 12:43:04 +01:00
Patrick Cloke ebd8baf61f
Clear our destination directories before copying files to GitHub pages. (#10869)
This should fix stale deleted files being still accessible.
2021-09-21 16:32:46 +00:00
David Robertson f455b0e420
GHA: reintroduce an env var for `$GITHUB_HEAD_REF` (#10659)
This should ensure GHA runs synapse against the same-named sytest branch
2021-09-20 17:35:16 +01:00
Dan Callahan 2efc838f05
Avoid duplicate issues from Twisted trunk failures (#10672)
Setting `update_existing: true` in the `create-an-issue` GitHub Action
will avoid opening duplicate issues if an open issue already exists with
an identical title.

If no open issues match the title, then a new issue will be created.

This helps avoid spamming our issue tracker should there be a failure
when testing against Twisted's trunk.

This PR also pins the SHA of the `create-an-issue` action to mitigate
the risk of a malicious actor gaining access to JasonEtco's account.

See GitHub's page on security hardening third party actions for more:
https://docs.github.com/en/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions

Signed-off-by: Dan Callahan <danc@element.io>
2021-08-23 13:06:49 +00:00
Richard van der Hoff 4db65f911a
Run a nightly CI build against Twisted trunk. (#10651)
This creates a GHA workflow which runs at 8am every day, and runs mypy, trial and sytest against Twisted's current trunk. If any of the jobs fail, it opens an issue.
2021-08-23 11:12:45 +01:00
David Robertson 878528913d Remove buildkite-era comment 2021-08-12 11:48:36 +01:00
David Robertson 74fcd5aab9 portdb also uses coverage, so provide $TOP there 2021-08-12 10:41:01 +01:00
David Robertson cb5976ebd7 set TOP in sytest containers 2021-08-11 20:08:48 +01:00
David Robertson 33ef86aa25 Rename ci to .ci 2021-08-11 19:59:57 +01:00
David Robertson 58e5da5aa0 Remove buildkite from portdb CI tests 2021-08-10 16:38:13 +01:00
David Robertson c5988a8eb7 Remove unused BUILDKITE_BRANCH env var 2021-08-10 16:38:13 +01:00
David Robertson 3d67b8c82b Move sytest worker-blacklist to ci directory 2021-08-10 16:38:13 +01:00
David Robertson 03fb99a5c8 check-newsfragment: pass pr number explicitly
use PULL_REQUEST_NUMBER instead of BUILDKITE_PULL_REQUEST
remove the other user of BUILDKITE_PULL_REQUEST, namely merge_base_branch.sh
2021-08-10 16:38:13 +01:00
David Robertson 8da9e3cb69 Move test_old_deps.sh to new ci dir 2021-08-10 16:38:13 +01:00
Erik Johnston e33f14e8d5
Don't fail CI when lint-newfile job was skipped (#10529) 2021-08-05 11:22:27 +01:00
Richard van der Hoff a6ea32a798
Fix the `tests-done` github actions step, again (#10512) 2021-08-02 21:06:34 +01:00
Patrick Cloke 13944678c3
Use new go test running syntax for complement. (#10488)
Updates CI and the helper script t ensures all tests are run (in parallel).
2021-07-27 12:08:51 +00:00
Richard van der Hoff f22252d4f9
Enable docker image caching for the deb build (#10431) 2021-07-26 11:36:01 +01:00
Erik Johnston ab82fd6ed1 Merge branch 'release-v1.39' into develop 2021-07-23 09:19:24 +01:00
Richard van der Hoff 4c3fdfc808
Fix an error in the docker workflow (#10461) 2021-07-22 21:50:30 +01:00
Richard van der Hoff 5e2df47f72
Cancel redundant GHA workflows (#10451) 2021-07-22 11:35:06 +01:00
Richard van der Hoff f1347bcfdc
Fix the tests-done Github Actions job (#10444) 2021-07-22 11:10:30 +01:00
Richard van der Hoff c6509991f3
Move the docker image build to Github Actions (#10416)
it's flaky on circleCI, and having to manage multiple CI providers is painful.
2021-07-21 12:33:35 +01:00
Richard van der Hoff 97c8ae90f7
Add a github actions job recording success of other jobs. (#10430) 2021-07-20 11:41:19 +01:00
Richard van der Hoff 07e0992a76
Make GHA config more efficient (#10383)
A few things here:

* Build the debs for single distro for each PR, so that we can see if it breaks. Do the same for develop. Building all the debs ties up the GHA workers for ages.
* Stop building the debs for release branches. Again, it takes ages, and I don't think anyone is actually going to stop and look at them. We'll know they are working when we make an RC.
* Change the configs so that if we manually cancel a workflow, it actually does something.
2021-07-14 14:41:23 +01:00