Commit Graph

143 Commits

Author SHA1 Message Date
Slavi Pantaleev
63301b0ef1 Improvements around Synapse worker/metrics ports exposure
There was a `matrix_nginx_proxy_enabled|default(False)` check, but:
- it didn't seem to work reliably for some reason (hmm)
- referring to a `matrix_nginx_proxy_*` variable from within the
  `matrix-synapse` role is not ideal
- exposing always happened on `127.0.0.1`, which may not be good enough
  for some rarer setups (where the own webserver is external to the host)
2021-01-25 08:25:43 +02:00
Marcel Partap
edc21f15e5 Restrict publishing worker (metrics) ports to localhost 2021-01-24 08:53:09 +01:00
Marcel Partap
183adec3d8 Merge remote-tracking branch 'origin/master' into synapse-workers 2021-01-23 15:04:11 +01:00
Marcel Partap
f2c7d79238 Drop probably incorrect comment from synapse homeserver.yaml.j2 2021-01-23 14:44:36 +01:00
Slavi Pantaleev
1692a28fe4 Work around annoying Docker warning about undefined $HOME
> WARNING: Error loading config file: .dockercfg: $HOME is not defined

.. which appeared in Docker 20.10.
2021-01-15 00:23:01 +02:00
Slavi Pantaleev
d5945c6e78 Upgrade Synapse (v1.24.0 -> v1.25.0) for amd64 2021-01-13 13:02:49 +02:00
Marcel Partap
cd8100544b Merge remote-tracking branch 'origin/master' into synapse-workers
Sync with upstream
2021-01-08 20:58:50 +01:00
Slavi Pantaleev
d08b27784f Fix systemd services autostart problem with Docker 20.10
The Docker 19.04 -> 20.10 upgrade contains the following change
in `/usr/lib/systemd/system/docker.service`:

```
-BindsTo=containerd.service
-After=network-online.target firewalld.service containerd.service
+After=network-online.target firewalld.service containerd.service multi-user.target
-Requires=docker.socket
+Requires=docker.socket containerd.service
Wants=network-online.target
```

The `multi-user.target` requirement in `After` seems to be in conflict
with our `WantedBy=multi-user.target` and `After=docker.service` /
`Requires=docker.service` definitions, causing the following error on
startup for all of our systemd services:

> Job matrix-synapse.service/start deleted to break ordering cycle starting with multi-user.target/start

A workaround which appears to work is to add `DefaultDependencies=no`
to all of our services.
2020-12-10 11:43:20 +02:00
Slavi Pantaleev
aa86e0dac6 Upgrade Synapse (v1.23.0 -> v1.24.0)
Because the ARM images are not pushed yet, we hold back to v1.23.0
for now.
2020-12-09 13:31:10 +02:00
Slavi Pantaleev
c07c927d9f Automatically enable openid listeners when ma1sd enabled
ma1sd requires the openid endpoints for certain functionality.
Example: 90b2b5301c/src/main/java/io/kamax/mxisd/auth/AccountManager.java (L67-L99)

If federation is disabled, we still need to expose these openid APIs on the
federation port.

Previously, we were doing similar magic for Dimension.
As per its documentation, when running unfederated, one is to enable
the openid listener as well. As per their recommendation, people
are advised to do enable it on the Client-Server API port
and use the `federationUrl` variable to override where the federation
port is (making federation requests go to the Client-Server API).

Because ma1sd always uses the federation port (unless you do some
DNS overwriting magic using its configuration -- which we'd rather not
do), it's better if we just default to putting the `openid` listener
where it belongs - on the federation port.

With this commit, we retain the "automatically enable openid APIs" thing
we've been doing for Dimension, but move it to the federation port instead.
We also now do the same thing when ma1sd is enabled.
2020-12-08 16:59:20 +02:00
Marcel Partap
e892ac464f synapse workers: untangle config template and specify bind address
.. to mitigate log noise - WARNING:
Failed to listen on 0.0.0.0, continuing because listening on [::]
2020-12-01 23:49:23 +01:00
Marcel Partap
f201bca519 synapse workers: define and expose METRICS port for each worker
As seen on TV:
https://github.com/matrix-org/synapse/blob/master/docs/metrics-howto.md#monitoring-workers
2020-12-01 22:49:15 +01:00
Marcel Partap
b73ac965ac Merge remote-tracking branch 'origin/master' into synapse-workers 2020-12-01 21:24:26 +01:00
Slavi Pantaleev
1fca917ad1 Replace some -v instances with --mount
`-v` magically creates the source destination as a directory,
if it doesn't exist already. We'd like to avoid this magic
and the potential breakage that it might cause.

We'd rather fail while Docker tries to find things to `--mount`
than have it automatically create directories and fail anyway,
while having contaminated the filesystem.

There's a lot more `-v` instances remaining to be fixed later on.
This is just some start.

Things like `matrix_synapse_container_additional_volumes` and
`matrix_nginx_proxy_container_additional_volumes` were not changed to
use `--mount`, as options for each one are passed differently
(`ro` is `ro`, but `rw` doesn't exist and `slave` is `bind-propagation=slave`).
To avoid breaking people's custom volume mounts, we keep it as it is for now.

A deficiency with `--mount` is that it lacks the `z` option (SELinux
ownership changes), and some of our `-v` instances use that. I'm not
sure how supported SELinux is for us right now, but it might be,
and breaking that would not be a good idea.
2020-11-24 10:26:05 +02:00
Slavi Pantaleev
b627d93cdc Update homeserver.yaml to keep up with Synapse v1.23.0
Related to #724 (Github Pull Request)
2020-11-18 16:57:50 +02:00
Marcel Partap
2d1b9f2dbf synapse workers: reworkings + get endpoints from upstream docs via awk
(yes, a bit awkward and brittle… xD)
2020-10-28 07:13:19 +01:00
Marcel Partap
87bd64ce9e Merge remote-tracking branch 'origin/master' into synapse-workers 2020-10-23 23:45:07 +02:00
Marcel Partap
a4125d5446 synapse workers: polishing, cleansing and installation of jq dependency 2020-10-23 20:49:53 +02:00
Marcel Partap
501efee07e synapse workers: supply systemd with actual worker PIDs (requires jq)
also, worker.yaml.j2:
  - hone worker_name
  - remove worker_pid_file entry (would only be used if worker_daemonize
    set to true; also, synapse only knows about the container namespace
    and thus can not provide the required host-view PID)
2020-10-22 20:53:41 +02:00
Aaron Raimist
78529cbd47
Upgrade Synapse (v1.20.1 -> v1.21.0) 2020-10-12 23:59:34 -05:00
Marcel Partap
d2e61af224 Add worker_name to synapse worker config template
& restrict federation listener; frontend_proxy / user_dir don't need it
2020-10-11 21:52:08 +02:00
Marcel Partap
e9241f5fb9 Improve synapse-workers systemd service template
Is the PID magic gonna work? or will it need an ExecStartPost hack..
2020-10-11 21:09:19 +02:00
Marcel Partap
40024e9b81 Prevent workers failing if their config doesn't exist
- cherry-pick "Ensure worker config exists in systemd service (#7528)"
  from synapse d74cdc1a42e8b487d74c214b1d0ca575429d546a:
  "check that the worker config file exists instead of silently failing."
2020-10-11 21:09:19 +02:00
Marcel Partap
93a8ea7e4a Merge remote-tracking branch 'master' into feature/add-worker-support 2020-10-11 20:59:05 +02:00
Slavi Pantaleev
dd217137b6 Upgrade Synapse (v1.19.3 -> v1.20.0) 2020-09-22 19:28:07 +03:00
Max Klenk
4fdfc0a34f
add missing ratelimiting options required for load testing 2020-09-11 09:46:20 +02:00
Max Klenk
a25a429a52
add redis support 2020-09-10 13:39:00 +02:00
Max Klenk
06bc430c7c
refactor to use new workers and routes they serve 2020-08-28 13:53:39 +02:00
Max Klenk
567d0318b0
Merge branch 'synapse-workers' into feature/add-worker-support 2020-08-27 15:22:12 +02:00
Slavi Pantaleev
9952ec6c16 Upgrade Synapse (v1.18.0 -> v1.19.0) 2020-08-17 17:02:40 +03:00
Slavi Pantaleev
f78a5d4ee8 Upgrade Synapse (v1.17.0 -> v1.18.0) 2020-07-30 14:21:44 +03:00
Slavi Pantaleev
200f912c04 Upgrade Synapse (v1.16.1 -> v1.17.0)
Fixes #579 (Github Issue).
2020-07-13 14:08:50 +03:00
Slavi Pantaleev
928982cffe Upgrade Synapse (v1.15.2 -> v1.16.0) 2020-07-08 14:08:46 +03:00
Slavi Pantaleev
18ab677a96 Remove useless file 2020-07-08 00:22:47 +03:00
Slavi Pantaleev
f758ee90cb
Add |to_json to some values 2020-07-04 09:31:52 +03:00
Justin Croonenberghs
35c2655fa4 Removed troublesome #s 2020-07-03 19:01:03 -05:00
Justin Croonenberghs
1f21f0c09a Add variables for reCAPTCHA validation 2020-07-03 18:33:25 -05:00
Slavi Pantaleev
6538ae34f5 Upgrade Synapse (v1.14 -> v1.15)
Fixes #539 (Github Issue).
2020-06-11 16:02:01 +03:00
Slavi Pantaleev
f56a9a0f5f
Merge pull request #524 from cnvandijk/fix-executable-path
Remove hardcoded paths to commands on the host machine
2020-05-28 15:39:25 +03:00
Slavi Pantaleev
8bae39050e Update settings for Synapse v1.14.0 2020-05-28 15:23:05 +03:00
Chris van Dijk
6334f6c1ea Remove hardcoded command paths in systemd unit files
Depending on the distro, common commands like sleep and chown may either
be located in /bin or /usr/bin.

Systemd added path lookup to ExecStart in v239, allowing only the
command name to be put in unit files and not the full path as
historically required. At least Ubuntu 18.04 LTS is however still on
v237 so we should maintain portability for a while longer.
2020-05-27 23:14:54 +02:00
Slavi Pantaleev
8fb3ce6f6d Upgrade Synapse (v1.12.4 -> v1.13.0) 2020-05-19 21:35:32 +03:00
Chris van Dijk
3f4bc9b881 Move config supprt for unfederated dimension into group_vars 2020-04-22 19:23:56 +02:00
Marcel Partap
66a4073512 Publish synapse worker ports, need to be accessible to nginx 2020-04-19 19:05:03 +02:00
Marcel Partap
353bc7c362 Add initial support for synapse workers
· needs documentation; no checks yet for port clashes or typos in worker name
· according to https://github.com/matrix-org/synapse/wiki/Workers-setup-with-nginx#results
  about 90% of requests go to the synchrotron endpoint
· thus, the synchrotron worker is especially suited to be load-balanced
· most of the other workers are documented to support only a single instance
· https://github.com/matrix-org/synapse/blob/master/docs/workers.md
2020-04-19 19:05:03 +02:00
Chris van Dijk
3ddb8cd148 Add support for running dimension in an unfederated environment
This config change follows:

  https://github.com/turt2live/matrix-dimension/blob/master/docs/unfederated.md
2020-04-18 19:00:20 +02:00
Slavi Pantaleev
c13c29f48e Fix |to_nice_yaml breaking formatting
Well, actually 8cd9cde won't work, unless we put the
`|to_nice_yaml` thing on a new line.

We can, but that takes more lines and makes things look uglier.
Using `|to_json` seems good enough.

The whole file is parsed as YAML later on and merged with the
`_extension` variable before being dumped as YAML again in the end.
2020-04-18 11:06:53 +03:00
Slavi Pantaleev
8cd9cdead0 Improve consistency 2020-04-18 11:05:47 +03:00
Tom
e54428b160 Expose allowed local 3pids as a configurable option 2020-04-18 00:11:30 +01:00
Slavi Pantaleev
845f5f007b Make Synapse use ma1sd (if enabled) for threepid registration 2020-04-03 10:08:37 +03:00