diff --git a/CHANGELOG.md b/CHANGELOG.md index df4957eb5..2b9b149e1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,42 @@ +# 2022-09-15 + +## (Potential Backward Compatibility Break) Major changes to Synapse workers + +People who are interested in running a Synapse worker setup should know that **our Synapse worker implementation is much more powerful now**: + +- we've added [Stream writers support](#stream-writers-support) +- see the [Potential Backward Incompatibilities after these Synapse worker changes](#potential-backward-incompatibilities-after-these-synapse-worker-changes) + +### Stream writers support + +From now on, the playbook lets you easily set up various [stream writer workers](https://matrix-org.github.io/synapse/latest/workers.html#stream-writers) which can handle different streams (`events` stream; `typing` URL endpoints, `to_device` URL endpoints, `account_data` URL endpoints, `receipts` URL endpoints, `presence` URL endpoints). All of this work was previously handled by the main Synapse process, but can now be offloaded to stream writer worker processes. + +If you're using `matrix_synapse_workers_preset: one-of-each`, you'll automatically get 6 additional workers (one for each of the above stream types). Our `little-federation-helper` preset (meant to be quite minimal and focusing in improved federation performance) does not include stream writer workers. + +If you'd like to customize the number of workers we also make that possible using these variables: + +```yaml +# Synapse only supports more than 1 worker for the `events` stream. +# All other streams can utilize either 0 or 1 workers, not more than that. +matrix_synapse_workers_stream_writer_events_stream_workers_count: 5 +matrix_synapse_workers_stream_writer_typing_stream_workers_count: 1 +matrix_synapse_workers_stream_writer_to_device_stream_workers_count: 1 +matrix_synapse_workers_stream_writer_account_data_stream_workers_count: 1 +matrix_synapse_workers_stream_writer_receipts_stream_workers_count: 1 +matrix_synapse_workers_stream_writer_presence_stream_workers_count: 1 +``` + +### Potential Backward Incompatibilities after these Synapse worker changes + +Below we'll discuss **potential backward incompatibilities**. + +- **Worker names** (container names, systemd services, worker configuration files) **have changed**. Workers are now labeled sequentially (e.g. `matrix-synapse-worker_generic_worker-18111` -> `matrix-synapse-worker-generic-0`). The playbook will handle these changes automatically. + +- **Metric endpoints have also changed** (`/metrics/synapse/worker/generic_worker-18111` -> `/metrics/synapse/worker/generic-worker-0`). If you're [collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server), consider revisiting our [Collecting Synapse worker metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-synapse-worker-metrics-to-an-external-prometheus-server) docs and updating your Prometheus configuration. **If you're collecting metrics to the integrated Prometheus server** (not enabled by default), **your Prometheus configuration will be updated automatically**. Old data (from before this change) may stick around though. + +- **the format of `matrix_synapse_workers_enabled_list` has changed**. You were never advised to use this variable for directly creating workers (we advise people to control workers using `matrix_synapse_workers_preset` or by tweaking `matrix_synapse_workers_*_workers_count` variables only), but some people may have started using the `matrix_synapse_workers_enabled_list` variable to gain more control over workers. If you're one of them, you'll need to adjust its value. See `roles/matrix-synapse/defaults/main.yml` for more information on the new format. The playbook will also do basic validation and complain if you got something wrong. + + # 2022-09-09 ## Cactus Comments support diff --git a/docs/configuring-playbook-prometheus-grafana.md b/docs/configuring-playbook-prometheus-grafana.md index b2878c12b..f86566a94 100644 --- a/docs/configuring-playbook-prometheus-grafana.md +++ b/docs/configuring-playbook-prometheus-grafana.md @@ -90,11 +90,11 @@ matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_b Using `matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks` only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true` (see above). -Note : The playbook will hash the basic_auth password for you on setup. Thus, you need to give the plain-text version of the password as a variable. +Note : The playbook will hash the basic_auth password for you on setup. Thus, you need to give the plain-text version of the password as a variable. ### Collecting Synapse worker metrics to an external Prometheus server -If you are using workers (`matrix_synapse_workers_enabled: true`) and have enabled `matrix_synapse_metrics_proxying_enabled` as described above, the playbook will also automatically expose all Synapse worker threads' metrics to `https://matrix.DOMAIN/metrics/synapse/worker/TYPE-ID`, where `TYPE` corresponds to the type and `ID` to the instanceId of a worker as exemplified in `matrix_synapse_workers_enabled_list`. +If you are using workers (`matrix_synapse_workers_enabled: true`) and have enabled `matrix_synapse_metrics_proxying_enabled` as described above, the playbook will also automatically expose all Synapse worker threads' metrics to `https://matrix.DOMAIN/metrics/synapse/worker/ID`, where `ID` corresponds to the worker `id` as exemplified in `matrix_synapse_workers_enabled_list`. The playbook also generates an exemplary config file (`/matrix/synapse/external_prometheus.yml.template`) with all the correct paths which you can copy to your Prometheus server and adapt to your needs. Make sure to edit the specified `password_file` path and contents and path to your `synapse-v2.rules`. It will look a bit like this: @@ -111,8 +111,8 @@ scrape_configs: labels: job: "master" index: 1 - - job_name: 'synapse-generic_worker-1' - metrics_path: /metrics/synapse/worker/generic_worker-18111 + - job_name: 'matrix-synapse-synapse-worker-generic-worker-0' + metrics_path: /metrics/synapse/worker/generic-worker-0 scheme: https basic_auth: username: prometheus diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index c0d831e1e..e891a3d9a 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -1760,6 +1760,11 @@ matrix_nginx_proxy_synapse_workers_enabled: "{{ matrix_synapse_workers_enabled } matrix_nginx_proxy_synapse_workers_list: "{{ matrix_synapse_workers_enabled_list }}" matrix_nginx_proxy_synapse_generic_worker_client_server_locations: "{{ matrix_synapse_workers_generic_worker_client_server_endpoints }}" matrix_nginx_proxy_synapse_generic_worker_federation_locations: "{{ matrix_synapse_workers_generic_worker_federation_endpoints }}" +matrix_nginx_proxy_synapse_stream_writer_typing_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_typing_stream_worker_client_server_endpoints }}" +matrix_nginx_proxy_synapse_stream_writer_to_device_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_to_device_stream_worker_client_server_endpoints }}" +matrix_nginx_proxy_synapse_stream_writer_account_data_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_account_data_stream_worker_client_server_endpoints }}" +matrix_nginx_proxy_synapse_stream_writer_receipts_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_receipts_stream_worker_client_server_endpoints }}" +matrix_nginx_proxy_synapse_stream_writer_presence_stream_worker_client_server_locations: "{{ matrix_synapse_workers_stream_writer_presence_stream_worker_client_server_endpoints }}" matrix_nginx_proxy_synapse_media_repository_locations: "{{matrix_synapse_workers_media_repository_endpoints|default([]) }}" matrix_nginx_proxy_synapse_user_dir_locations: "{{ matrix_synapse_workers_user_dir_endpoints|default([]) }}" @@ -1767,6 +1772,8 @@ matrix_nginx_proxy_systemd_wanted_services_list: | {{ ['matrix-' + matrix_homeserver_implementation + '.service'] + + (matrix_synapse_webserving_workers_systemd_services_list if matrix_homeserver_implementation == 'synapse' and matrix_synapse_workers_enabled) + + (['matrix-corporal.service'] if matrix_corporal_enabled else []) + (['matrix-ma1sd.service'] if matrix_ma1sd_enabled else []) diff --git a/roles/matrix-nginx-proxy/defaults/main.yml b/roles/matrix-nginx-proxy/defaults/main.yml index a74842158..88c99ea9b 100644 --- a/roles/matrix-nginx-proxy/defaults/main.yml +++ b/roles/matrix-nginx-proxy/defaults/main.yml @@ -623,6 +623,11 @@ matrix_nginx_proxy_synapse_workers_enabled: false matrix_nginx_proxy_synapse_workers_list: [] matrix_nginx_proxy_synapse_generic_worker_client_server_locations: [] matrix_nginx_proxy_synapse_generic_worker_federation_locations: [] +matrix_nginx_proxy_synapse_stream_writer_typing_stream_worker_client_server_locations: [] +matrix_nginx_proxy_synapse_stream_writer_to_device_stream_worker_client_server_locations: [] +matrix_nginx_proxy_synapse_stream_writer_account_data_stream_worker_client_server_locations: [] +matrix_nginx_proxy_synapse_stream_writer_receipts_stream_worker_client_server_locations: [] +matrix_nginx_proxy_synapse_stream_writer_presence_stream_worker_client_server_locations: [] matrix_nginx_proxy_synapse_media_repository_locations: [] matrix_nginx_proxy_synapse_user_dir_locations: [] diff --git a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 index 81e31a7c0..bbce7462b 100644 --- a/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 +++ b/roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2 @@ -1,23 +1,52 @@ #jinja2: lstrip_blocks: "True" {% set generic_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'generic_worker') | list %} +{% set stream_writer_typing_stream_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'stream_writer') | selectattr('stream_writer_stream', 'equalto', 'typing') | list %} +{% set stream_writer_to_device_stream_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'stream_writer') | selectattr('stream_writer_stream', 'equalto', 'to_device') | list %} +{% set stream_writer_account_data_stream_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'stream_writer') | selectattr('stream_writer_stream', 'equalto', 'account_data') | list %} +{% set stream_writer_receipts_stream_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'stream_writer') | selectattr('stream_writer_stream', 'equalto', 'receipts') | list %} +{% set stream_writer_presence_stream_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'stream_writer') | selectattr('stream_writer_stream', 'equalto', 'presence') | list %} {% set media_repository_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'media_repository') | list %} {% set user_dir_workers = matrix_nginx_proxy_synapse_workers_list | selectattr('type', 'equalto', 'user_dir') | list %} + +{% macro render_worker_upstream(name, workers, matrix_nginx_proxy_enabled) %} +{% if workers | length > 0 %} + upstream {{ name }} { + {% for worker in workers %} + {% if matrix_nginx_proxy_enabled %} + server "{{ worker.name }}:{{ worker.port }}"; + {% else %} + server "127.0.0.1:{{ worker.port }}"; + {% endif %} + {% endfor %} + } +{% endif %} +{% endmacro %} + +{% macro render_locations_to_upstream(locations, upstream_name) %} + {% for location in locations %} + location ~ {{ location }} { + proxy_pass http://{{ upstream_name }}$request_uri; + proxy_set_header Host $host; + } + {% endfor %} +{% endmacro %} + {% if matrix_nginx_proxy_synapse_workers_enabled %} {% if matrix_nginx_proxy_synapse_cache_enabled %} proxy_cache_path {{ matrix_nginx_proxy_synapse_cache_path }} levels=1:2 keys_zone={{ matrix_nginx_proxy_synapse_cache_keys_zone_name }}:{{ matrix_nginx_proxy_synapse_cache_keys_zone_size }} inactive={{ matrix_nginx_proxy_synapse_cache_inactive_time }} max_size={{ matrix_nginx_proxy_synapse_cache_max_size_mb }}m; {% endif %} # Round Robin "upstream" pools for workers - {% if generic_workers %} - upstream generic_worker_upstream { + {% if generic_workers |length > 0 %} + upstream generic_workers_upstream { # ensures that requests from the same client will always be passed # to the same server (except when this server is unavailable) hash $http_x_forwarded_for; {% for worker in generic_workers %} {% if matrix_nginx_proxy_enabled %} - server "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.port }}"; + server "{{ worker.name }}:{{ worker.port }}"; {% else %} server "127.0.0.1:{{ worker.port }}"; {% endif %} @@ -25,29 +54,15 @@ } {% endif %} - {% if media_repository_workers %} - upstream media_repository_upstream { - {% for worker in media_repository_workers %} - {% if matrix_nginx_proxy_enabled %} - server "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.port }}"; - {% else %} - server "127.0.0.1:{{ worker.port }}"; - {% endif %} - {% endfor %} - } - {% endif %} + {{ render_worker_upstream('stream_writer_typing_stream_workers_upstream', stream_writer_typing_stream_workers, matrix_nginx_proxy_enabled) }} + {{ render_worker_upstream('stream_writer_to_device_stream_workers_upstream', stream_writer_to_device_stream_workers, matrix_nginx_proxy_enabled) }} + {{ render_worker_upstream('stream_writer_account_data_stream_workers_upstream', stream_writer_account_data_stream_workers, matrix_nginx_proxy_enabled) }} + {{ render_worker_upstream('stream_writer_receipts_stream_workers_upstream', stream_writer_receipts_stream_workers, matrix_nginx_proxy_enabled) }} + {{ render_worker_upstream('stream_writer_presence_stream_workers_upstream', stream_writer_presence_stream_workers, matrix_nginx_proxy_enabled) }} - {% if user_dir_workers %} - upstream user_dir_upstream { - {% for worker in user_dir_workers %} - {% if matrix_nginx_proxy_enabled %} - server "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.port }}"; - {% else %} - server "127.0.0.1:{{ worker.port }}"; - {% endif %} - {% endfor %} - } - {% endif %} + {{ render_worker_upstream('media_repository_workers_upstream', media_repository_workers, matrix_nginx_proxy_enabled) }} + + {{ render_worker_upstream('user_dir_workers_upstream', user_dir_workers, matrix_nginx_proxy_enabled) }} {% endif %} server { @@ -65,21 +80,41 @@ server { {% if matrix_nginx_proxy_synapse_workers_enabled %} {# Workers redirects BEGIN #} - {% if generic_workers %} + {% if generic_workers | length > 0 %} # https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappgeneric_worker - {% for location in matrix_nginx_proxy_synapse_generic_worker_client_server_locations %} - location ~ {{ location }} { - proxy_pass http://generic_worker_upstream$request_uri; - proxy_set_header Host $host; - } - {% endfor %} + {{ render_locations_to_upstream(matrix_nginx_proxy_synapse_generic_worker_client_server_locations, 'generic_workers_upstream') }} {% endif %} - {% if media_repository_workers %} + {% if stream_writer_typing_stream_workers | length > 0 %} + # https://matrix-org.github.io/synapse/latest/workers.html#the-typing-stream + {{ render_locations_to_upstream(matrix_nginx_proxy_synapse_stream_writer_typing_stream_worker_client_server_locations, 'stream_writer_typing_stream_workers_upstream') }} + {% endif %} + + {% if stream_writer_to_device_stream_workers | length > 0 %} + # https://matrix-org.github.io/synapse/latest/workers.html#the-to_device-stream + {{ render_locations_to_upstream(matrix_nginx_proxy_synapse_stream_writer_to_device_stream_worker_client_server_locations, 'stream_writer_to_device_stream_workers_upstream') }} + {% endif %} + + {% if stream_writer_account_data_stream_workers | length > 0 %} + # https://matrix-org.github.io/synapse/latest/workers.html#the-account_data-stream + {{ render_locations_to_upstream(matrix_nginx_proxy_synapse_stream_writer_account_data_stream_worker_client_server_locations, 'stream_writer_account_data_stream_workers_upstream') }} + {% endif %} + + {% if stream_writer_receipts_stream_workers | length > 0 %} + # https://matrix-org.github.io/synapse/latest/workers.html#the-receipts-stream + {{ render_locations_to_upstream(matrix_nginx_proxy_synapse_stream_writer_receipts_stream_worker_client_server_locations, 'stream_writer_receipts_stream_workers_upstream') }} + {% endif %} + + {% if stream_writer_presence_stream_workers | length > 0 %} + # https://matrix-org.github.io/synapse/latest/workers.html#the-presence-stream + {{ render_locations_to_upstream(matrix_nginx_proxy_synapse_stream_writer_presence_stream_worker_client_server_locations, 'stream_writer_presence_stream_workers_upstream') }} + {% endif %} + + {% if media_repository_workers | length > 0 %} # https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappmedia_repository {% for location in matrix_nginx_proxy_synapse_media_repository_locations %} location ~ {{ location }} { - proxy_pass http://media_repository_upstream$request_uri; + proxy_pass http://media_repository_workers_upstream$request_uri; proxy_set_header Host $host; client_body_buffer_size 25M; @@ -97,15 +132,10 @@ server { {% endfor %} {% endif %} - {% if user_dir_workers %} + {% if user_dir_workers | length > 0 %} # FIXME: obsolete if matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled is set # https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappuser_dir - {% for location in matrix_nginx_proxy_synapse_user_dir_locations %} - location ~ {{ location }} { - proxy_pass http://user_dir_upstream$request_uri; - proxy_set_header Host $host; - } - {% endfor %} + {{ render_locations_to_upstream(matrix_nginx_proxy_synapse_user_dir_locations, 'user_dir_workers_upstream') }} {% endif %} {# Workers redirects END #} {% endif %} @@ -150,20 +180,15 @@ server { gzip_types text/plain application/json; {% if matrix_nginx_proxy_synapse_workers_enabled %} - {% if generic_workers %} + {% if generic_workers | length > 0 %} # https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappgeneric_worker - {% for location in matrix_nginx_proxy_synapse_generic_worker_federation_locations %} - location ~ {{ location }} { - proxy_pass http://generic_worker_upstream$request_uri; - proxy_set_header Host $host; - } - {% endfor %} + {{ render_locations_to_upstream(matrix_nginx_proxy_synapse_generic_worker_federation_locations, 'generic_workers_upstream') }} {% endif %} - {% if media_repository_workers %} + {% if media_repository_workers | length > 0 %} # https://github.com/matrix-org/synapse/blob/master/docs/workers.md#synapseappmedia_repository {% for location in matrix_nginx_proxy_synapse_media_repository_locations %} location ~ {{ location }} { - proxy_pass http://media_repository_upstream$request_uri; + proxy_pass http://media_repository_workers_upstream$request_uri; proxy_set_header Host $host; client_body_buffer_size 25M; diff --git a/roles/matrix-prometheus/templates/prometheus.yml.j2 b/roles/matrix-prometheus/templates/prometheus.yml.j2 index f3262f485..628f4f4f9 100644 --- a/roles/matrix-prometheus/templates/prometheus.yml.j2 +++ b/roles/matrix-prometheus/templates/prometheus.yml.j2 @@ -37,11 +37,12 @@ scrape_configs: index: 0 {% for worker in matrix_prometheus_scraper_synapse_workers_enabled_list %} {% if worker.metrics_port != 0 %} - - targets: ['matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.metrics_port }}'] + - targets: ['{{ worker.name }}:{{ worker.metrics_port }}'] labels: instance: {{ matrix_domain }} + worker_id: {{ worker.id }} job: {{ worker.type }} - index: {{ worker.instanceId }} + app: {{ worker.app }} {% endif %} {% endfor %} {% endif %} diff --git a/roles/matrix-synapse/defaults/main.yml b/roles/matrix-synapse/defaults/main.yml index a7dab5b67..4213c4f18 100644 --- a/roles/matrix-synapse/defaults/main.yml +++ b/roles/matrix-synapse/defaults/main.yml @@ -398,6 +398,12 @@ matrix_synapse_workers_presets: federation_sender_workers_count: 1 media_repository_workers_count: 0 user_dir_workers_count: 0 + stream_writer_events_stream_workers_count: 0 + stream_writer_typing_stream_workers_count: 0 + stream_writer_to_device_stream_workers_count: 0 + stream_writer_account_data_stream_workers_count: 0 + stream_writer_receipts_stream_workers_count: 0 + stream_writer_presence_stream_workers_count: 0 one-of-each: generic_workers_count: 1 pusher_workers_count: 1 @@ -409,6 +415,12 @@ matrix_synapse_workers_presets: # user_dir workers are deprecated since Synapse v1.59. This will be removed. # See: https://github.com/matrix-org/synapse/blob/v1.59.0/docs/upgrade.md#deprecation-of-the-synapseappappservice-and-synapseappuser_dir-worker-application-types user_dir_workers_count: 0 + stream_writer_events_stream_workers_count: 1 + stream_writer_typing_stream_workers_count: 1 + stream_writer_to_device_stream_workers_count: 1 + stream_writer_account_data_stream_workers_count: 1 + stream_writer_receipts_stream_workers_count: 1 + stream_writer_presence_stream_workers_count: 1 # Controls whether the matrix-synapse container exposes the various worker ports # (see `port` and `metrics_port` in `matrix_synapse_workers_enabled_list`) outside of the container. @@ -421,6 +433,71 @@ matrix_synapse_workers_generic_workers_count: "{{ matrix_synapse_workers_presets matrix_synapse_workers_generic_workers_port_range_start: 18111 matrix_synapse_workers_generic_workers_metrics_range_start: 19111 +# matrix_synapse_workers_stream_writer_events_stream_workers_count controls how many stream writers that handle the `events` stream to spawn. +# More than 1 worker is also supported of this type. +matrix_synapse_workers_stream_writer_events_stream_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['stream_writer_events_stream_workers_count'] }}" + +# matrix_synapse_workers_stream_writer_typing_stream_workers_count controls how many stream writers that handle the `typing` stream to spawn. +# The count of these workers can only be 0 or 1. +matrix_synapse_workers_stream_writer_typing_stream_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['stream_writer_typing_stream_workers_count'] }}" + +# matrix_synapse_workers_stream_writer_to_device_stream_workers_count controls how many stream writers that handle the `to_device` stream to spawn. +# The count of these workers can only be 0 or 1. +matrix_synapse_workers_stream_writer_to_device_stream_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['stream_writer_to_device_stream_workers_count'] }}" + +# matrix_synapse_workers_stream_writer_account_data_stream_workers_count controls how many stream writers that handle the `account_data` stream to spawn. +# The count of these workers can only be 0 or 1. +matrix_synapse_workers_stream_writer_account_data_stream_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['stream_writer_account_data_stream_workers_count'] }}" + +# matrix_synapse_workers_stream_writer_receipts_stream_workers_count controls how many stream writers that handle the `receipts` stream to spawn. +# The count of these workers can only be 0 or 1. +matrix_synapse_workers_stream_writer_receipts_stream_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['stream_writer_receipts_stream_workers_count'] }}" + +# matrix_synapse_workers_stream_writer_presence_stream_workers_count controls how many stream writers that handle the `presence` stream to spawn. +# The count of these workers can only be 0 or 1. +matrix_synapse_workers_stream_writer_presence_stream_workers_count: "{{ matrix_synapse_workers_presets[matrix_synapse_workers_preset]['stream_writer_presence_stream_workers_count'] }}" + +# A list of stream writer workers to enable. This list is built automatically based on other variables. +# You're encouraged to enable/disable stream writer workers by setting `matrix_synapse_workers_stream_writer_*_stream_workers_count` variables, instead of adjusting this list manually. +matrix_synapse_workers_stream_writers: | + {{ + [] + + + ([{'stream': 'events'}] * matrix_synapse_workers_stream_writer_events_stream_workers_count | int) + + + ([{'stream': 'typing'}] * matrix_synapse_workers_stream_writer_typing_stream_workers_count | int) + + + ([{'stream': 'to_device'}] * matrix_synapse_workers_stream_writer_to_device_stream_workers_count | int) + + + ([{'stream': 'account_data'}] * matrix_synapse_workers_stream_writer_account_data_stream_workers_count | int) + + + ([{'stream': 'receipts'}] * matrix_synapse_workers_stream_writer_receipts_stream_workers_count | int) + + + ([{'stream': 'presence'}] * matrix_synapse_workers_stream_writer_presence_stream_workers_count | int) + }} + +# matrix_synapse_stream_writers populates the `stream_writers` Synapse configuration used when Synapse workers are in use (`matrix_synapse_workers_enabled`). +# What you see below is an initial default value which will be adjusted at runtime based on the value of `matrix_synapse_workers_stream_writers`. +# Adjusting this value manually is generally not necessary. +# +# It's tempting to initialize this like this: +# matrix_synapse_stream_writers: +# - typing: [] +# - events: [] +# - to_device: [] +# - account_data: [] +# - receipts: [] +# - presence: [] +# .. but Synapse does not like empty lists (see https://github.com/matrix-org/synapse/issues/13804) +matrix_synapse_stream_writers: {} + +# `matrix_synapse_workers_stream_writer_workers_` variables control the port numbers of various stream writer workers +# defined in `matrix_synapse_workers_stream_writers`. +# It should be noted that not all of the background worker types will need to expose HTTP services, etc. +matrix_synapse_workers_stream_writer_workers_http_port_range_start: 20011 +matrix_synapse_workers_stream_writer_workers_replication_port_range_start: 25011 +matrix_synapse_workers_stream_writer_workers_metrics_range_start: 19111 + # matrix_synapse_workers_pusher_workers_count can only be 0 or 1 for now. # More instances are not supported due to a playbook limitation having to do with keeping `pusher_instances` in `homeserver.yaml` updated. # See https://github.com/matrix-org/synapse/commit/ddfdf945064925eba761ae3748e38f3a1c73c328 @@ -463,25 +540,36 @@ matrix_synapse_workers_user_dir_workers_metrics_range_start: 19661 # as certain workers can only be spawned just once. # # Each worker instance in the list defines the following fields: -# - `type` - the type of worker (`generic_worker`, etc.) -# - `instanceId` - a string that identifies the worker. The combination of (`type` + `instanceId`) represents the name of the worker and must be unique. +# - `id` - a string that uniquely identifies the worker +# - `name` - a string that will be used as the container and systemd service name +# - `type` - the type of worker (`generic_worker`, `stream_writer`, `pusher`, etc.) +# - `app` - the Synapse app (https://matrix-org.github.io/synapse/latest/workers.html#available-worker-applications) that powers this worker (`generic_worker`, `federation_sender`, etc.). +# The `app` usually matches the `type`, but not always. For example, `type = stream_writer` workers are served by the `generic_worker` type. # - `port` - an HTTP port where the worker listens for requests (can be `0` for workers that don't do HTTP request processing) # - `metrics_port` - an HTTP port where the worker exports Prometheus metrics +# - `replication_port` - an HTTP port where the worker serves `replication` endpoints (used by stream writers, etc.) +# - `webserving` - tells whether this type of worker serves web (client or federation) requests, so that it can be injected as a dependency to the reverse-proxy # # Example of what this needs to look like, if you're defining it manually: # matrix_synapse_workers_enabled_list: -# - { type: generic_worker, instanceId: '18111', port: 18111, metrics_port: 19111 } -# - { type: generic_worker, instanceId: '18112', port: 18112, metrics_port: 19112 } -# - { type: generic_worker, instanceId: '18113', port: 18113, metrics_port: 19113 } -# - { type: generic_worker, instanceId: '18114', port: 18114, metrics_port: 19114 } -# - { type: generic_worker, instanceId: '18115', port: 18115, metrics_port: 19115 } -# - { type: generic_worker, instanceId: '18116', port: 18116, metrics_port: 19116 } -# - { type: pusher, instanceId: '0', port: 0, metrics_port: 19200 } -# - { type: appservice, instanceId: '0', port: 0, metrics_port: 19300 } -# - { type: federation_sender, instanceId: '0', port: 0, metrics_port: 19400 } -# - { type: media_repository, instanceId: '18551', port: 18551, metrics_port: 19551 } +# - { 'id': 'generic-worker-0', 'name': 'matrix-synapse-worker-generic-0', 'type': 'generic_worker', 'app': 'generic_worker', 'port': 18111, 'metrics_port': 19111, 'webserving': true } +# - { 'id': 'generic-worker-1', 'name': 'matrix-synapse-worker-generic-1', 'type': 'generic_worker', 'app': 'generic_worker', 'port': 18112, 'metrics_port': 19112, 'webserving': true } +# - { 'id': 'generic-worker-2', 'name': 'matrix-synapse-worker-generic-2', 'type': 'generic_worker', 'app': 'generic_worker', 'port': 18113, 'metrics_port': 19113, 'webserving': true } +# - { 'id': 'generic-worker-3', 'name': 'matrix-synapse-worker-generic-3', 'type': 'generic_worker', 'app': 'generic_worker', 'port': 18114, 'metrics_port': 19114, 'webserving': true } +# - { 'id': 'generic-worker-4', 'name': 'matrix-synapse-worker-generic-4', 'type': 'generic_worker', 'app': 'generic_worker', 'port': 18115, 'metrics_port': 19115, 'webserving': true } +# - { 'id': 'generic-worker-5', 'name': 'matrix-synapse-worker-generic-5', 'type': 'generic_worker', 'app': 'generic_worker', 'port': 18116, 'metrics_port': 19116, 'webserving': true } +# - { 'id': 'stream-writer-0-events', 'name': 'matrix-synapse-worker-stream-writer-0-events', 'type': 'stream_writer', 'app': 'generic_worker', 'stream_writer_stream': 'events', 'port': 0, 'replication_port': 25011, metrics_port: 19111, 'webserving': false } +# - { 'id': 'stream-writer-1-typing', 'name': 'matrix-synapse-worker-stream-writer-1-typing', 'type': 'stream_writer', 'app': 'generic_worker', 'stream_writer_stream': 'typing', 'port': 20012, 'replication_port': 25012, metrics_port: 19112, 'webserving': true } +# - { 'id': 'pusher-0', 'name': 'matrix-synapse-worker-pusher-0', 'type': 'pusher', 'app': 'pusher', 'port': 0, 'metrics_port': 19200, 'webserving': false } +# - { 'id': 'appservice-0', 'name': 'matrix-synapse-worker-appservice-0', 'type': 'appservice', 'port': 0, 'metrics_port': 19300, 'webserving': false } +# - { 'id': 'federation-sender-0', 'name': 'matrix-synapse-worker-federation-sender-0', 'type': 'federation_sender', 'port': 0, 'metrics_port': 19400, 'webserving': false } +# - { 'id': 'media-repository-0', 'name': 'matrix-synapse-worker-media-repository-0', 'type': 'media_repository', 'port': 18551, 'metrics_port': 19551, 'webserving': true } matrix_synapse_workers_enabled_list: [] +# matrix_synapse_instance_map holds the instance map used for mapping worker names (for certain generic workers only!) to where they live (host, port which handles replication traffic). +# This is populated automatically based on `matrix_synapse_workers_enabled_list` during runtime, so you're not required to tweak it manually. +matrix_synapse_instance_map: {} + # Redis information matrix_synapse_redis_enabled: false matrix_synapse_redis_host: "" diff --git a/roles/matrix-synapse/tasks/init.yml b/roles/matrix-synapse/tasks/init.yml index 77696bced..8610a33b7 100644 --- a/roles/matrix-synapse/tasks/init.yml +++ b/roles/matrix-synapse/tasks/init.yml @@ -8,15 +8,15 @@ # Unless `matrix_synapse_workers_enabled_list` is explicitly defined, # we'll generate it dynamically. -- ansible.builtin.import_tasks: "{{ role_path }}/tasks/synapse/workers/init.yml" +- ansible.builtin.include_tasks: "{{ role_path }}/tasks/synapse/workers/init.yml" when: "matrix_synapse_enabled and matrix_synapse_workers_enabled and matrix_synapse_workers_enabled_list|length == 0" - ansible.builtin.set_fact: matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-synapse.service'] }}" when: matrix_synapse_enabled | bool -- name: Ensure systemd services for workers are injected - ansible.builtin.include_tasks: "{{ role_path }}/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml" +- name: Ensure workers are injected into various places + ansible.builtin.include_tasks: "{{ role_path }}/tasks/synapse/workers/util/inject_worker.yml" with_items: "{{ matrix_synapse_workers_enabled_list }}" loop_control: loop_var: matrix_synapse_worker_details @@ -65,9 +65,9 @@ matrix_synapse_worker_nginx_metrics_configuration_block: | {% for worker in matrix_synapse_workers_enabled_list %} {% if worker.metrics_port != 0 %} - location /metrics/synapse/worker/{{ worker.type }}-{{ worker.instanceId }} { + location /metrics/synapse/worker/{{ worker.id }} { resolver 127.0.0.11 valid=5s; - set $backend "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.metrics_port }}"; + set $backend "{{ worker.name }}:{{ worker.metrics_port }}"; proxy_pass http://$backend/_synapse/metrics; proxy_set_header Host $host; } diff --git a/roles/matrix-synapse/tasks/synapse/workers/init.yml b/roles/matrix-synapse/tasks/synapse/workers/init.yml index 3aa619238..fe613e70d 100644 --- a/roles/matrix-synapse/tasks/synapse/workers/init.yml +++ b/roles/matrix-synapse/tasks/synapse/workers/init.yml @@ -7,18 +7,47 @@ - name: Build generic workers ansible.builtin.set_fact: worker: + id: "generic-worker-{{ item }}" + name: "matrix-synapse-worker-generic-{{ item }}" type: 'generic_worker' - instanceId: "{{ matrix_synapse_workers_generic_workers_port_range_start + item }}" + app: 'generic_worker' + webserving: true port: "{{ matrix_synapse_workers_generic_workers_port_range_start + item }}" metrics_port: "{{ matrix_synapse_workers_generic_workers_metrics_range_start + item }}" register: "matrix_synapse_workers_list_results_generic_workers" loop: "{{ range(0, matrix_synapse_workers_generic_workers_count | int) | list }}" +- name: Build stream writer workers + ansible.builtin.set_fact: + worker: + id: "stream-writer-{{ item }}-{{ item.stream }}" + # Names must not include understores. Certain stream writer streams (to_device, account_data, ..) do, so we fix them up. + name: "matrix-synapse-worker-stream-writer-{{ idx }}-{{ item.stream | replace('_', '-') }}" + type: 'stream_writer' + app: "generic_worker" + webserving: "{{ item.stream in matrix_synapse_workers_webserving_stream_writer_types }}" + stream_writer_stream: "{{ item.stream }}" + port: "{{ matrix_synapse_workers_stream_writer_workers_http_port_range_start + idx }}" + replication_port: "{{ matrix_synapse_workers_stream_writer_workers_replication_port_range_start + idx }}" + metrics_port: "{{ matrix_synapse_workers_stream_writer_workers_metrics_range_start + idx }}" + register: "matrix_synapse_workers_list_results_stream_writer_workers" + loop: "{{ matrix_synapse_workers_stream_writers }}" + loop_control: + index_var: idx + +- name: Populate matrix_synapse_stream_writers from enabled stream writer workers list + ansible.builtin.set_fact: + matrix_synapse_stream_writers: "{{ matrix_synapse_stream_writers | combine ({item.ansible_facts.worker.stream_writer_stream: [item.ansible_facts.worker.name]}) }}" + with_items: "{{ matrix_synapse_workers_list_results_stream_writer_workers.results }}" + - name: Build federation sender workers ansible.builtin.set_fact: worker: + id: "federation-sender-{{ item }}" + name: "matrix-synapse-worker-federation-sender-{{ item }}" type: 'federation_sender' - instanceId: "{{ item }}" + app: 'federation_sender' + webserving: false port: 0 metrics_port: "{{ matrix_synapse_workers_federation_sender_workers_metrics_range_start + item }}" register: "matrix_synapse_workers_list_results_federation_sender_workers" @@ -28,8 +57,11 @@ - name: Build pusher workers ansible.builtin.set_fact: worker: + id: "pusher-{{ item }}" + name: "matrix-synapse-worker-pusher-{{ item }}" type: 'pusher' - instanceId: "{{ item }}" + app: 'pusher' + webserving: false port: 0 metrics_port: "{{ matrix_synapse_workers_pusher_workers_metrics_range_start + item }}" register: "matrix_synapse_workers_list_results_pusher_workers" @@ -39,8 +71,11 @@ - name: Build appservice workers ansible.builtin.set_fact: worker: + id: "appservice-{{ item }}" + name: "matrix-synapse-worker-appservice-{{ item }}" type: 'appservice' - instanceId: "{{ item }}" + app: 'appservice' + webserving: false port: 0 metrics_port: "{{ matrix_synapse_workers_appservice_workers_metrics_range_start + item }}" register: "matrix_synapse_workers_list_results_appservice_workers" @@ -49,8 +84,11 @@ - name: Build media_repository workers ansible.builtin.set_fact: worker: + id: "media-repository-{{ item }}" + name: "matrix-synapse-worker-media-repository-{{ item }}" type: 'media_repository' - instanceId: "{{ matrix_synapse_workers_media_repository_workers_port_range_start + item }}" + app: 'media_repository' + webserving: true port: "{{ matrix_synapse_workers_media_repository_workers_port_range_start + item }}" metrics_port: "{{ matrix_synapse_workers_media_repository_workers_metrics_range_start + item }}" register: "matrix_synapse_workers_list_results_media_repository_workers" @@ -62,6 +100,8 @@ {{ matrix_synapse_workers_list_results_generic_workers.results + + matrix_synapse_workers_list_results_stream_writer_workers.results + + matrix_synapse_workers_list_results_federation_sender_workers.results + matrix_synapse_workers_list_results_pusher_workers.results diff --git a/roles/matrix-synapse/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml b/roles/matrix-synapse/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml deleted file mode 100644 index 2ecb3f2bf..000000000 --- a/roles/matrix-synapse/tasks/synapse/workers/util/inject_systemd_services_for_worker.yml +++ /dev/null @@ -1,19 +0,0 @@ ---- -# The tasks below run before `validate_config.yml`. -# To avoid failing with a cryptic error message, we'll do validation here. -# -# This check is mostly relevant to people who explicitly define `matrix_synapse_workers_enabled_list` -# (Synapse Workers users from the earlier days of this PR - https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/456). -# -# In the future, it should be possible to remove this check. -# Our own code which dynamically builds `matrix_synapse_workers_enabled_list` does things right. -- name: Fail if instanceId not defined for worker - ansible.builtin.fail: - msg: "Synapse workers (like {{ matrix_synapse_worker_details | to_json }}) need to define an instanceId property (type + instanceId must be unique)" - when: "'instanceId' not in matrix_synapse_worker_details" - -- ansible.builtin.set_fact: - matrix_synapse_worker_systemd_service_name: "matrix-synapse-worker-{{ matrix_synapse_worker_details.type }}-{{ matrix_synapse_worker_details.instanceId }}.service" - -- ansible.builtin.set_fact: - matrix_systemd_services_list: "{{ matrix_systemd_services_list + [matrix_synapse_worker_systemd_service_name] }}" diff --git a/roles/matrix-synapse/tasks/synapse/workers/util/inject_worker.yml b/roles/matrix-synapse/tasks/synapse/workers/util/inject_worker.yml new file mode 100644 index 000000000..4f6b43986 --- /dev/null +++ b/roles/matrix-synapse/tasks/synapse/workers/util/inject_worker.yml @@ -0,0 +1,65 @@ +--- +# The tasks below run before `validate_config.yml`. +# To avoid failing with a cryptic error message, we'll do validation here. +# +# This check is mostly relevant to people who explicitly define `matrix_synapse_workers_enabled_list` +# (Synapse Workers users from the earlier days of this PR - https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/456). +# +# In the future, it should be possible to remove this check. +# Our own code which dynamically builds `matrix_synapse_workers_enabled_list` does things right. +- name: Fail if required property not defined for worker + ansible.builtin.fail: + msg: "Synapse workers (like {{ matrix_synapse_worker_details | to_json }}) need to define a `{{ item }}` property" + with_items: + - id + - name + - type + - app + - port + - webserving + when: "item not in matrix_synapse_worker_details" + +# Names are used for container names and systemd services. +# Routing happens based on container names, so Synapse processes that try to route to workers with underscores in the name will complain. Example: +# > InvalidCodepoint Codepoint U+005F at position 46 of 'matrix-synapse-worker-stream-writer-3-account_data' not allowed +- name: Fail if worker name includes underscore + ansible.builtin.fail: + msg: "Unrecognized Synapse worker `name`: `{{ matrix_synapse_worker_details.name }}`. It must not include underscores" + when: "'_' in matrix_synapse_worker_details.name" + +- name: Fail if worker type unknown + ansible.builtin.fail: + msg: "Unrecognized Synapse worker `type`: `{{ matrix_synapse_worker_details.type }}`. Supported types are: {{ matrix_synapse_known_worker_types | join(', ') }}" + when: "matrix_synapse_worker_details.type not in matrix_synapse_known_worker_types" + +- name: Fail if worker app unknown + ansible.builtin.fail: + msg: "Unrecognized Synapse worker `app`: `{{ matrix_synapse_worker_details.app }}`. Supported types are: {{ matrix_synapse_workers_avail_list | join(', ') }}" + when: "matrix_synapse_worker_details.app not in matrix_synapse_workers_avail_list" + +- block: + - name: Fail if stream_writer_stream not defined for stream_writer worker + ansible.builtin.fail: + msg: >- + Synapse stream_writer workers (such as {{ item }}) need to define a valid `stream_writer_stream` property + (not `{{ matrix_synapse_worker_details.stream_writer_stream|default('undefined') }}`). + Supported types are: {{ matrix_synapse_workers_known_stream_writer_stream_types | join(', ') }} + when: "'stream_writer_stream' not in matrix_synapse_worker_details or matrix_synapse_worker_details.stream_writer_stream not in matrix_synapse_workers_known_stream_writer_stream_types" + + - name: Fail if replication_port not defined for stream_writer worker + ansible.builtin.fail: + msg: "Synapse background workers of type stream_writer (such as {{ item }}) need to define a valid `replication_port` property" + when: "'replication_port' not in matrix_synapse_worker_details" + when: "matrix_synapse_worker_details.type == 'stream_writer'" + +- ansible.builtin.set_fact: + matrix_systemd_services_list: "{{ matrix_systemd_services_list + [matrix_synapse_worker_details.name + '.service'] }}" + +- ansible.builtin.set_fact: + matrix_synapse_webserving_workers_systemd_services_list: "{{ matrix_synapse_webserving_workers_systemd_services_list + [matrix_synapse_worker_details.name + '.service'] }}" + when: matrix_synapse_worker_details.webserving | bool + +# Inject stream writers and various other background workers into the instance map. +- ansible.builtin.set_fact: + matrix_synapse_instance_map: "{{ matrix_synapse_instance_map | combine({matrix_synapse_worker_details.name: {'host': matrix_synapse_worker_details.name, 'port': matrix_synapse_worker_details.replication_port}}) }}" + when: matrix_synapse_worker_details.type in matrix_synapse_known_instance_map_eligible_worker_types diff --git a/roles/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml b/roles/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml index d6d4924fb..d3f309175 100644 --- a/roles/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml +++ b/roles/matrix-synapse/tasks/synapse/workers/util/setup_files_for_worker.yml @@ -1,13 +1,9 @@ --- - ansible.builtin.set_fact: - matrix_synapse_worker_systemd_service_name: "matrix-synapse-worker-{{ matrix_synapse_worker_details.type }}-{{ matrix_synapse_worker_details.instanceId }}" - -- ansible.builtin.set_fact: - matrix_synapse_worker_container_name: "{{ matrix_synapse_worker_systemd_service_name }}" - -- ansible.builtin.set_fact: - matrix_synapse_worker_config_file_name: "worker.{{ matrix_synapse_worker_details.type }}_{{ matrix_synapse_worker_details.instanceId }}.yaml" + matrix_synapse_worker_systemd_service_name: "{{ matrix_synapse_worker_details.name }}" + matrix_synapse_worker_container_name: "{{ matrix_synapse_worker_details.name }}" + matrix_synapse_worker_config_file_name: "worker.{{ matrix_synapse_worker_details.name }}.yaml" - name: Ensure configuration exists for {{ matrix_synapse_worker_systemd_service_name }} ansible.builtin.template: diff --git a/roles/matrix-synapse/tasks/validate_config.yml b/roles/matrix-synapse/tasks/validate_config.yml index bcb71c756..1fc708166 100644 --- a/roles/matrix-synapse/tasks/validate_config.yml +++ b/roles/matrix-synapse/tasks/validate_config.yml @@ -32,6 +32,11 @@ with_items: - "matrix_synapse_workers_pusher_workers_count" - "matrix_synapse_workers_federation_sender_workers_count" + - "matrix_synapse_workers_stream_writer_typing_stream_workers_count" + - "matrix_synapse_workers_stream_writer_to_device_stream_workers_count" + - "matrix_synapse_workers_stream_writer_account_data_stream_workers_count" + - "matrix_synapse_workers_stream_writer_receipts_stream_workers_count" + - "matrix_synapse_workers_stream_writer_presence_stream_workers_count" - name: (Deprecation) Catch and report renamed settings ansible.builtin.fail: diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 index 04c4b3cf5..86d03be41 100644 --- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 +++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 @@ -352,13 +352,13 @@ worker_app: synapse.app.homeserver # thx https://oznetnerd.com/2017/04/18/jinja2-selectattr-filter/ # reduce the main worker's offerings to core homeserver business -{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'federation_sender') | list %} +{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'federation_sender') | list | length > 0 %} send_federation: false {% endif %} -{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list %} +{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'media_repository') | list | length > 0 %} enable_media_repo: false {% endif %} -{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'pusher') | list %} +{% if matrix_synapse_workers_enabled_list | selectattr('type', 'equalto', 'pusher') | list | length > 0 %} start_pushers: false {% endif %} @@ -2870,6 +2870,7 @@ opentracing: # worker1: # host: localhost # port: 8034 +instance_map: {{ matrix_synapse_instance_map | to_json }} # Experimental: When using workers you can define which workers should # handle event persistence and typing notifications. Any worker @@ -2878,6 +2879,7 @@ opentracing: #stream_writers: # events: worker1 # typing: worker1 +stream_writers: {{ matrix_synapse_stream_writers | to_json }} # The worker that is used to run background tasks (e.g. cleaning up expired # data). If not provided this defaults to the main process. diff --git a/roles/matrix-synapse/templates/synapse/prometheus/external_prometheus.yml.example.j2 b/roles/matrix-synapse/templates/synapse/prometheus/external_prometheus.yml.example.j2 index b194c3c2d..1501697d7 100644 --- a/roles/matrix-synapse/templates/synapse/prometheus/external_prometheus.yml.example.j2 +++ b/roles/matrix-synapse/templates/synapse/prometheus/external_prometheus.yml.example.j2 @@ -24,8 +24,8 @@ scrape_configs: job: "master" index: "0" {% for worker in matrix_synapse_workers_enabled_list %} - - job_name: 'synapse-{{ worker.type }}-{{ worker.instanceId }}' - metrics_path: /metrics/synapse/worker/{{ worker.type }}-{{ worker.instanceId }} + - job_name: '{{ worker.name }}' + metrics_path: /metrics/synapse/worker/{{ worker.id }} scheme: {{ 'https' if matrix_nginx_proxy_https_enabled|default(true) else 'http' }} {% if matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled|default(true) %} basic_auth: @@ -35,6 +35,7 @@ scrape_configs: static_configs: - targets: ['{{ matrix_server_fqn_matrix }}:{{ matrix_nginx_proxy_container_https_host_bind_port|default(443) if matrix_nginx_proxy_https_enabled|default(true) else matrix_nginx_proxy_container_http_host_bind_port|default(80) }}'] labels: + worker_id: {{ worker.id }} job: "{{ worker.type }}" - index: "{{ worker.instanceId }}" + app: {{ worker.app }} {% endfor %} diff --git a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 index 4a38251d7..96b65a0ae 100644 --- a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 +++ b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse-worker.service.j2 @@ -43,7 +43,7 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name {{ matrix_synapse_wor {{ arg }} \ {% endfor %} {{ matrix_synapse_docker_image }} \ - run -m synapse.app.{{ matrix_synapse_worker_details.type }} -c /data/homeserver.yaml -c /data/{{ matrix_synapse_worker_config_file_name }} + run -m synapse.app.{{ matrix_synapse_worker_details.app }} -c /data/homeserver.yaml -c /data/{{ matrix_synapse_worker_config_file_name }} ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill {{ matrix_synapse_worker_container_name }} 2>/dev/null || true' diff --git a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 index 027114fb3..eed50ad1e 100644 --- a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 +++ b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 @@ -12,7 +12,7 @@ Wants={{ service }} {% if matrix_synapse_workers_enabled %} {% for matrix_synapse_worker_details in matrix_synapse_workers_enabled_list %} -Wants=matrix-synapse-worker-{{ matrix_synapse_worker_details.type }}-{{ matrix_synapse_worker_details.port }}.service +Wants={{ matrix_synapse_worker_details.name }}.service {% endfor %} {% endif %} diff --git a/roles/matrix-synapse/templates/synapse/worker.yaml.j2 b/roles/matrix-synapse/templates/synapse/worker.yaml.j2 index 33789b0ca..c9637a839 100644 --- a/roles/matrix-synapse/templates/synapse/worker.yaml.j2 +++ b/roles/matrix-synapse/templates/synapse/worker.yaml.j2 @@ -1,27 +1,38 @@ #jinja2: lstrip_blocks: "True" -worker_app: synapse.app.{{ matrix_synapse_worker_details.type }} -worker_name: {{ matrix_synapse_worker_details.type ~ ':' ~ matrix_synapse_worker_details.port }} +worker_app: synapse.app.{{ matrix_synapse_worker_details.app }} +worker_name: {{ matrix_synapse_worker_details.name }} {% if matrix_synapse_replication_listener_enabled %} worker_replication_host: matrix-synapse worker_replication_http_port: {{ matrix_synapse_replication_http_port }} {% endif %} -{% set has_listeners = (matrix_synapse_worker_details.type not in [ 'appservice', 'federation_sender', 'pusher' ] or matrix_synapse_metrics_enabled) %} - {% set http_resources = [] %} -{% if matrix_synapse_worker_details.type in ['generic_worker', 'user_dir'] %} +{% if matrix_synapse_worker_details.type == 'user_dir' %} {% set http_resources = http_resources + ['client'] %} {% endif %} -{% if matrix_synapse_worker_details.type in ['generic_worker'] %} - {% set http_resources = http_resources+ ['federation'] %} +{% if matrix_synapse_worker_details.type == 'generic_worker' %} + {% set http_resources = http_resources + ['client', 'federation'] %} {% endif %} -{% if matrix_synapse_worker_details.type in ['media_repository'] %} +{# + None of the background workers need to handle federation traffic. + Only some of the stream writers need to handle client traffic. +#} +{% if matrix_synapse_worker_details.type == 'stream_writer' and matrix_synapse_worker_details.webserving %} + {% set http_resources = http_resources + ['client'] %} +{% endif %} +{% if matrix_synapse_worker_details.type == 'media_repository' %} {% set http_resources = http_resources + ['media'] %} {% endif %} -{% if http_resources|length > 0 or matrix_synapse_metrics_enabled %} +{% set replication_http_resources = [] %} +{% if matrix_synapse_worker_details.type == 'stream_writer' %} + {# All background workers need to handle replication traffic. #} + {% set replication_http_resources = replication_http_resources + ['replication'] %} +{% endif %} + +{% if http_resources|length > 0 or matrix_synapse_metrics_enabled or replication_http_resources|length > 0 %} worker_listeners: {% if http_resources|length > 0 %} - type: http @@ -36,6 +47,13 @@ worker_listeners: bind_addresses: ['0.0.0.0'] port: {{ matrix_synapse_worker_details.metrics_port }} {% endif %} +{% if replication_http_resources|length > 0 %} + - type: http + bind_addresses: ['::'] + port: {{ matrix_synapse_worker_details.replication_port }} + resources: + - names: {{ replication_http_resources|to_json }} +{% endif %} {% endif %} {% if matrix_synapse_worker_details.type == 'generic_worker' %} diff --git a/roles/matrix-synapse/vars/main.yml b/roles/matrix-synapse/vars/main.yml index 2d9b62cf7..e8b0e49c6 100644 --- a/roles/matrix-synapse/vars/main.yml +++ b/roles/matrix-synapse/vars/main.yml @@ -36,3 +36,58 @@ matrix_synapse_workers_generic_worker_federation_endpoints: "{{ matrix_synapse_w # matrix_synapse_workers_generic_worker_federation_endpoints_regex contains the regex used in matrix_synapse_workers_generic_worker_federation_endpoints. # It's intentionally put in a separate variable, to avoid tripping ansible-lint's var-spacing rule. matrix_synapse_workers_generic_worker_federation_endpoints_regex: '.*(/_matrix/federation|/_matrix/key).*' + +# matrix_synapse_workers_stream_writer_typing_stream_worker_client_server_endpoints contains the endpoints serviced by the `typing` stream writer. +# See: https://matrix-org.github.io/synapse/latest/workers.html#the-typing-stream +matrix_synapse_workers_stream_writer_typing_stream_worker_client_server_endpoints: + - ^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/typing + +# matrix_synapse_workers_stream_writer_to_device_stream_worker_client_server_endpoints contains the endpoints serviced by the `to_device` stream writer. +# See: https://matrix-org.github.io/synapse/latest/workers.html#the-to_device-stream +matrix_synapse_workers_stream_writer_to_device_stream_worker_client_server_endpoints: + - ^/_matrix/client/(r0|v3|unstable)/sendToDevice/ + +# matrix_synapse_workers_stream_writer_account_data_stream_worker_client_server_endpoints contains the endpoints serviced by the `account_data` stream writer. +# See: https://matrix-org.github.io/synapse/latest/workers.html#the-account_data-stream +matrix_synapse_workers_stream_writer_account_data_stream_worker_client_server_endpoints: + - ^/_matrix/client/(r0|v3|unstable)/.*/tags + - ^/_matrix/client/(r0|v3|unstable)/.*/account_data + +# matrix_synapse_workers_stream_writer_receipts_stream_worker_client_server_endpoints contains the endpoints serviced by the `recepts` stream writer. +# See: https://matrix-org.github.io/synapse/latest/workers.html#the-receipts-stream +matrix_synapse_workers_stream_writer_receipts_stream_worker_client_server_endpoints: + - ^/_matrix/client/(r0|v3|unstable)/rooms/.*/receipt + - ^/_matrix/client/(r0|v3|unstable)/rooms/.*/read_markers + +# matrix_synapse_workers_stream_writer_presence_stream_worker_client_server_endpoints contains the endpoints serviced by the `presence` stream writer. +# See: https://matrix-org.github.io/synapse/latest/workers.html#the-presence-stream +matrix_synapse_workers_stream_writer_presence_stream_worker_client_server_endpoints: + - ^/_matrix/client/(api/v1|r0|v3|unstable)/presence/ + +# matrix_synapse_workers_known_stream_writer_stream_types contains the list of stream writer stream types that the playbook recognizes. +# This is used for validation purposes. If adding support for a new type, besides adding it to this list, +# don't forget to actually configure it where appropriate (see worker.yaml.j2`, the nginx proxy configuration, etc). +matrix_synapse_workers_known_stream_writer_stream_types: ['events', 'typing', 'to_device', 'account_data', 'receipts', 'presence'] + +# matrix_synapse_workers_webserving_stream_writer_types contains a list of stream writer types that serve web (client) requests. +# Not all stream writers serve web requests. Some just perform background tasks. +matrix_synapse_workers_webserving_stream_writer_types: ['typing', 'to_device', 'account_data', 'receipts', 'presence'] + +# matrix_synapse_workers_systemd_services_list contains a list of systemd services (one for each worker systemd service which serves web requests). +# This list is built during runtime. +# Not all workers serve web requests. Those that don't won't be injected here. +matrix_synapse_webserving_workers_systemd_services_list: [] + +# matrix_synapse_known_worker_types contains the list of known worker types. +# A worker type is different than a worker app (e.g. `generic_worker`). +# For example, the `stream_writer` worker type is served by the `generic_worker` app, but is a separate type that we recognize. +matrix_synapse_known_worker_types: | + {{ + matrix_synapse_workers_avail_list + + + ['stream_writer'] + }} + +# matrix_synapse_known_instance_map_eligible_worker_types contains the list of worker types that are to be injected into `matrix_synapse_instance_map`. +matrix_synapse_known_instance_map_eligible_worker_types: + - stream_writer