feat: add automatic creation of reverse-proxy routing

add server_name to matrix-synapse.conf only if matrix_nginx_proxy_enabled

docs/alternative-architectures.md

@@ -21,6 +21,6 @@ matrix_architecture: "arm32"
 
 ## Implementation details
 
-For `amd64`, prebuilt container images (see the [container images we use](container-images.md)) are used everywhere, because all images are available for this architecture.
+For `amd64`, prebuilt container images (see the [container images we use](container-images.md)) are used for all components (except [Hydrogen](configuring-playbook-client-hydrogen.md), which goes through self-building).
 
 For other architectures, components which have a prebuilt image make use of it. If the component is not available for the specific architecture, [self-building](self-building.md) will be used. Not all components support self-building though, so your mileage may vary.

docs/ansible.md

@@ -51,7 +51,7 @@ docker run -it --rm \
 -v `pwd`:/work \
 -v $HOME/.ssh/id_rsa:/root/.ssh/id_rsa:ro \
 --entrypoint=/bin/sh \
-docker.io/devture/ansible:2.9.14-r0
+docker.io/devture/ansible:2.10.7-r0
 ```
 
 The above command tries to mount an SSH key (`$HOME/.ssh/id_rsa`) into the container (at `/root/.ssh/id_rsa`).

docs/configuring-playbook-prometheus-grafana.md

@@ -55,9 +55,41 @@ Name | Description
 `matrix_synapse_metrics_enabled`|Set this to `true` to make Synapse expose metrics (locally, on the container network)
 `matrix_nginx_proxy_proxy_synapse_metrics`|Set this to `true` to make matrix-nginx-proxy expose the Synapse metrics at `https://matrix.DOMAIN/_synapse/metrics`
 `matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled`|Set this to `true` to password-protect (using HTTP Basic Auth) `https://matrix.DOMAIN/_synapse/metrics` (the username is always `prometheus`, the password is defined in `matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key`)
-`matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key`|Set this to a password to use for HTTP Basic Auth for protecting `https://matrix.DOMAIN/_synapse/metrics` (the username is always `prometheus` - it's not configurable)
+`matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key`|Set this to a password to use for HTTP Basic Auth for protecting `https://matrix.DOMAIN/_synapse/metrics` (the username is always `prometheus` - it's not configurable). Do not write the password in plain text. See `man 1 htpasswd` or use `htpasswd -c mypass.htpasswd prometheus` to generate the expected hash for nginx. 
 `matrix_server_fqn_grafana`|Use this variable to override the domain at which the Grafana web user-interface is at (defaults to `stats.DOMAIN`)
 
+### Collecting worker metrics to an external Prometheus server
+
+If you are using workers (`matrix_synapse_workers_enabled`) and have enabled `matrix_nginx_proxy_proxy_synapse_metrics` as described above, the playbook will also automatically proxy the all worker threads's metrics to `https://matrix.DOMAIN/_synapse-worker-TYPE-ID/metrics`, where `TYPE` corresponds to the type and `ID` to the instanceId of a worker as exemplified in `matrix_synapse_workers_enabled_list`.
+
+The playbook also generates an exemplary prometheus.yml config file (`matrix_base_data_path/external_prometheus.yml.template`) with all the correct paths which you can copy to your Prometheus server and adapt to your needs, especially edit the specified `password_file` path and contents and path to your `synapse-v2.rules`.
+It will look a bit like this:
+```yaml
+scrape_configs:
+  - job_name: 'synapse'
+    metrics_path: /_synapse/metrics
+    scheme: https
+    basic_auth:
+      username: prometheus
+      password_file: /etc/prometheus/password.pwd
+    static_configs:
+      - targets: ['matrix.DOMAIN:443']
+        labels:
+          job: "master"
+          index: 1
+  - job_name: 'synapse-generic_worker-1'
+    metrics_path: /_synapse-worker-generic_worker-18111/metrics
+    scheme: https
+    basic_auth:
+      username: prometheus
+      password_file: /etc/prometheus/password.pwd
+    static_configs:
+      - targets: ['matrix.DOMAIN:443']
+        labels:
+          job: "generic_worker"
+          index: 18111
+```
+
 ### Collecting system and Postgres metrics to an external Prometheus server (advanced)
 
 When you normally enable the Prometheus and Grafana via the playbook, it will also show general system (via node-exporter) and Postgres (via postgres-exporter) stats. If you are instead collecting your metrics to an external Prometheus server, you can follow this advanced configuration example to also export these stats.

examples/caddy/matrix-synapse

@@ -5,7 +5,7 @@ https://matrix.DOMAIN {
 
 	root /matrix/static-files
 
-	header {
+	header / {
 		Access-Control-Allow-Origin *
 		Strict-Transport-Security "mag=age=31536000;"
 		X-Frame-Options "DENY"
@@ -13,10 +13,10 @@ https://matrix.DOMAIN {
 	}
 
 	# Identity server traffic
-	proxy /_matrix/identity matrix-msisd:8090 {
+	proxy /_matrix/identity matrix-ma1sd:8090 {
 		transparent
 	}
-	proxy /_matrix/client/r0/user_directory/search matrix-msisd:8090 {
+	proxy /_matrix/client/r0/user_directory/search matrix-ma1sd:8090 {
 		transparent
 	}
 

group_vars/matrix_servers

@@ -1212,7 +1212,7 @@ matrix_ma1sd_container_image_self_build: "{{ matrix_architecture != 'amd64' }}"
 # Normally, matrix-nginx-proxy is enabled and nginx can reach ma1sd over the container network.
 # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose
 # ma1sd's web-server port.
-matrix_ma1sd_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:8090' }}"
+matrix_ma1sd_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:' + matrix_ma1sd_default_port|string }}"
 
 # We enable Synapse integration via its Postgres database by default.
 # When using another Identity store, you might wish to disable this and define
@@ -1294,8 +1294,8 @@ matrix_nginx_proxy_proxy_matrix_corporal_api_addr_with_container: "matrix-corpor
 matrix_nginx_proxy_proxy_matrix_corporal_api_addr_sans_container: "127.0.0.1:41081"
 
 matrix_nginx_proxy_proxy_matrix_identity_api_enabled: "{{ matrix_ma1sd_enabled }}"
-matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container: "matrix-ma1sd:8090"
+matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container: "matrix-ma1sd:{{ matrix_ma1sd_default_port }}"
-matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container: "127.0.0.1:8090"
+matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container: "127.0.0.1:{{ matrix_ma1sd_default_port }}"
 
 # By default, we do TLS termination for the Matrix Federation API (port 8448) at matrix-nginx-proxy.
 # Unless this is handled there OR Synapse's federation listener port is disabled, we'll reverse-proxy.
@@ -1339,6 +1339,8 @@ matrix_nginx_proxy_synapse_media_repository_locations: "{{matrix_synapse_workers
 matrix_nginx_proxy_synapse_user_dir_locations: "{{ matrix_synapse_workers_user_dir_endpoints|default([]) }}"
 matrix_nginx_proxy_synapse_frontend_proxy_locations: "{{ matrix_synapse_workers_frontend_proxy_endpoints|default([]) }}"
 
+matrix_nginx_proxy_proxy_synapse_workers_enabled_list: "{{ matrix_synapse_workers_enabled_list }}"
+
 matrix_nginx_proxy_systemd_wanted_services_list: |
   {{
     (['matrix-synapse.service'] if matrix_synapse_enabled else [])
@@ -1707,8 +1709,8 @@ matrix_synapse_container_image_self_build: "{{ matrix_architecture not in ['arm6
 
 # When ma1sd is enabled, we can use it to validate email addresses and phone numbers.
 # Synapse can validate email addresses by itself as well, but it's probably not what we want by default when we have an identity server.
-matrix_synapse_account_threepid_delegates_email: "{{ 'http://matrix-ma1sd:8090' if matrix_ma1sd_enabled else '' }}"
+matrix_synapse_account_threepid_delegates_email: "{{ 'http://matrix-ma1sd:' + matrix_ma1sd_default_port|string if matrix_ma1sd_enabled else '' }}"
-matrix_synapse_account_threepid_delegates_msisdn: "{{ 'http://matrix-ma1sd:8090' if matrix_ma1sd_enabled else '' }}"
+matrix_synapse_account_threepid_delegates_msisdn: "{{ 'http://matrix-ma1sd:' + matrix_ma1sd_default_port|string if matrix_ma1sd_enabled else '' }}"
 
 # Normally, matrix-nginx-proxy is enabled and nginx can reach Synapse over the container network.
 # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it,

roles/matrix-awx/surveys/bridge_discord_appservice.json.j2

@@ -15,7 +15,7 @@
       "type": "multiplechoice"
     },
     {
-      "question_name": "Discord Client ID",
+      "question_name": "Discord OAuth2 Client ID",
       "question_description": "The OAuth2 'CLIENT ID' which can be found in the 'OAuth2' tab of your new discord application: https://discord.com/developers/applications",
       "required": true,
       "min": 0,

roles/matrix-awx/tasks/set_variables_ma1sd.yml

@@ -30,7 +30,7 @@
     insertafter: '# Synapse Extension Start'
   with_dict:
     'matrix_synapse_awx_password_provider_rest_auth_enabled': 'true'
-    'matrix_synapse_awx_password_provider_rest_auth_endpoint': '"http://matrix-ma1sd:8090"'
+    'matrix_synapse_awx_password_provider_rest_auth_endpoint': '"http://matrix-ma1sd:{{ matrix_ma1sd_default_port }}"'
   when: awx_matrix_ma1sd_auth_store == 'LDAP/AD'
 
 - name: Remove entire ma1sd configuration extension

roles/matrix-bridge-appservice-irc/defaults/main.yml

@@ -7,7 +7,7 @@ matrix_appservice_irc_container_self_build: false
 matrix_appservice_irc_docker_repo: "https://github.com/matrix-org/matrix-appservice-irc.git"
 matrix_appservice_irc_docker_src_files_path: "{{ matrix_base_data_path }}/appservice-irc/docker-src"
 
-matrix_appservice_irc_version: release-0.31.0
+matrix_appservice_irc_version: release-0.32.1
 matrix_appservice_irc_docker_image: "{{ matrix_container_global_registry_prefix }}matrixdotorg/matrix-appservice-irc:{{ matrix_appservice_irc_version }}"
 matrix_appservice_irc_docker_image_force_pull: "{{ matrix_appservice_irc_docker_image.endswith(':latest') }}"
 

roles/matrix-bridge-heisenbridge/defaults/main.yml

@@ -3,7 +3,7 @@
 
 matrix_heisenbridge_enabled: true
 
-matrix_heisenbridge_version: 1.3.0
+matrix_heisenbridge_version: 1.5.0
 matrix_heisenbridge_docker_image: "{{ matrix_container_global_registry_prefix }}hif1/heisenbridge:{{ matrix_heisenbridge_version }}"
 matrix_heisenbridge_docker_image_force_pull: "{{ matrix_heisenbridge_docker_image.endswith(':latest') }}"
 

roles/matrix-client-element/defaults/main.yml

@@ -2,8 +2,12 @@ matrix_client_element_enabled: true
 
 matrix_client_element_container_image_self_build: false
 matrix_client_element_container_image_self_build_repo: "https://github.com/vector-im/riot-web.git"
+# Controls whether to patch webpack.config.js when self-building, so that building can pass on low-memory systems (< 4 GB RAM):
+# - https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1357
+# - https://github.com/vector-im/element-web/issues/19544
+matrix_client_element_container_image_self_build_low_memory_system_patch_enabled: "{{ ansible_memtotal_mb < 4096 }}"
 
-matrix_client_element_version: v1.9.2
+matrix_client_element_version: v1.9.3
 matrix_client_element_docker_image: "{{ matrix_client_element_docker_image_name_prefix }}vectorim/element-web:{{ matrix_client_element_version }}"
 matrix_client_element_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_element_container_image_self_build else matrix_container_global_registry_prefix }}"
 matrix_client_element_docker_image_force_pull: "{{ matrix_client_element_docker_image.endswith(':latest') }}"

roles/matrix-client-element/tasks/main.yml

@@ -20,8 +20,14 @@
     - setup-all
     - setup-client-element
 
-- import_tasks: "{{ role_path }}/tasks/setup.yml"
+- import_tasks: "{{ role_path }}/tasks/setup_install.yml"
-  when: run_setup|bool
+  when: "run_setup|bool and matrix_client_element_enabled|bool"
+  tags:
+    - setup-all
+    - setup-client-element
+
+- import_tasks: "{{ role_path }}/tasks/setup_uninstall.yml"
+  when: "run_setup|bool and not matrix_client_element_enabled|bool"
   tags:
     - setup-all
     - setup-client-element

roles/matrix-client-element/tasks/setup_install.yml

@@ -1,9 +1,5 @@
 ---
 
-#
-# Tasks related to setting up Element
-#
-
 - name: Ensure Element paths exists
   file:
     path: "{{ item.path }}"
@@ -14,7 +10,7 @@
   with_items:
     - { path: "{{ matrix_client_element_data_path }}", when: true }
     - { path: "{{ matrix_client_element_docker_src_files_path }}", when: "{{ matrix_client_element_container_image_self_build }}" }
-  when: matrix_client_element_enabled|bool and item.when
+  when: "item.when|bool"
 
 - name: Ensure Element Docker image is pulled
   docker_image:
@@ -22,7 +18,7 @@
     source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}"
     force_source: "{{ matrix_client_element_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}"
     force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_client_element_docker_image_force_pull }}"
-  when: matrix_client_element_enabled|bool and not matrix_client_element_container_image_self_build
+  when: "not matrix_client_element_container_image_self_build|bool"
 
 - name: Ensure Element repository is present on self-build
   git:
@@ -31,7 +27,21 @@
     version: "{{ matrix_client_element_docker_image.split(':')[1] }}"
     force: "yes"
   register: matrix_client_element_git_pull_results
-  when: "matrix_client_element_enabled|bool and matrix_client_element_container_image_self_build|bool"
+  when: "matrix_client_element_container_image_self_build|bool"
+
+# See:
+# - https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1357
+# - https://github.com/vector-im/element-web/issues/19544
+- name: Patch webpack.config.js to support building on low-memory (<4G RAM) devices
+  lineinfile:
+    path: "{{ matrix_client_element_docker_src_files_path }}/webpack.config.js"
+    regexp: '(\s+)splitChunks: \{'
+    line: '\1splitChunks: { maxSize: 100000,'
+    backrefs: yes
+    owner: root
+    group: root
+    mode: '0644'
+  when: "matrix_client_element_container_image_self_build|bool and matrix_client_element_container_image_self_build_low_memory_system_patch_enabled|bool"
 
 - name: Ensure Element Docker image is built
   docker_image:
@@ -43,7 +53,7 @@
       dockerfile: Dockerfile
       path: "{{ matrix_client_element_docker_src_files_path }}"
       pull: yes
-  when: "matrix_client_element_enabled|bool and matrix_client_element_container_image_self_build|bool"
+  when: "matrix_client_element_container_image_self_build|bool"
 
 - name: Ensure Element configuration installed
   copy:
@@ -52,7 +62,6 @@
     mode: 0644
     owner: "{{ matrix_user_username }}"
     group: "{{ matrix_user_groupname }}"
-  when: matrix_client_element_enabled|bool
 
 - name: Ensure Element config files installed
   template:
@@ -65,7 +74,7 @@
     - {src: "{{ role_path }}/templates/nginx.conf.j2", name: "nginx.conf"}
     - {src: "{{ matrix_client_element_page_template_welcome_path }}", name: "welcome.html"}
     - {src: "{{ matrix_client_element_embedded_pages_home_path }}", name: "home.html"}
-  when: "matrix_client_element_enabled|bool and item.src is not none"
+  when: "item.src is not none"
 
 - name: Copy Element costum files
   copy:
@@ -85,7 +94,7 @@
     state: absent
   with_items:
     - {src: "{{ matrix_client_element_embedded_pages_home_path }}", name: "home.html"}
-  when: "matrix_client_element_enabled|bool and item.src is none"
+  when: "item.src is none"
 
 - name: Ensure matrix-client-element.service installed
   template:
@@ -93,50 +102,8 @@
     dest: "{{ matrix_systemd_path }}/matrix-client-element.service"
     mode: 0644
   register: matrix_client_element_systemd_service_result
-  when: matrix_client_element_enabled|bool
 
 - name: Ensure systemd reloaded after matrix-client-element.service installation
   service:
     daemon_reload: yes
-  when: "matrix_client_element_enabled and matrix_client_element_systemd_service_result.changed"
+  when: "matrix_client_element_systemd_service_result.changed|bool"
-
-#
-# Tasks related to getting rid of Element (if it was previously enabled)
-#
-
-- name: Check existence of matrix-client-element.service
-  stat:
-    path: "{{ matrix_systemd_path }}/matrix-client-element.service"
-  register: matrix_client_element_service_stat
-  when: "not matrix_client_element_enabled|bool"
-
-- name: Ensure matrix-client-element is stopped
-  service:
-    name: matrix-client-element
-    state: stopped
-    daemon_reload: yes
-  register: stopping_result
-  when: "not matrix_client_element_enabled|bool and matrix_client_element_service_stat.stat.exists"
-
-- name: Ensure matrix-client-element.service doesn't exist
-  file:
-    path: "{{ matrix_systemd_path }}/matrix-client-element.service"
-    state: absent
-  when: "not matrix_client_element_enabled|bool and matrix_client_element_service_stat.stat.exists"
-
-- name: Ensure systemd reloaded after matrix-client-element.service removal
-  service:
-    daemon_reload: yes
-  when: "not matrix_client_element_enabled|bool and matrix_client_element_service_stat.stat.exists"
-
-- name: Ensure Element paths doesn't exist
-  file:
-    path: "{{ matrix_client_element_data_path }}"
-    state: absent
-  when: "not matrix_client_element_enabled|bool"
-
-- name: Ensure Element Docker image doesn't exist
-  docker_image:
-    name: "{{ matrix_client_element_docker_image }}"
-    state: absent
-  when: "not matrix_client_element_enabled|bool"

roles/matrix-client-element/tasks/setup_uninstall.yml

@@ -0,0 +1,35 @@
+---
+
+- name: Check existence of matrix-client-element.service
+  stat:
+    path: "{{ matrix_systemd_path }}/matrix-client-element.service"
+  register: matrix_client_element_service_stat
+
+- name: Ensure matrix-client-element is stopped
+  service:
+    name: matrix-client-element
+    state: stopped
+    daemon_reload: yes
+  register: stopping_result
+  when: "matrix_client_element_service_stat.stat.exists|bool"
+
+- name: Ensure matrix-client-element.service doesn't exist
+  file:
+    path: "{{ matrix_systemd_path }}/matrix-client-element.service"
+    state: absent
+  when: "matrix_client_element_service_stat.stat.exists|bool"
+
+- name: Ensure systemd reloaded after matrix-client-element.service removal
+  service:
+    daemon_reload: yes
+  when: "matrix_client_element_service_stat.stat.exists|bool"
+
+- name: Ensure Element paths doesn't exist
+  file:
+    path: "{{ matrix_client_element_data_path }}"
+    state: absent
+
+- name: Ensure Element Docker image doesn't exist
+  docker_image:
+    name: "{{ matrix_client_element_docker_image }}"
+    state: absent

roles/matrix-client-hydrogen/defaults/main.yml

@@ -5,7 +5,7 @@ matrix_client_hydrogen_enabled: true
 matrix_client_hydrogen_container_image_self_build: true
 matrix_client_hydrogen_container_image_self_build_repo: "https://github.com/vector-im/hydrogen-web.git"
 
-matrix_client_hydrogen_version: v0.2.7
+matrix_client_hydrogen_version: v0.2.19
 matrix_client_hydrogen_docker_image: "{{ matrix_client_hydrogen_docker_image_name_prefix }}vectorim/hydrogen-web:{{ matrix_client_hydrogen_version }}"
 matrix_client_hydrogen_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_hydrogen_container_image_self_build }}"
 matrix_client_hydrogen_docker_image_force_pull: "{{ matrix_client_hydrogen_docker_image.endswith(':latest') }}"

roles/matrix-client-hydrogen/tasks/main.yml

@@ -8,8 +8,14 @@
     - setup-all
     - setup-client-hydrogen
 
-- import_tasks: "{{ role_path }}/tasks/setup.yml"
+- import_tasks: "{{ role_path }}/tasks/setup_install.yml"
-  when: run_setup|bool
+  when: "run_setup|bool and matrix_client_hydrogen_enabled|bool"
+  tags:
+    - setup-all
+    - setup-client-hydrogen
+
+- import_tasks: "{{ role_path }}/tasks/setup_uninstall.yml"
+  when: "run_setup|bool and not matrix_client_hydrogen_enabled|bool"
   tags:
     - setup-all
     - setup-client-hydrogen

roles/matrix-client-hydrogen/tasks/setup_install.yml

@@ -1,9 +1,5 @@
 ---
 
-#
-# Tasks related to setting up Hydrogen
-#
-
 - name: Ensure Hydrogen paths exists
   file:
     path: "{{ item.path }}"
@@ -14,7 +10,7 @@
   with_items:
     - { path: "{{ matrix_client_hydrogen_data_path }}", when: true }
     - { path: "{{ matrix_client_hydrogen_docker_src_files_path }}", when: "{{ matrix_client_hydrogen_container_image_self_build }}" }
-  when: matrix_client_hydrogen_enabled|bool and item.when
+  when: "item.when|bool"
 
 - name: Ensure Hydrogen Docker image is pulled
   docker_image:
@@ -22,7 +18,7 @@
    source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}"
    force_source: "{{ matrix_client_hydrogen_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}"
    force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_client_hydrogen_docker_image_force_pull }}"
-  when: matrix_client_hydrogen_enabled|bool and not matrix_client_hydrogen_container_image_self_build
+  when: "not matrix_client_hydrogen_container_image_self_build|bool"
 
 - name: Ensure Hydrogen repository is present on self-build
   git:
@@ -31,7 +27,7 @@
     version: "{{ matrix_client_hydrogen_docker_image.split(':')[1] }}"
     force: "yes"
   register: matrix_client_hydrogen_git_pull_results
-  when: "matrix_client_hydrogen_enabled|bool and matrix_client_hydrogen_container_image_self_build|bool"
+  when: "matrix_client_hydrogen_container_image_self_build|bool"
 
 - name: Ensure Hydrogen configuration installed
   copy:
@@ -40,7 +36,7 @@
     mode: 0644
     owner: "{{ matrix_user_username }}"
     group: "{{ matrix_user_groupname }}"
-  when: "matrix_client_hydrogen_enabled|bool and matrix_client_hydrogen_container_image_self_build|bool"
+  when: "matrix_client_hydrogen_container_image_self_build|bool"
 
 - name: Ensure Hydrogen additional config files installed
   template:
@@ -51,7 +47,7 @@
     group: "{{ matrix_user_groupname }}"
   with_items:
     - {src: "{{ role_path }}/templates/nginx.conf.j2", name: "nginx.conf"}
-  when: "matrix_client_hydrogen_enabled|bool and item.src is not none"
+  when: "item.src is not none"
 
 # This step MUST come after the steps to install the configuration files because the config files
 # are currently only read at build time, not at run time like most other components in the playbook
@@ -64,7 +60,7 @@
       dockerfile: Dockerfile
       path: "{{ matrix_client_hydrogen_docker_src_files_path }}"
       pull: yes
-  when: "matrix_client_hydrogen_enabled|bool and matrix_client_hydrogen_container_image_self_build|bool"
+  when: "matrix_client_hydrogen_container_image_self_build|bool"
 
 - name: Ensure matrix-client-hydrogen.service installed
   template:
@@ -72,50 +68,8 @@
     dest: "{{ matrix_systemd_path }}/matrix-client-hydrogen.service"
     mode: 0644
   register: matrix_client_hydrogen_systemd_service_result
-  when: matrix_client_hydrogen_enabled|bool
 
 - name: Ensure systemd reloaded after matrix-client-hydrogen.service installation
   service:
     daemon_reload: yes
-  when: "matrix_client_hydrogen_enabled and matrix_client_hydrogen_systemd_service_result.changed"
+  when: "matrix_client_hydrogen_systemd_service_result.changed|bool"
-
-#
-# Tasks related to getting rid of Hydrogen (if it was previously enabled)
-#
-
-- name: Check existence of matrix-client-hydrogen.service
-  stat:
-    path: "{{ matrix_systemd_path }}/matrix-client-hydrogen.service"
-  register: matrix_client_hydrogen_service_stat
-  when: "not matrix_client_hydrogen_enabled|bool"
-
-- name: Ensure matrix-client-hydrogen is stopped
-  service:
-    name: matrix-client-hydrogen
-    state: stopped
-    daemon_reload: yes
-  register: stopping_result
-  when: "not matrix_client_hydrogen_enabled|bool and matrix_client_hydrogen_service_stat.stat.exists"
-
-- name: Ensure matrix-client-hydrogen.service doesn't exist
-  file:
-    path: "{{ matrix_systemd_path }}/matrix-client-hydrogen.service"
-    state: absent
-  when: "not matrix_client_hydrogen_enabled|bool and matrix_client_hydrogen_service_stat.stat.exists"
-
-- name: Ensure systemd reloaded after matrix-client-hydrogen.service removal
-  service:
-    daemon_reload: yes
-  when: "not matrix_client_hydrogen_enabled|bool and matrix_client_hydrogen_service_stat.stat.exists"
-
-- name: Ensure Hydrogen paths doesn't exist
-  file:
-    path: "{{ matrix_client_hydrogen_data_path }}"
-    state: absent
-  when: "not matrix_client_hydrogen_enabled|bool"
-
-- name: Ensure Hydrogen Docker image doesn't exist
-  docker_image:
-    name: "{{ matrix_client_hydrogen_docker_image }}"
-    state: absent
-  when: "not matrix_client_hydrogen_enabled|bool"

roles/matrix-client-hydrogen/tasks/setup_uninstall.yml

@@ -0,0 +1,35 @@
+---
+
+- name: Check existence of matrix-client-hydrogen.service
+  stat:
+    path: "{{ matrix_systemd_path }}/matrix-client-hydrogen.service"
+  register: matrix_client_hydrogen_service_stat
+
+- name: Ensure matrix-client-hydrogen is stopped
+  service:
+    name: matrix-client-hydrogen
+    state: stopped
+    daemon_reload: yes
+  register: stopping_result
+  when: "matrix_client_hydrogen_service_stat.stat.exists|bool"
+
+- name: Ensure matrix-client-hydrogen.service doesn't exist
+  file:
+    path: "{{ matrix_systemd_path }}/matrix-client-hydrogen.service"
+    state: absent
+  when: "matrix_client_hydrogen_service_stat.stat.exists|bool"
+
+- name: Ensure systemd reloaded after matrix-client-hydrogen.service removal
+  service:
+    daemon_reload: yes
+  when: "matrix_client_hydrogen_service_stat.stat.exists|bool"
+
+- name: Ensure Hydrogen paths doesn't exist
+  file:
+    path: "{{ matrix_client_hydrogen_data_path }}"
+    state: absent
+
+- name: Ensure Hydrogen Docker image doesn't exist
+  docker_image:
+    name: "{{ matrix_client_hydrogen_docker_image }}"
+    state: absent

roles/matrix-grafana/defaults/main.yml

@@ -3,7 +3,7 @@
 
 matrix_grafana_enabled: false
 
-matrix_grafana_version: 8.1.4
+matrix_grafana_version: 8.2.2
 matrix_grafana_docker_image: "{{ matrix_container_global_registry_prefix }}grafana/grafana:{{ matrix_grafana_version }}"
 matrix_grafana_docker_image_force_pull: "{{ matrix_grafana_docker_image.endswith(':latest') }}"
 

roles/matrix-ma1sd/defaults/main.yml

@@ -19,7 +19,8 @@ matrix_ma1sd_docker_src_files_path: "{{ matrix_ma1sd_base_path }}/docker-src/ma1
 matrix_ma1sd_config_path: "{{ matrix_ma1sd_base_path }}/config"
 matrix_ma1sd_data_path: "{{ matrix_ma1sd_base_path }}/data"
 
-# Controls whether the matrix-ma1sd container exposes its HTTP port (tcp/8090 in the container).
+matrix_ma1sd_default_port: 8090
+# Controls whether the matrix-ma1sd container exposes its HTTP port (tcp/{{ matrix_ma1sd_default_port }} in the container).
 #
 # Takes an "<ip>:<port>" or "<port>" value (e.g. "127.0.0.1:8090"), or empty string to not expose.
 matrix_ma1sd_container_http_host_bind_port: ''

roles/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2

@@ -26,7 +26,7 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-ma1sd \
 			--tmpfs=/tmp:rw,exec,nosuid,size=10m \
 			--network={{ matrix_docker_network }} \
 			{% if matrix_ma1sd_container_http_host_bind_port %}
-			-p {{ matrix_ma1sd_container_http_host_bind_port }}:8090 \
+			-p {{ matrix_ma1sd_container_http_host_bind_port }}:{{ matrix_ma1sd_default_port }} \
 			{% endif %}
 			{% if matrix_ma1sd_verbose_logging %}
 			-e MA1SD_LOG_LEVEL=debug \

roles/matrix-nginx-proxy/defaults/main.yml

@@ -165,23 +165,24 @@ matrix_nginx_proxy_proxy_matrix_corporal_api_addr_sans_container: "127.0.0.1:410
 # This can be used to forward the API endpoint to another service, augmenting the functionality of Synapse's own User Directory Search.
 # To learn more, see: https://github.com/ma1uta/ma1sd/blob/master/docs/features/directory.md
 matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled: false
-matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_with_container: "matrix-ma1sd:8090"
+matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_with_container: "matrix-ma1sd:{{ matrix_ma1sd_default_port }}"
-matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_sans_container: "127.0.0.1:8090"
+matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_sans_container: "127.0.0.1:{{ matrix_ma1sd_default_port }}"
 
 # Controls whether proxying for 3PID-based registration (`/_matrix/client/r0/register/(email|msisdn)/requestToken`) should be done (on the matrix domain).
 # This allows another service to control registrations involving 3PIDs.
 # To learn more, see: https://github.com/ma1uta/ma1sd/blob/master/docs/features/registration.md
 matrix_nginx_proxy_proxy_matrix_3pid_registration_enabled: false
-matrix_nginx_proxy_proxy_matrix_3pid_registration_addr_with_container: "matrix-ma1sd:8090"
+matrix_nginx_proxy_proxy_matrix_3pid_registration_addr_with_container: "matrix-ma1sd:{{ matrix_ma1sd_default_port }}"
-matrix_nginx_proxy_proxy_matrix_3pid_registration_addr_sans_container: "127.0.0.1:8090"
+matrix_nginx_proxy_proxy_matrix_3pid_registration_addr_sans_container: "127.0.0.1:{{ matrix_ma1sd_default_port }}"
 
 # Controls whether proxying for the Identity API (`/_matrix/identity`) should be done (on the matrix domain)
 matrix_nginx_proxy_proxy_matrix_identity_api_enabled: false
-matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container: "matrix-ma1sd:8090"
+matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container: "matrix-ma1sd:{{ matrix_ma1sd_default_port }}"
-matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container: "127.0.0.1:8090"
+matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container: "127.0.0.1:{{ matrix_ma1sd_default_port }}"
 
 # Controls whether proxying for metrics (`/_synapse/metrics`) should be done (on the matrix domain)
 matrix_nginx_proxy_proxy_synapse_metrics: false
+matrix_nginx_proxy_synapse_workers_enabled_list: []
 matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled: false
 # The following value will be written verbatim to the htpasswd file that stores the password for nginx to check against and needs to be encoded appropriately.
 # Read the manpage at `man 1 htpasswd` to learn more, then encrypt your password, and paste the encrypted value here.
@@ -226,7 +227,7 @@ matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_prefix_regexes: |
     +
     (['/_synapse/admin'] if matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_synapse_admin_api_enabled else [])
     +
-    (['/_synapse/metrics'] if matrix_nginx_proxy_proxy_synapse_metrics else [])
+    (['/_synapse.*/metrics'] if matrix_nginx_proxy_proxy_synapse_metrics else [])
   }}
 
 # Specifies where requests for the root URI (`/`) on the `matrix.` domain should be redirected.

roles/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml

@@ -38,6 +38,15 @@
     mode: 0400
   when: "matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled|bool and matrix_nginx_proxy_proxy_synapse_metrics|bool"
 
+- name: Generate sample prometheus.yml for external scraping
+  template:
+    src: "{{ role_path }}/templates/prometheus/external_prometheus.yml.example.j2"
+    dest: "{{ matrix_base_data_path }}/external_prometheus.yml.example"
+    owner: "{{ matrix_user_username }}"
+    group: "{{ matrix_user_groupname }}"
+    mode: 0644
+  when: matrix_nginx_proxy_proxy_synapse_metrics|bool
+
 - name: Ensure Matrix nginx-proxy configured (generic)
   template:
     src: "{{ role_path }}/templates/nginx/conf.d/nginx-http.conf.j2"
@@ -270,3 +279,9 @@
     path: "{{ matrix_nginx_proxy_data_path }}/matrix-synapse-metrics-htpasswd"
     state: absent
   when: "not matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled|bool or not matrix_nginx_proxy_proxy_synapse_metrics|bool"
+
+- name: Ensure sample prometheus.yml for external scraping is deleted
+  file:
+    path: "{{ matrix_base_data_path }}/external_prometheus.yml.example"
+    state: absent
+  when: "not matrix_nginx_proxy_proxy_synapse_metrics|bool"

roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2

@@ -62,7 +62,9 @@
 
 server {
 	listen 12080;
+	{% if matrix_nginx_proxy_enabled %}
 		server_name {{ matrix_nginx_proxy_proxy_synapse_hostname }};
+	{% endif %}
 
 	server_tokens off;
 	root /dev/null;
@@ -153,6 +155,24 @@ server {
 	}
 	{% endif %}
 
+	{% if matrix_nginx_proxy_enabled and matrix_nginx_proxy_proxy_synapse_metrics %}
+		{% for worker in matrix_nginx_proxy_proxy_synapse_workers_enabled_list %}
+			{% if worker.metrics_port != 0 %}
+				location /_synapse-worker-{{ worker.type }}-{{ worker.instanceId }}/metrics {
+					resolver 127.0.0.11 valid=5s;
+					set $backend "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.metrics_port }}";
+					proxy_pass http://$backend/_synapse/metrics;
+					proxy_set_header Host $host;
+
+					{% if matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled %}
+						auth_basic "protected";
+						auth_basic_user_file /nginx-data/matrix-synapse-metrics-htpasswd;
+					{% endif %}
+				}
+			{% endif %}
+		{% endfor %}
+	{% endif %}
+
 	{# Everything else just goes to the API server ##}
 	location / {
 		{% if matrix_nginx_proxy_enabled %}
@@ -176,8 +196,10 @@ server {
 {% if matrix_nginx_proxy_proxy_synapse_federation_api_enabled %}
 server {
 	listen 12088;
-
+	{% if matrix_nginx_proxy_enabled %}
 		server_name {{ matrix_nginx_proxy_proxy_synapse_hostname }};
+	{% endif %}
+
 	server_tokens off;
 
 	root /dev/null;

roles/matrix-nginx-proxy/templates/prometheus/external_prometheus.yml.example.j2

@@ -0,0 +1,40 @@
+global:
+  scrape_interval: 5s
+
+  # Attach these labels to any time series or alerts when communicating with
+  # external systems (federation, remote storage, Alertmanager).
+  external_labels:
+    monitor: 'synapse-{{ matrix_domain }}'
+
+rule_files:
+  - /etc/prometheus/synapse-v2.rules
+
+scrape_configs:
+  - job_name: 'synapse'
+    metrics_path: /_synapse/metrics
+    scheme: {{ 'https' if matrix_nginx_proxy_https_enabled else 'http' }}
+{% if matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled %}
+    basic_auth:
+      username: prometheus
+      password_file: /path/to/your/passwordfile.pwd
+{% endif %}
+    static_configs:
+      - targets: ['{{ matrix_server_fqn_matrix }}:{{ matrix_nginx_proxy_container_https_host_bind_port if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_http_host_bind_port }}']
+        labels:
+          job: "master"
+          index: 1
+{% for worker in matrix_nginx_proxy_proxy_synapse_workers_enabled_list %}
+  - job_name: 'synapse-{{ worker.type }}-{{ worker.instanceId }}'
+    metrics_path: /_synapse-worker-{{ worker.type }}-{{ worker.instanceId }}/metrics
+    scheme: {{ 'https' if matrix_nginx_proxy_https_enabled else 'http' }}
+{% if matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled %}
+    basic_auth:
+      username: prometheus
+      password_file: /path/to/your/passwordfile.pwd
+{% endif %}
+    static_configs:
+      - targets: ['{{ matrix_server_fqn_matrix }}:{{ matrix_nginx_proxy_container_https_host_bind_port if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_http_host_bind_port }}']
+        labels:
+          job: "{{ worker.type }}"
+          index: {{ worker.instanceId }}
+{% endfor %}

roles/matrix-postgres/templates/systemd/matrix-postgres.service.j2

@@ -11,6 +11,10 @@ Environment="HOME={{ matrix_systemd_unit_home_path }}"
 ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-postgres 2>/dev/null'
 ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-postgres 2>/dev/null'
 
+# We need /dev/shm to be larger than the default to allow VACUUM to work.
+# See:
+# - https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1362
+# - https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1268
 ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-postgres \
 			--log-driver=none \
 			--user={{ matrix_user_uid }}:{{ matrix_user_gid }} \
@@ -18,6 +22,7 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-postgres \
 			--read-only \
 			--tmpfs=/tmp:rw,noexec,nosuid,size=100m \
 			--tmpfs=/run/postgresql:rw,noexec,nosuid,size=100m \
+			--shm-size=256m \
 			--network={{ matrix_docker_network }} \
 			{% if matrix_postgres_container_postgres_bind_port %}
 			-p {{ matrix_postgres_container_postgres_bind_port }}:5432 \

roles/matrix-prometheus/defaults/main.yml

@@ -3,7 +3,7 @@
 
 matrix_prometheus_enabled: false
 
-matrix_prometheus_version: v2.29.2
+matrix_prometheus_version: v2.30.3
 matrix_prometheus_docker_image: "{{ matrix_container_global_registry_prefix }}prom/prometheus:{{ matrix_prometheus_version }}"
 matrix_prometheus_docker_image_force_pull: "{{ matrix_prometheus_docker_image.endswith(':latest') }}"
 

roles/matrix-synapse/defaults/main.yml

@@ -15,8 +15,8 @@ matrix_synapse_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_cont
 # amd64 gets released first.
 # arm32 relies on self-building, so the same version can be built immediately.
 # arm64 users need to wait for a prebuilt image to become available.
-matrix_synapse_version: v1.45.1
+matrix_synapse_version: v1.46.0
-matrix_synapse_version_arm64: v1.45.1
+matrix_synapse_version_arm64: v1.46.0
 matrix_synapse_docker_image_tag: "{{ matrix_synapse_version if matrix_architecture in ['arm32', 'amd64'] else matrix_synapse_version_arm64 }}"
 matrix_synapse_docker_image_force_pull: "{{ matrix_synapse_docker_image.endswith(':latest') }}"
 

roles/matrix-synapse/templates/synapse/homeserver.yaml.j2

@@ -492,6 +492,48 @@ redaction_retention_period: {{ matrix_synapse_redaction_retention_period }}
 
 user_ips_max_age: {{ matrix_synapse_user_ips_max_age }}
 
+# Inhibits the /requestToken endpoints from returning an error that might leak
+# information about whether an e-mail address is in use or not on this
+# homeserver.
+# Note that for some endpoints the error situation is the e-mail already being
+# used, and for others the error is entering the e-mail being unused.
+# If this option is enabled, instead of returning an error, these endpoints will
+# act as if no error happened and return a fake session ID ('sid') to clients.
+#
+#request_token_inhibit_3pid_errors: true
+
+# A list of domains that the domain portion of 'next_link' parameters
+# must match.
+#
+# This parameter is optionally provided by clients while requesting
+# validation of an email or phone number, and maps to a link that
+# users will be automatically redirected to after validation
+# succeeds. Clients can make use this parameter to aid the validation
+# process.
+#
+# The whitelist is applied whether the homeserver or an
+# identity server is handling validation.
+#
+# The default value is no whitelist functionality; all domains are
+# allowed. Setting this value to an empty list will instead disallow
+# all domains.
+#
+#next_link_domain_whitelist: ["matrix.org"]
+
+# Templates to use when generating email or HTML page contents.
+#
+templates:
+  # Directory in which Synapse will try to find template files to use to generate
+  # email or HTML page contents.
+  # If not set, or a file is not found within the template directory, a default
+  # template from within the Synapse package will be used.
+  #
+  # See https://matrix-org.github.io/synapse/latest/templates.html for more
+  # information about using custom templates.
+  #
+  #custom_template_directory: /path/to/custom/templates/
+
+
 # Message retention policy at the server level.
 #
 # Room admins and mods can define a retention period for their rooms using the
@@ -561,47 +603,6 @@ retention:
   #  - shortest_max_lifetime: 3d
   #    interval: 1d
 
-# Inhibits the /requestToken endpoints from returning an error that might leak
-# information about whether an e-mail address is in use or not on this
-# homeserver.
-# Note that for some endpoints the error situation is the e-mail already being
-# used, and for others the error is entering the e-mail being unused.
-# If this option is enabled, instead of returning an error, these endpoints will
-# act as if no error happened and return a fake session ID ('sid') to clients.
-#
-#request_token_inhibit_3pid_errors: true
-
-# A list of domains that the domain portion of 'next_link' parameters
-# must match.
-#
-# This parameter is optionally provided by clients while requesting
-# validation of an email or phone number, and maps to a link that
-# users will be automatically redirected to after validation
-# succeeds. Clients can make use this parameter to aid the validation
-# process.
-#
-# The whitelist is applied whether the homeserver or an
-# identity server is handling validation.
-#
-# The default value is no whitelist functionality; all domains are
-# allowed. Setting this value to an empty list will instead disallow
-# all domains.
-#
-#next_link_domain_whitelist: ["matrix.org"]
-
-# Templates to use when generating email or HTML page contents.
-#
-templates:
-  # Directory in which Synapse will try to find template files to use to generate
-  # email or HTML page contents.
-  # If not set, or a file is not found within the template directory, a default
-  # template from within the Synapse package will be used.
-  #
-  # See https://matrix-org.github.io/synapse/latest/templates.html for more
-  # information about using custom templates.
-  #
-  #custom_template_directory: /path/to/custom/templates/
-
 
 ## TLS ##