(BC Break) Redo how metrics are exposed to external Prometheus servers

CHANGELOG.md

@@ -1,3 +1,32 @@
+# 2022-06-23
+
+## (Potential Backward Compatibility Break) Changes around metrics collection
+
+**TLDR**: we've made extensive **changes to metrics exposure/collection, which concern people using an external Prometheus server**. If you don't know what that is, you don't need to read below.
+
+**Why do major changes to metrics**? Because various services were exposing metrics in different, hacky, ways. Synapse was exposing metrics at `/_synapse/metrics` and `/_synapse-worker-.../metrics` on the `matrix.DOMAIN`. The Hookshot role was **repurposing** the Granana web UI domain (`stats.DOMAIN`) for exposing its metrics on `stats.DOMAIN/hookshot/metrics`, while protecting these routes using Basic Authentication **normally used for Synapse** (`/_synapse/metrics`). Node-exporter and Postgres-exporter roles were advising for more `stats.DOMAIN` usage in manual ways. Each role was doing things differently and mixing variables from other roles. Each metrics endpoint was ending up in a different place, protected by who knows what Basic Authentication credentials (if protected at all).
+
+**The solution**: a completely revamped way to expose metrics to an external Prometheus server. We are **introducing new `https://matrix.DOMAIN/metrics/*` endpoints**, where various services *can* expose their metrics, for collection by external Prometheus servers. To enable the `/metrics/*` endpoints, use `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. There's also a way to protect access using [Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication). See the `matrix-nginx-proxy` role or our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation for additional variables around `matrix_nginx_proxy_proxy_matrix_metrics_enabled`.
+
+**If you are using the [Hookshot bridge](docs/configuring-playbook-bridge-hookshot.md)**, you may find that:
+1. **Metrics may not be enabled by default anymore**:
+  - If Prometheus is enabled (`matrix_prometheus_enabled: true`), then Hookshot metrics will be enabled automatically (`matrix_hookshot_metrics_enabled: true`). These metrics will be collected from the local (in-container) Prometheus over the container network.
+  - **If Prometheus is not enabled** (you are either not using Prometheus or are using an external one), **Hookshot metrics will not be enabled by default anymore**. Feel free to enable them by setting `matrix_hookshot_metrics_enabled: true`. Also, see below.
+2. When metrics are meant to be **consumed by an external Prometheus server**, `matrix_hookshot_metrics_proxying_enabled` needs to be set to `true`, so that metrics would be exposed (proxied) "publicly" on `https://matrix.DOMAIN/metrics/hookshot`. To make use of this, you'll also need to enable the new `https://matrix.DOMAIN/metrics/*` endpoints mentioned above, using `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. Learn more in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation.
+3. **We've changed the URL we're exposing Hookshot metrics at** for external Prometheus servers. Until now, you were advised to consume Hookshot metrics from `https://stats.DOMAIN/hookshot/metrics` (working in conjunction with `matrix_nginx_proxy_proxy_synapse_metrics`). From now on, **this no longer works**. As described above, you need to start consuming metrics from `https://matrix.DOMAIN/metrics/hookshot`.
+
+**If you're using node-exporter** (`matrix_prometheus_node_exporter_enabled: true`) and would like to collect its metrics from an external Prometheus server, see `matrix_prometheus_node_exporter_metrics_proxying_enabled` described in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation. You will be able to collect its metrics from `https://matrix.DOMAIN/metrics/node-exporter`.
+
+**If you're using [postgres-exporter](docs/configuring-playbook-prometheus-postgres.md)** (`matrix_prometheus_postgres_exporter_enabled: true`) and would like to collect its metrics from an external Prometheus server, see `matrix_prometheus_postgres_exporter_metrics_proxying_enabled` described in our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation. You will be able to collect its metrics from `https://matrix.DOMAIN/metrics/postgres-exporter`.
+
+**If you're using Synapse** and would like to collect its metrics from an external Prometheus server, you may find that:
+
+1. Exposing metrics is now done using `matrix_synapse_metrics_proxying_enabled`, not `matrix_nginx_proxy_proxy_synapse_metrics: true`. You may still need to enable metrics using `matrix_synapse_metrics_enabled: true` before exposing them.
+2. Protecting metrics endpoints using [Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) is now done in another way. See our [Collecting metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) documentation
+3. If Synapse metrics are exposed, they will be made available at `https://matrix.DOMAIN/metrics/synapse/main-process` or `https://matrix.DOMAIN/metrics/synapse/worker/TYPE-ID` (when workers are enabled), not at `https://matrix.DOMAIN/_synapse/metrics` and `https://matrix.DOMAIN/_synapse-worker-.../metrics`
+4. The playbook still generates an `external_prometheus.yml.example` sample file for scraping Synapse from Prometheus as described in [Collecting Synapse worker metrics to an external Prometheus server](docs/configuring-playbook-prometheus-grafana.md#collecting-synapse-worker-metrics-to-an-external-prometheus-server), but it's now saved under `/matrix/synapse` (not `/matrix`).
+
+
 # 2022-06-13
 
 ## go-skype-bridge bridging support

docs/configuring-playbook-bridge-hookshot.md

@@ -32,8 +32,8 @@ Unless indicated otherwise, the following endpoints are reachable on your `matri
 | figma endpoint | `/hookshot/webhooks/figma/webhook` | `matrix_hookshot_figma_endpoint` | Figma |
 | provisioning | `/hookshot/v1/` | `matrix_hookshot_provisioning_endpoint` | Dimension [provisioning](#provisioning-api) |
 | appservice | `/hookshot/_matrix/app/` | `matrix_hookshot_appservice_endpoint` | Matrix server |
-| widgets | `/hookshot/widgetapi/` | `/matrix_hookshot_widgets_endpoint` | Widgets |
+| widgets | `/hookshot/widgetapi/` | `matrix_hookshot_widgets_endpoint` | Widgets |
-| metrics | `/hookshot/metrics/` (on `stats.` subdomain) | `matrix_hookshot_metrics_endpoint` | Prometheus |
+| metrics | `/metrics/hookshot` | `matrix_hookshot_metrics_enabled` and `matrix_hookshot_metrics_proxying_enabled`. Requires `/metrics/*` endpoints to also be enabled via `matrix_nginx_proxy_proxy_matrix_metrics_enabled` (see the `matrix-nginx-proxy` role). Read more in the [Metrics section](#metrics) below. | Prometheus |
 
 See also `matrix_hookshot_matrix_nginx_proxy_configuration` in [init.yml](/roles/matrix-bridge-hookshot/tasks/init.yml).
 
@@ -63,7 +63,14 @@ The provisioning API will be enabled automatically if you set `matrix_dimension_
 
 ### Metrics
 
-If metrics are enabled, they will be automatically available in the builtin Prometheus and Grafana, but you need to set up your own Dashboard for now. If additionally metrics proxying for use with external Prometheus is enabled (`matrix_nginx_proxy_proxy_synapse_metrics`), hookshot metrics will also be available (at `matrix_hookshot_metrics_endpoint`, default `/hookshot/metrics`, on the stats subdomain) and with the same password. See also [the Prometheus and Grafana docs](../configuring-playbook-prometheus-grafana.md).
+Metrics are **only enabled by default** if the builtin [Prometheus](configuring-playbook-prometheus-grafana.md) is enabled (by default, Prometheus isn't enabled). If so, metrics will automatically be collected by Prometheus and made available in Grafana. You will, however, need to set up your own Dashboard for displaying them.
+
+To explicitly enable metrics, use `matrix_hookshot_metrics_enabled: true`. This only exposes metrics over the container network, however.
+
+**To collect metrics from an external Prometheus server**, besides enabling metrics as described above, you will also need to:
+
+- enable the `https://matrix.DOMAIN/metrics/*` endpoints on `matrix.DOMAIN` using `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true` (see the `matrix-nginx-role` or [the Prometheus and Grafana docs](configuring-playbook-prometheus-grafana.md) for enabling this feature)
+- expose the Hookshot metrics under `https://matrix.DOMAIN/metrics/hookshot` by setting `matrix_hookshot_metrics_proxying_enabled: true`
 
 ### Collision with matrix-appservice-webhooks
 

docs/configuring-playbook-prometheus-grafana.md

@@ -9,8 +9,12 @@ Remember to add `stats.<your-domain>` to DNS as described in [Configuring DNS](c
 ```yaml
 matrix_prometheus_enabled: true
 
+# You can remove this, if unnecessary.
 matrix_prometheus_node_exporter_enabled: true
 
+# You can remove this, if unnecessary.
+matrix_prometheus_postgres_exporter_enabled: true
+
 matrix_grafana_enabled: true
 
 matrix_grafana_anonymous_access: false
@@ -34,6 +38,7 @@ Name | Description
 -----|----------
 `matrix_prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about.
 `matrix_prometheus_node_exporter_enabled`|[Node Exporter](https://prometheus.io/docs/guides/node-exporter/) is an addon of sorts to Prometheus that collects generic system information such as CPU, memory, filesystem, and even system temperatures
+`matrix_prometheus_postgres_exporter_enabled`|[Postgres Exporter](configuring-playbook-prometheus-postgres.md) is an addon of sorts to expose Postgres database metrics to Prometheus.
 `matrix_grafana_enabled`|[Grafana](https://grafana.com/) is the visual component. It shows (on the `stats.<your-domain>` subdomain) the dashboards with the graphs that we're interested in
 `matrix_grafana_anonymous_access`|By default you need to log in to see graphs. If you want to publicly share your graphs (e.g. when asking for help in [`#synapse:matrix.org`](https://matrix.to/#/#synapse:matrix.org?via=matrix.org&via=privacytools.io&via=mozilla.org)) you'll want to enable this option.
 `matrix_grafana_default_admin_user`<br>`matrix_grafana_default_admin_password`|By default Grafana creates a user with `admin` as the username and password. If you feel this is insecure and you want to change it beforehand, you can do that here
@@ -48,28 +53,54 @@ Most of our docker containers run with limited system access, but the `prometheu
 
 ## Collecting metrics to an external Prometheus server
 
-If you wish, you could expose homeserver metrics without enabling (installing) Prometheus and Grafana via the playbook. This may be useful for hooking Matrix services to an external Prometheus/Grafana installation.
+**If the integrated Prometheus server is enabled** (`matrix_prometheus_enabled: true`), metrics are collected by it from each service via communication that happens over the container network. Each service does not need to expose its metrics "publicly".
 
-To do this, you may be interested in the following variables:
+When you'd like **to collect metrics from an external Prometheus server**, you need to expose service metrics outside of the container network.
+
+The playbook provides a single endpoint (`https://matrix.DOMAIN/metrics/*`), under which various services may expose their metrics (e.g. `/metrics/node-exporter`, `/metrics/postgres-exporter`, `/metrics/hookshot`, etc). To enable this `/metrics/*` feature, use `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. To protect access using [Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication), see `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled` below.
+
+The following variables may be of interest:
 
 Name | Description
 -----|----------
+`matrix_nginx_proxy_proxy_matrix_metrics_enabled`|Set this to `true` to enable metrics exposure for various services on `https://matrix.DOMAIN/metrics/*`. Refer to the individual `matrix_SERVICE_metrics_proxying_enabled` variables below for exposing metrics for each individual service.
+`matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled`|Set this to `true` to protect all `https://matrix.DOMAIN/metrics/*` endpoints with [Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication) (see the other variables below for supplying the actual credentials). When enabled, all endpoints beneath `/metrics` will be protected with the same credentials
+`matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username`|Set this to the Basic Authentication username you'd like to protect `/metrics/*` with. You also need to set `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password`. If one username/password pair is not enough, you can leave the `username` and `password` variables unset and use `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content` instead
+`matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password`|Set this to the Basic Authentication password you'd like to protect `/metrics/*` with
+`matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content`|Set this to the Basic Authentication credentials (raw `htpasswd` file content) used to protect `/metrics/*`. This htpasswd-file needs to be generated with the `htpasswd` tool and can include multiple username/password pairs. If you only need one credential, use `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username` and `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password` instead.
 `matrix_synapse_metrics_enabled`|Set this to `true` to make Synapse expose metrics (locally, on the container network)
-`matrix_nginx_proxy_proxy_synapse_metrics`|Set this to `true` to make matrix-nginx-proxy expose the Synapse metrics at `https://matrix.DOMAIN/_synapse/metrics`
+`matrix_synapse_metrics_proxying_enabled`|Set this to `true` to expose Synapse's metrics on `https://matrix.DOMAIN/metrics/synapse/main-process` and `https://matrix.DOMAIN/metrics/synapse/worker/TYPE-ID` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`). Read [below](#collecting-synapse-worker-metrics-to-an-external-prometheus-server) if you're running a Synapse worker setup (`matrix_synapse_workers_enabled: true`).
-`matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled`|Set this to `true` to password-protect (using HTTP Basic Auth) `https://matrix.DOMAIN/_synapse/metrics` (the username is always `prometheus`, the password is defined in `matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key`)
+`matrix_prometheus_node_exporter_enabled`|Set this to `true` to enable the node (general system stats) exporter (locally, on the container network)
-`matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key`|Set this to a password to use for HTTP Basic Auth for protecting `https://matrix.DOMAIN/_synapse/metrics` (the username is always `prometheus` - it's not configurable). Do not write the password in plain text. See `man 1 htpasswd` or use `htpasswd -c mypass.htpasswd prometheus` to generate the expected hash for nginx. 
+`matrix_prometheus_node_exporter_metrics_proxying_enabled`|Set this to `true` to expose the node (general system stats) metrics on `https://matrix.DOMAIN/metrics/node-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`)
-`matrix_server_fqn_grafana`|Use this variable to override the domain at which the Grafana web user-interface is at (defaults to `stats.DOMAIN`)
+`matrix_prometheus_postgres_exporter_enabled`|Set this to `true` to enable the [Postgres exporter](configuring-playbook-prometheus-postgres.md) (locally, on the container network)
+`matrix_prometheus_postgres_exporter_metrics_proxying_enabled`|Set this to `true` to expose the [Postgres exporter](configuring-playbook-prometheus-postgres.md) metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`)
+`matrix_bridge_hookshot_metrics_enabled`|Set this to `true` to make [Hookshot](configuring-playbook-bridge-hookshot.md) expose metrics (locally, on the container network)
+`matrix_bridge_hookshot_metrics_proxying_enabled`|Set this to `true` to expose the [Hookshot](configuring-playbook-bridge-hookshot.md) metrics on `https://matrix.DOMAIN/metrics/hookshot` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`)
+`matrix_SERVICE_metrics_proxying_enabled`|Various other services/roles may provide similar `_metrics_enabled` and `_metrics_proxying_enabled` variables for exposing their metrics. Refer to each role for details. Only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`
+`matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks`|Add nginx `location` blocks to this list if you'd like to expose additional exporters manually (see below)
 
-### Collecting worker metrics to an external Prometheus server
+Example for how to make use of `matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks` for exposing additional metrics locations:
+```nginx
+matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks:
+  - 'location /metrics/another-service {
+  resolver 127.0.0.11 valid=5s;
+  proxy_pass http://matrix-another-service:9100/metrics;
+  }'
+```
 
-If you are using workers (`matrix_synapse_workers_enabled`) and have enabled `matrix_nginx_proxy_proxy_synapse_metrics` as described above, the playbook will also automatically proxy the all worker threads's metrics to `https://matrix.DOMAIN/_synapse-worker-TYPE-ID/metrics`, where `TYPE` corresponds to the type and `ID` to the instanceId of a worker as exemplified in `matrix_synapse_workers_enabled_list`.
+Using `matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks` only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true` (see above).
 
-The playbook also generates an exemplary prometheus.yml config file (`matrix_base_data_path/external_prometheus.yml.template`) with all the correct paths which you can copy to your Prometheus server and adapt to your needs, especially edit the specified `password_file` path and contents and path to your `synapse-v2.rules`.
+
+### Collecting Synapse worker metrics to an external Prometheus server
+
+If you are using workers (`matrix_synapse_workers_enabled: true`) and have enabled `matrix_synapse_metrics_proxying_enabled` as described above, the playbook will also automatically expose all Synapse worker threads' metrics to `https://matrix.DOMAIN/metrics/synapse/worker/TYPE-ID`, where `TYPE` corresponds to the type and `ID` to the instanceId of a worker as exemplified in `matrix_synapse_workers_enabled_list`.
+
+The playbook also generates an exemplary config file (`/matrix/synapse/external_prometheus.yml.template`) with all the correct paths which you can copy to your Prometheus server and adapt to your needs. Make sure to edit the specified `password_file` path and contents and path to your `synapse-v2.rules`.
 It will look a bit like this:
 ```yaml
 scrape_configs:
   - job_name: 'synapse'
-    metrics_path: /_synapse/metrics
+    metrics_path: /metrics/synapse/main-process
     scheme: https
     basic_auth:
       username: prometheus
@@ -80,7 +111,7 @@ scrape_configs:
           job: "master"
           index: 1
   - job_name: 'synapse-generic_worker-1'
-    metrics_path: /_synapse-worker-generic_worker-18111/metrics
+    metrics_path: /metrics/synapse/worker/generic_worker-18111
     scheme: https
     basic_auth:
       username: prometheus
@@ -92,38 +123,6 @@ scrape_configs:
           index: 18111
 ```
 
-### Collecting system and Postgres metrics to an external Prometheus server (advanced)
-
-When you normally enable the Prometheus and Grafana via the playbook, it will also show general system (via node-exporter) and Postgres (via postgres-exporter) stats. If you are instead collecting your metrics to an external Prometheus server, you can follow this advanced configuration example to also export these stats.
-
-It would be possible to use `matrix_prometheus_node_exporter_container_http_host_bind_port` etc., but that is not always the best choice, for example because your server is on a public network.
-
-Use the following variables in addition to the ones mentioned above:
-
-Name | Description
------|----------
-`matrix_nginx_proxy_proxy_grafana_enabled`|Set this to `true` to make the stats subdomain (`matrix_server_fqn_grafana`) available via the Nginx proxy
-`matrix_ssl_additional_domains_to_obtain_certificates_for`|Add `"{{ matrix_server_fqn_grafana }}"` to this list to have letsencrypt fetch a certificate for the stats subdomain
-`matrix_prometheus_node_exporter_enabled`|Set this to `true` to enable the node (general system stats) exporter
-`matrix_prometheus_postgres_exporter_enabled`|Set this to `true` to enable the Postgres exporter
-`matrix_nginx_proxy_proxy_grafana_additional_server_configuration_blocks`|Add locations to this list depending on which of the above exporters you enabled (see below)
-
-```nginx
-matrix_nginx_proxy_proxy_grafana_additional_server_configuration_blocks:
-  - 'location /node-exporter/ {
-  resolver 127.0.0.11 valid=5s;
-  proxy_pass http://matrix-prometheus-node-exporter:9100/;
-  auth_basic "protected";
-  auth_basic_user_file /nginx-data/matrix-synapse-metrics-htpasswd;
-  }'
-  - 'location /postgres-exporter/ {
-  resolver 127.0.0.11 valid=5s;
-  proxy_pass http://matrix-prometheus-postgres-exporter:9187/;
-  auth_basic "protected";
-  auth_basic_user_file /nginx-data/matrix-synapse-metrics-htpasswd;
-  }'
-```
-You can customize the `location`s to your liking, just point your Prometheus to there later (e.g. `stats.DOMAIN/node-exporter/metrics`). Nginx is very picky about the `proxy_pass`syntax: take care to follow the example closely and note the trailing slash as well as absent use of variables. postgres-exporter uses the nonstandard port 9187.
 
 ## More information
 

docs/configuring-playbook-prometheus-postgres.md

@@ -7,11 +7,6 @@ You can enable this with the following settings in your configuration file (`inv
 
 ```yaml
 matrix_prometheus_postgres_exporter_enabled: true
-
-# the role creates a postgres user as credential. You can configure these if required:
-matrix_prometheus_postgres_exporter_database_username: 'matrix_prometheus_postgres_exporter'
-matrix_prometheus_postgres_exporter_database_password: 'some-password'
-
 ```
 
 ## What does it do?
@@ -20,7 +15,8 @@ Name | Description
 -----|----------
 `matrix_prometheus_postgres_exporter_enabled`|Enable the postgres prometheus exporter. This sets up the docker container, connects it to the database and adds a 'job' to the prometheus config which tells prometheus about this new exporter. The default is 'false'
 `matrix_prometheus_postgres_exporter_database_username`| The 'username' for the user that the exporter uses to connect to the database. The default is 'matrix_prometheus_postgres_exporter'
-`matrix_prometheus_postgres_exporter_database_password`| The 'password' for the user that the exporter uses to connect to the database.
+`matrix_prometheus_postgres_exporter_database_password`| The 'password' for the user that the exporter uses to connect to the database. By default, this is auto-generated by the playbook
+`matrix_prometheus_postgres_exporter_metrics_proxying_enabled`|If set to `true`, exposes the Postgres exporter metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` for usage with an [external Prometheus server](configuring-playbook-prometheus-grafana.md#collecting-metrics-to-an-external-prometheus-server) (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`)
 
 
 ## More information

group_vars/matrix_servers

@@ -711,8 +711,13 @@ matrix_hookshot_container_http_host_bind_ports: "{{ [] if matrix_nginx_proxy_ena
 
 matrix_hookshot_provisioning_enabled: "{{ matrix_hookshot_provisioning_secret and matrix_dimension_enabled }}"
 
-matrix_hookshot_proxy_metrics: "{{ matrix_nginx_proxy_proxy_synapse_metrics }}"
+# We only enable metrics (locally, in the container network) for the bridge if Prometheus is enabled.
-matrix_hookshot_proxy_metrics_basic_auth_enabled: "{{ matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled }}"
+#
+# People using an external Prometheus server will need to toggle all of these to be able to consume metrics remotely:
+# - `matrix_hookshot_metrics_enabled`
+# - `matrix_hookshot_metrics_proxying_enabled`
+# - `matrix_nginx_proxy_proxy_matrix_metrics_enabled`
+matrix_hookshot_metrics_enabled: "{{ matrix_prometheus_enabled }}"
 
 matrix_hookshot_urlprefix_port_enabled: "{{ matrix_nginx_proxy_container_https_host_bind_port == 443 if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_https_host_bind_port == 80 }}"
 matrix_hookshot_urlprefix_port: ":{{ matrix_nginx_proxy_container_https_host_bind_port if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_http_host_bind_port }}"
@@ -1586,13 +1591,6 @@ matrix_nginx_proxy_proxy_matrix_federation_port: "{{ matrix_federation_public_po
 
 matrix_nginx_proxy_container_federation_host_bind_port: "{{ matrix_federation_public_port }}"
 
-# This used to be hooked to `matrix_synapse_metrics_enabled`, but we don't do it anymore.
-# The fact that someone wishes to enable Synapse metrics does not necessarily mean they want to make them public.
-# A local Prometheus can consume them over the container network.
-matrix_nginx_proxy_proxy_synapse_metrics: false
-matrix_nginx_proxy_proxy_synapse_metrics_addr_with_container: "matrix-synapse:{{ matrix_synapse_metrics_port }}"
-matrix_nginx_proxy_proxy_synapse_metrics_addr_sans_container: "127.0.0.1:{{ matrix_synapse_metrics_port }}"
-
 matrix_nginx_proxy_proxy_matrix_user_directory_search_enabled: "{{ matrix_ma1sd_enabled }}"
 matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_with_container: "{{ matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container }}"
 matrix_nginx_proxy_proxy_matrix_user_directory_search_addr_sans_container: "{{ matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container }}"
@@ -1614,8 +1612,6 @@ matrix_nginx_proxy_synapse_media_repository_locations: "{{matrix_synapse_workers
 matrix_nginx_proxy_synapse_user_dir_locations: "{{ matrix_synapse_workers_user_dir_endpoints|default([]) }}"
 matrix_nginx_proxy_synapse_frontend_proxy_locations: "{{ matrix_synapse_workers_frontend_proxy_endpoints|default([]) }}"
 
-matrix_nginx_proxy_proxy_synapse_workers_enabled_list: "{{ matrix_synapse_workers_enabled_list }}"
-
 matrix_nginx_proxy_systemd_wanted_services_list: |
   {{
     ['matrix-' + matrix_homeserver_implementation + '.service']

roles/matrix-bridge-hookshot/defaults/main.yml

@@ -29,13 +29,20 @@ matrix_hookshot_public_endpoint: /hookshot
 matrix_hookshot_appservice_port: 9993
 matrix_hookshot_appservice_endpoint: "{{ matrix_hookshot_public_endpoint }}/_matrix/app"
 
-# Metrics work only in conjunction with matrix_synapse_metrics_enabled etc
+# Controls whether metrics are enabled in the bridge configuration.
-matrix_hookshot_metrics_enabled: true
+# Enabling them is usually enough for a local (in-container) Prometheus to consume them.
+# If metrics need to be consumed by another (external) Prometheus server, consider exposing them via `matrix_hookshot_metrics_proxying_enabled`.
+matrix_hookshot_metrics_enabled: false
+
+# Controls whether Hookshot metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/hookshot`.
+# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`.
+# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`.
+matrix_hookshot_metrics_proxying_enabled: false
+
 # There is no need to edit ports.
 # Read the documentation to learn about using hookshot metrics with external Prometheus
 # If you still want something different, use matrix_hookshot_container_http_host_bind_ports below to expose ports instead.
 matrix_hookshot_metrics_port: 9001
-matrix_hookshot_metrics_endpoint: "{{ matrix_hookshot_public_endpoint }}/metrics"
 
 # There is no need to edit ports. use matrix_hookshot_container_http_host_bind_ports below to expose ports instead.
 matrix_hookshot_webhook_port: 9000

roles/matrix-bridge-hookshot/tasks/init.yml

@@ -103,11 +103,10 @@
             [matrix_hookshot_matrix_nginx_proxy_configuration]
           }}
 
-    - name: Generate Matrix hookshot proxying configuration for matrix-nginx-proxy
+    - name: Generate hookshot metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/hookshot)
       set_fact:
-        matrix_hookshot_matrix_nginx_proxy_metrics_configuration: |
+        matrix_hookshot_matrix_nginx_proxy_metrics_configuration_matrix_domain: |
-          {% if matrix_hookshot_metrics_enabled and matrix_hookshot_proxy_metrics %}
+          location /metrics/hookshot {
-          location {{ matrix_hookshot_metrics_endpoint }} {
             {% if matrix_nginx_proxy_enabled|default(False) %}
               {# Use the embedded DNS resolver in Docker containers to discover the service #}
               resolver 127.0.0.11 valid=5s;
@@ -117,24 +116,18 @@
               {# Generic configuration for use outside of our container setup #}
               proxy_pass http://127.0.0.1:{{ matrix_hookshot_metrics_port }}/metrics;
             {% endif %}
-            proxy_set_header Host $host;
-            {% if matrix_hookshot_proxy_metrics_basic_auth_enabled %}
-              auth_basic "protected";
-              auth_basic_user_file /nginx-data/matrix-synapse-metrics-htpasswd;
-            {% endif %}
           }
-          {% endif %}
+      when: matrix_hookshot_metrics_enabled|bool and matrix_hookshot_metrics_proxying_enabled|bool
 
-    - name: Register hookshot metrics proxying configuration with matrix-nginx-proxy
+    - name: Register hookshot metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/hookshot)
       set_fact:
-        matrix_nginx_proxy_proxy_grafana_additional_server_configuration_blocks: |
+        matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: |
           {{
-            matrix_nginx_proxy_proxy_grafana_additional_server_configuration_blocks|default([])
+            matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks|default([])
             +
-            [matrix_hookshot_matrix_nginx_proxy_metrics_configuration]
+            [matrix_hookshot_matrix_nginx_proxy_metrics_configuration_matrix_domain]
           }}
-  tags:
+      when: matrix_hookshot_metrics_enabled|bool and matrix_hookshot_metrics_proxying_enabled|bool
-    - always
   when: matrix_hookshot_enabled|bool
 
 - name: Warn about reverse-proxying if matrix-nginx-proxy not used

roles/matrix-bridge-hookshot/tasks/validate_config.yml

@@ -57,3 +57,16 @@
   when: "matrix_hookshot_provisioning_enabled and vars[item] == ''"
   with_items:
     - "matrix_hookshot_provisioning_secret"
+
+- name: (Deprecation) Catch and report old metrics usage
+  fail:
+    msg: >-
+      Your configuration contains a variable (`{{ item }}`), which refers to the old metrics collection system for Hookshot,
+      which exposed metrics on `https://stats.DOMAIN/hookshot/metrics`.
+
+      We now recommend exposing Hookshot metrics in another way, from another URL.
+      Refer to the changelog for more details: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#2022-06-22
+  with_items:
+    - matrix_hookshot_proxy_metrics
+    - matrix_hookshot_metrics_endpoint
+  when: "item in vars"

roles/matrix-nginx-proxy/defaults/main.yml

@@ -192,6 +192,58 @@ matrix_nginx_proxy_proxy_grafana_hostname: "{{ matrix_server_fqn_grafana }}"
 matrix_nginx_proxy_proxy_sygnal_enabled: false
 matrix_nginx_proxy_proxy_sygnal_hostname: "{{ matrix_server_fqn_sygnal }}"
 
+# Controls whether proxying for (Prometheus) metrics (`/metrics/*`) for the various services should be done (on the matrix domain)
+# If the internal Prometheus server (`matrix-prometheus` role) is used, proxying is not necessary, since Prometheus can access each container directly.
+# This is only useful when an external Prometheus will be collecting metrics.
+#
+# To control what kind of metrics are exposed under `/metrics/` (e.g `/metrics/node-exporter`, `/metrics/postgres-exporter`, etc.),
+# use `matrix_SERVICE_metrics_proxying_enabled` variables in each respective role.
+# Roles inject themselves into the matrix-nginx-proxy configuration.
+#
+# To protect the metrics endpoints, see `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled`
+matrix_nginx_proxy_proxy_matrix_metrics_enabled: false
+
+# Controls whether Basic Auth is enabled for all `/metrics/*` endpoints.
+#
+# You can provide the Basic Auth credentials in 2 ways:
+# 1. A single username/password pair using `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username` and `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password`
+# 2. Using raw content (`htpasswd`-generated file) provided in `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content`
+matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled: false
+
+# `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username` and `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password` specify
+# the Basic Auth username/password for protecting `/metrics/*` endpoints.
+# Alternatively, use `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content`.
+matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username: ""
+matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password: ""
+
+# `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content` value will be written verbatim to the htpasswd file protecting `/metrics/*` endpoints.
+# Use this when a single username/password is not enough and you'd like to get more control over credentials.
+#
+# Read the manpage at `man 1 htpasswd` to learn more, then encrypt your password, and paste the encrypted value here.
+# e.g. `htpasswd -c mypass.htpasswd prometheus` and enter `mysecurepw` when prompted yields `prometheus:$apr1$wZhqsn.U$7LC3kMmjUbjNAZjyMyvYv/`
+# The whole thing is needed here. matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content: "prometheus:$apr1$wZhqsn.U$7LC3kMmjUbjNAZjyMyvYv/"
+matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content: ""
+
+# Specifies the path to the htpasswd file holding the htpasswd credentials for protecting `/metrics/*` endpoints
+# This is not meant to be modified.
+matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_path: "{{ matrix_nginx_proxy_data_path_in_container if matrix_nginx_proxy_enabled else matrix_nginx_proxy_data_path }}/matrix-metrics-htpasswd"
+
+# Specifies the Apache container image to use
+# when `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username` and `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password` are provided.
+# This image provides the `htpasswd` tool which we use for generating the htpasswd file protecting `/metrics/*`.
+# To avoid using this, use `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content` instead of supplying username/password.
+# Learn more in: `roles/matrix-nginx-proxy/tasks/nginx-proxy/setup_metrics_auth.yml`.
+matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_image: "{{ matrix_container_global_registry_prefix }}httpd:{{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_image_tag }}"
+matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_image_tag: "2.4.54-alpine3.16"
+matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_force_pull: "{{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_image_tag.endswith(':latest') }}"
+
+# A list of strings containing additional configuration blocks to add to the `location /metrics` configuration (matrix-domain.conf).
+# Do not modify `matrix_nginx_proxy_proxy_matrix_metrics_additional_location_configuration_blocks` and `matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks`.
+# If you'd like to inject your own configuration blocks, use `matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks`.
+matrix_nginx_proxy_proxy_matrix_metrics_additional_location_configuration_blocks: "{{ matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks + matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks }}"
+matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: []
+matrix_nginx_proxy_proxy_matrix_metrics_additional_user_location_configuration_blocks: []
+
 # Controls whether proxying for the matrix-corporal API (`/_matrix/corporal`) should be done (on the matrix domain)
 matrix_nginx_proxy_proxy_matrix_corporal_api_enabled: false
 matrix_nginx_proxy_proxy_matrix_corporal_api_addr_with_container: "matrix-corporal:41081"
@@ -216,18 +268,6 @@ matrix_nginx_proxy_proxy_matrix_identity_api_enabled: false
 matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container: "matrix-ma1sd:{{ matrix_ma1sd_container_port }}"
 matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container: "127.0.0.1:{{ matrix_ma1sd_container_port }}"
 
-# Controls whether proxying for metrics (`/_synapse/metrics`) should be done (on the matrix domain)
-matrix_nginx_proxy_proxy_synapse_metrics: false
-matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled: false
-# The following value will be written verbatim to the htpasswd file that stores the password for nginx to check against and needs to be encoded appropriately.
-# Read the manpage at `man 1 htpasswd` to learn more, then encrypt your password, and paste the encrypted value here.
-# e.g. `htpasswd -c mypass.htpasswd prometheus` and enter `mysecurepw` when prompted yields `prometheus:$apr1$wZhqsn.U$7LC3kMmjUbjNAZjyMyvYv/`
-# The part after `prometheus:` is needed here. matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key: "$apr1$wZhqsn.U$7LC3kMmjUbjNAZjyMyvYv/"
-matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key: ""
-matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_path: "{{ matrix_nginx_proxy_data_path_in_container if matrix_nginx_proxy_enabled else matrix_nginx_proxy_data_path }}/matrix-synapse-metrics-htpasswd"
-matrix_nginx_proxy_proxy_synapse_metrics_addr_with_container: "matrix-synapse:9100"
-matrix_nginx_proxy_proxy_synapse_metrics_addr_sans_container: "127.0.0.1:9100"
-
 # The addresses where the Matrix Client API is.
 # Certain extensions (like matrix-corporal) may override this in order to capture all traffic.
 matrix_nginx_proxy_proxy_matrix_client_api_addr_with_container: "matrix-nginx-proxy:12080"
@@ -260,8 +300,6 @@ matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_prefix_regexes: |
     (['/_synapse/oidc'] if matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_synapse_oidc_api_enabled else [])
     +
     (['/_synapse/admin'] if matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_synapse_admin_api_enabled else [])
-    +
-    (['/_synapse.*/metrics'] if matrix_nginx_proxy_proxy_synapse_metrics else [])
   }}
 
 # Specifies where requests for the root URI (`/`) on the `matrix.` domain should be redirected.

roles/matrix-nginx-proxy/tasks/nginx-proxy/setup_metrics_auth.yml

@@ -0,0 +1,53 @@
+# When we're dealing with raw htpasswd content, we just store it in the file directly.
+- name: Ensure matrix-metrics-htpasswd is present when generated from raw content (protecting /metrics/* URIs)
+  copy:
+    content: "{{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content }}"
+    dest: "{{ matrix_nginx_proxy_data_path }}/matrix-metrics-htpasswd"
+    owner: "{{ matrix_user_username }}"
+    group: "{{ matrix_user_groupname }}"
+    mode: 0600
+  when: not matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username
+
+# Alternatively, we need to use the `htpasswd` tool to generate the htpasswd file.
+# There's an Ansible module that helps with that, but it requires passlib (a Python module) to be installed on the server.
+# See: https://docs.ansible.com/ansible/2.3/htpasswd_module.html#requirements-on-host-that-executes-module
+# We support various distros, with various versions of Python. Installing additional Python modules can be a hassle.
+# As a workaround, we run `htpasswd` from an Apache container image.
+- block:
+  - name: Ensure Apache Docker image is pulled for generating matrix-metrics-htpasswd from username/password (protecting /metrics/* URIs)
+    docker_image:
+      name: "{{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_image }}"
+      source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}"
+      force_source: "{{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}"
+      force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_force_pull }}"
+
+  # We store the password in a file and make the `htpasswd` tool read it from there,
+  # as opposed to passing it directly on stdin (which will expose it to other processes on the server).
+  - name: Store metrics password in a temporary file
+    copy:
+      content: "{{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password }}"
+      dest: "/tmp/matrix-nginx-proxy-metrics-password"
+      mode: 0400
+      owner: "{{ matrix_user_uid }}"
+      group: "{{ matrix_user_gid }}"
+
+  - name: Generate matrix-metrics-htpasswd from username/password (protecting /metrics/* URIs)
+    command:
+      cmd: >-
+        {{ matrix_host_command_docker }} run
+        --rm
+        --user={{ matrix_user_uid }}:{{ matrix_user_gid }}
+        --cap-drop=ALL
+        --network=none
+        --mount type=bind,src={{ matrix_nginx_proxy_data_path }},dst=/data
+        --mount type=bind,src=/tmp/matrix-nginx-proxy-metrics-password,dst=/password,ro
+        --entrypoint=/bin/sh
+        {{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_apache_container_image }}
+        -c
+        'cat /password | htpasswd -i -c /data/matrix-metrics-htpasswd {{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username }} && chmod 600 /data/matrix-metrics-htpasswd'
+
+  - name: Delete temporary metrics password file
+    file:
+      path: /tmp/matrix-nginx-proxy-metrics-password
+      state: absent
+  when: matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username != ''

roles/matrix-nginx-proxy/tasks/setup_nginx_proxy.yml

@@ -31,23 +31,9 @@
     mode: 0644
   when: matrix_nginx_proxy_enabled|bool
 
-- name: Ensure matrix-synapse-metrics-htpasswd is present (protecting /_synapse/metrics URI)
+- name: Setup metrics
-  template:
+  include_tasks: "{{ role_path }}/tasks/nginx-proxy/setup_metrics_auth.yml"
-    src: "{{ role_path }}/templates/nginx/matrix-synapse-metrics-htpasswd.j2"
+  when: matrix_nginx_proxy_proxy_matrix_metrics_enabled|bool and matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled|bool
-    dest: "{{ matrix_nginx_proxy_data_path }}/matrix-synapse-metrics-htpasswd"
-    owner: "{{ matrix_user_username }}"
-    group: "{{ matrix_user_groupname }}"
-    mode: 0400
-  when: "matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled|bool and matrix_nginx_proxy_proxy_synapse_metrics|bool"
-
-- name: Generate sample prometheus.yml for external scraping
-  template:
-    src: "{{ role_path }}/templates/prometheus/external_prometheus.yml.example.j2"
-    dest: "{{ matrix_base_data_path }}/external_prometheus.yml.example"
-    owner: "{{ matrix_user_username }}"
-    group: "{{ matrix_user_groupname }}"
-    mode: 0644
-  when: matrix_nginx_proxy_proxy_synapse_metrics|bool
 
 - name: Ensure Matrix nginx-proxy configured (generic)
   template:
@@ -324,10 +310,15 @@
   file:
     path: "{{ matrix_nginx_proxy_data_path }}/matrix-synapse-metrics-htpasswd"
     state: absent
-  when: "not matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled|bool or not matrix_nginx_proxy_proxy_synapse_metrics|bool"
 
-- name: Ensure sample prometheus.yml for external scraping is deleted
+# This file is now generated by the matrix-synapse role and saved in the Synapse directory
+- name: (Cleanup) Ensure old sample prometheus.yml for external scraping is deleted
   file:
     path: "{{ matrix_base_data_path }}/external_prometheus.yml.example"
     state: absent
-  when: "not matrix_nginx_proxy_proxy_synapse_metrics|bool"
+
+- name: Ensure Matrix nginx-proxy htpasswd is deleted (protecting /metrics/* URIs)
+  file:
+    path: "{{ matrix_nginx_proxy_data_path }}/matrix-metrics-htpasswd"
+    state: absent
+  when: "not matrix_nginx_proxy_proxy_matrix_metrics_enabled|bool or not matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled|bool"

roles/matrix-nginx-proxy/tasks/validate_config.yml

@@ -27,6 +27,14 @@
       `matrix_nginx_proxy_ssl_preset` needs to be set to a known value.
   when: "matrix_nginx_proxy_ssl_preset not in ['modern', 'intermediate', 'old']"
 
+- name: Fail if Basic Auth enabled for metrics, but no credentials supplied
+  fail:
+    msg: |
+      Enabling Basic Auth for metrics (`matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled`) requires:
+      - either a username/password (provided in `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username` and `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password`)
+      - or raw htpasswd content (provided in `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content`)
+  when: "matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled|bool and (matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content == '' and (matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username == '' or matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password == ''))"
+
 - block:
     - name: (Deprecation) Catch and report renamed settings
       fail:
@@ -36,6 +44,7 @@
       with_items:
         - {'old': 'host_specific_matrix_ssl_support_email', 'new': 'matrix_ssl_lets_encrypt_support_email'}
         - {'old': 'host_specific_matrix_ssl_lets_encrypt_support_email', 'new': 'matrix_ssl_lets_encrypt_support_email'}
+        - {'old': 'matrix_nginx_proxy_proxy_synapse_workers_enabled_list', 'new': '<no longer used>'}
       when: "item.old in vars"
 
     - name: Fail if required variables are undefined
@@ -49,3 +58,17 @@
         - "matrix_nginx_proxy_proxy_synapse_client_api_addr_sans_container"
       when: "vars[item] == '' or vars[item] is none"
   when: "matrix_ssl_retrieval_method == 'lets-encrypt'"
+
+- name: (Deprecation) Catch and report old metrics usage
+  fail:
+    msg: >-
+      Your configuration contains a variable (`{{ item }}`), which refers to the old metrics collection system for Synapse,
+      which exposed metrics on `https://matrix.DOMAIN/_synapse/metrics` and `https://matrix.DOMAIN/_synapse-worker-TYPE-ID/metrics`.
+
+      We now recommend exposing Synapse metrics in another way, from another URL.
+      Refer to the changelog for more details: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#2022-06-22
+  with_items:
+    - matrix_nginx_proxy_proxy_synapse_metrics
+    - matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled
+    - matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key
+  when: "item in vars"

roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-domain.conf.j2

@@ -45,6 +45,19 @@
 		{{ render_nginx_status_location_block(matrix_nginx_proxy_proxy_matrix_nginx_status_allowed_addresses) }}
 	{% endif %}
 
+	{% if matrix_nginx_proxy_proxy_matrix_metrics_enabled %}
+	location /metrics {
+		{% if matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled %}
+			auth_basic "protected";
+			auth_basic_user_file {{ matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_path }};
+		{% endif %}
+
+		{% for configuration_block in matrix_nginx_proxy_proxy_matrix_metrics_additional_location_configuration_blocks %}
+			{{- configuration_block }}
+		{% endfor %}
+	}
+	{% endif %}
+
 	{% if matrix_nginx_proxy_proxy_matrix_corporal_api_enabled %}
 	location ^~ /_matrix/corporal {
 		{% if matrix_nginx_proxy_enabled %}

roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2

@@ -145,45 +145,6 @@ server {
 		{{- configuration_block }}
 	{% endfor %}
 
-	{% if matrix_nginx_proxy_proxy_synapse_metrics %}
-	location /_synapse/metrics {
-		{% if matrix_nginx_proxy_enabled %}
-			{# Use the embedded DNS resolver in Docker containers to discover the service #}
-			resolver 127.0.0.11 valid=5s;
-			set $backend "{{ matrix_nginx_proxy_proxy_synapse_metrics_addr_with_container }}";
-			proxy_pass http://$backend;
-		{% else %}
-			{# Generic configuration for use outside of our container setup #}
-			proxy_pass http://{{ matrix_nginx_proxy_proxy_synapse_metrics_addr_sans_container }};
-		{% endif %}
-
-		proxy_set_header Host $host;
-
-		{% if matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled %}
-			auth_basic "protected";
-			auth_basic_user_file {{ matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_path }};
-		{% endif %}
-	}
-	{% endif %}
-
-	{% if matrix_nginx_proxy_enabled and matrix_nginx_proxy_proxy_synapse_metrics %}
-		{% for worker in matrix_nginx_proxy_proxy_synapse_workers_enabled_list %}
-			{% if worker.metrics_port != 0 %}
-				location /_synapse-worker-{{ worker.type }}-{{ worker.instanceId }}/metrics {
-					resolver 127.0.0.11 valid=5s;
-					set $backend "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.metrics_port }}";
-					proxy_pass http://$backend/_synapse/metrics;
-					proxy_set_header Host $host;
-
-					{% if matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled %}
-						auth_basic "protected";
-						auth_basic_user_file {{ matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_path }};
-					{% endif %}
-				}
-			{% endif %}
-		{% endfor %}
-	{% endif %}
-
 	{# Everything else just goes to the API server ##}
 	location / {
 		{% if matrix_nginx_proxy_enabled %}

roles/matrix-nginx-proxy/templates/nginx/matrix-synapse-metrics-htpasswd.j2

@@ -1,3 +0,0 @@
-#jinja2: lstrip_blocks: "True"
-# User and password for protecting /_synapse/metrics URI
-prometheus:{{ matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key }}

roles/matrix-prometheus-node-exporter/defaults/main.yml

@@ -17,10 +17,17 @@ matrix_prometheus_node_exporter_systemd_required_services_list: ['docker.service
 # List of systemd services that matrix-prometheus.service wants
 matrix_prometheus_node_exporter_systemd_wanted_services_list: []
 
+# Controls whether node-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/node-exporter`.
+# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`.
+# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`.
+matrix_prometheus_node_exporter_metrics_proxying_enabled: false
+
 # Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9100 in the container).
 #
 # Takes an "<ip>:<port>" value (e.g. "127.0.0.1:9100"), or empty string to not expose.
 #
+# You likely don't need to do this. See `matrix_prometheus_node_exporter_metrics_proxying_enabled`.
+#
 # Official recommendations are to run this container with `--net=host`,
 # but we don't do that, since it:
 # - likely exposes the metrics web server way too publicly (before applying https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1008)

roles/matrix-prometheus-node-exporter/tasks/init.yml

@@ -3,3 +3,39 @@
 - set_fact:
     matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-prometheus-node-exporter.service'] }}"
   when: matrix_prometheus_node_exporter_enabled|bool
+
+- block:
+    - name: Fail if matrix-nginx-proxy role already executed
+      fail:
+        msg: >-
+          Trying to append node-exporter's reverse-proxying configuration to matrix-nginx-proxy,
+          but it's pointless since the matrix-nginx-proxy role had already executed.
+          To fix this, please change the order of roles in your playbook,
+          so that the matrix-nginx-proxy role would run after the matrix-prometheus-node-exporter role.
+      when: matrix_nginx_proxy_role_executed|default(False)|bool
+
+    - name: Generate node-exporter metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/node-exporter)
+      set_fact:
+        matrix_prometheus_node_exporter_nginx_metrics_configuration_block: |
+          location /metrics/node-exporter {
+            {% if matrix_nginx_proxy_enabled|default(False) %}
+              {# Use the embedded DNS resolver in Docker containers to discover the service #}
+              resolver 127.0.0.11 valid=5s;
+              set $backend "matrix-prometheus-node-exporter:9100";
+              proxy_pass http://$backend/metrics;
+            {% else %}
+              {# Generic configuration for use outside of our container setup #}
+              {# This may be implemented in the future. #}
+              return 404 "matrix-nginx-proxy is disabled, so metrics are unavailable";
+            {% endif %}
+          }
+
+    - name: Register node-exporter metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/node-exporter)
+      set_fact:
+        matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: |
+          {{
+            matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks|default([])
+            +
+            [matrix_prometheus_node_exporter_nginx_metrics_configuration_block]
+          }}
+  when: matrix_prometheus_node_exporter_enabled|bool and matrix_prometheus_node_exporter_metrics_proxying_enabled|bool

roles/matrix-prometheus-postgres-exporter/defaults/main.yml

@@ -28,11 +28,17 @@ matrix_prometheus_postgres_exporter_database_hostname: 'matrix-postgres'
 matrix_prometheus_postgres_exporter_database_port: 5432
 matrix_prometheus_postgres_exporter_database_name: 'matrix_prometheus_postgres_exporter'
 
+# Controls whether postgres-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/postgres-exporter`.
+# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`.
+# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`.
+matrix_prometheus_postgres_exporter_metrics_proxying_enabled: false
 
 # Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9187 in the container).
 #
 # Takes an "<ip>:<port>" value (e.g. "127.0.0.1:9187"), or empty string to not expose.
 #
+# You likely don't need to do this. See `matrix_prometheus_postgres_exporter_metrics_proxying_enabled`.
+#
 # Official recommendations are to run this container with `--net=host`,
 # but we don't do that, since it:
 # - likely exposes the metrics web server way too publicly (before applying https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1008)

roles/matrix-prometheus-postgres-exporter/tasks/init.yml

@@ -3,3 +3,39 @@
 - set_fact:
     matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-prometheus-postgres-exporter.service'] }}"
   when: matrix_prometheus_postgres_exporter_enabled|bool
+
+- block:
+    - name: Fail if matrix-nginx-proxy role already executed
+      fail:
+        msg: >-
+          Trying to append postgres-exporter's reverse-proxying configuration to matrix-nginx-proxy,
+          but it's pointless since the matrix-nginx-proxy role had already executed.
+          To fix this, please change the order of roles in your playbook,
+          so that the matrix-nginx-proxy role would run after the matrix-prometheus-postgres-exporter role.
+      when: matrix_nginx_proxy_role_executed|default(False)|bool
+
+    - name: Generate postgres-exporter metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/postgres-exporter)
+      set_fact:
+        matrix_prometheus_postgres_exporter_nginx_metrics_configuration_block: |
+          location /metrics/postgres-exporter {
+            {% if matrix_nginx_proxy_enabled|default(False) %}
+              {# Use the embedded DNS resolver in Docker containers to discover the service #}
+              resolver 127.0.0.11 valid=5s;
+              set $backend "matrix-prometheus-postgres-exporter:9187";
+              proxy_pass http://$backend/metrics;
+            {% else %}
+              {# Generic configuration for use outside of our container setup #}
+              {# This may be implemented in the future. #}
+              return 404 "matrix-nginx-proxy is disabled, so metrics are unavailable";
+            {% endif %}
+          }
+
+    - name: Register postgres-exporter metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/postgres-exporter)
+      set_fact:
+        matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: |
+          {{
+            matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks|default([])
+            +
+            [matrix_prometheus_postgres_exporter_nginx_metrics_configuration_block]
+          }}
+  when: matrix_prometheus_node_exporter_enabled|bool and matrix_prometheus_node_exporter_metrics_proxying_enabled|bool

roles/matrix-synapse/defaults/main.yml

@@ -334,6 +334,14 @@ matrix_url_preview_accept_language: ['en-US', 'en']
 matrix_synapse_metrics_enabled: false
 matrix_synapse_metrics_port: 9100
 
+# Controls whether Synapse metrics should be proxied (exposed) on:
+# - `matrix.DOMAIN/metrics/synapse/main-process` for the main process
+# - `matrix.DOMAIN/metrics/synapse/worker/{type}-{id}` for each worker process
+#
+# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`.
+# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`.
+matrix_synapse_metrics_proxying_enabled: false
+
 # Enable the Synapse manhole
 # See https://github.com/matrix-org/synapse/blob/master/docs/manhole.md
 matrix_synapse_manhole_enabled: false

roles/matrix-synapse/tasks/init.yml

@@ -25,3 +25,63 @@
 - set_fact:
     matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-goofys.service'] }}"
   when: matrix_s3_media_store_enabled|bool
+
+- block:
+    - name: Fail if matrix-nginx-proxy role already executed
+      fail:
+        msg: >-
+          Trying to append Synapse's reverse-proxying configuration to matrix-nginx-proxy,
+          but it's pointless since the matrix-nginx-proxy role had already executed.
+          To fix this, please change the order of roles in your playbook,
+          so that the matrix-nginx-proxy role would run after the matrix-synapse role.
+      when: matrix_nginx_proxy_role_executed|default(False)|bool
+
+    - name: Generate synapse metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/synapse/main-process)
+      set_fact:
+        matrix_synapse_nginx_metrics_configuration_block: |
+          location /metrics/synapse/main-process {
+            {% if matrix_nginx_proxy_enabled|default(False) %}
+              {# Use the embedded DNS resolver in Docker containers to discover the service #}
+              resolver 127.0.0.11 valid=5s;
+              set $backend "matrix-synapse:{{ matrix_synapse_metrics_port }}";
+              proxy_pass http://$backend/_synapse/metrics;
+            {% else %}
+              {# Generic configuration for use outside of our container setup #}
+              proxy_pass http://127.0.0.1:{{ matrix_synapse_metrics_port }}/_synapse/metrics;
+            {% endif %}
+          }
+
+    - name: Register synapse metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/synapse/main-process)
+      set_fact:
+        matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: |
+          {{
+            matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks|default([])
+            +
+            [matrix_synapse_nginx_metrics_configuration_block]
+          }}
+
+    - name: Generate synapse worker metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/synapse/worker)
+      set_fact:
+        matrix_synapse_worker_nginx_metrics_configuration_block: |
+          {% for worker in matrix_synapse_workers_enabled_list %}
+          {% if worker.metrics_port != 0 %}
+          location /metrics/synapse/worker/{{ worker.type }}-{{ worker.instanceId }} {
+            resolver 127.0.0.11 valid=5s;
+            set $backend "matrix-synapse-worker-{{ worker.type }}-{{ worker.instanceId }}:{{ worker.metrics_port }}";
+            proxy_pass http://$backend/_synapse/metrics;
+            proxy_set_header Host $host;
+          }
+          {% endif %}
+          {% endfor %}
+      when: matrix_synapse_workers_enabled_list|length > 0
+
+    - name: Register synapse worker metrics proxying configuration with matrix-nginx-proxy (matrix.DOMAIN/metrics/synapse/worker)
+      set_fact:
+        matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks: |
+          {{
+            matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks|default([])
+            +
+            [matrix_synapse_worker_nginx_metrics_configuration_block]
+          }}
+      when: matrix_synapse_workers_enabled_list|length > 0
+  when: matrix_synapse_enabled|bool and matrix_synapse_metrics_proxying_enabled|bool

roles/matrix-synapse/tasks/synapse/setup_install.yml

@@ -119,3 +119,12 @@
     src: "{{ role_path }}/templates/synapse/usr-local-bin/matrix-synapse-register-user.j2"
     dest: "{{ matrix_local_bin_path }}/matrix-synapse-register-user"
     mode: 0755
+
+- name: Generate sample prometheus.yml for external scraping
+  template:
+    src: "{{ role_path }}/templates/synapse/prometheus/external_prometheus.yml.example.j2"
+    dest: "{{ matrix_synapse_base_path }}/external_prometheus.yml.example"
+    owner: "{{ matrix_user_username }}"
+    group: "{{ matrix_user_groupname }}"
+    mode: 0644
+  when: matrix_synapse_metrics_proxying_enabled|bool

roles/matrix-synapse/tasks/synapse/setup_uninstall.yml

@@ -29,3 +29,9 @@
   docker_image:
     name: "{{ matrix_synapse_docker_image }}"
     state: absent
+
+- name: Ensure sample prometheus.yml for external scraping is deleted
+  file:
+    path: "{{ matrix_synapse_base_path }}/external_prometheus.yml.example"
+    state: absent
+  when: "not matrix_synapse_metrics_proxying_enabled|bool"

roles/matrix-synapse/templates/synapse/prometheus/external_prometheus.yml.example.j2

@@ -11,29 +11,29 @@ rule_files:
 
 scrape_configs:
   - job_name: 'synapse'
-    metrics_path: /_synapse/metrics
+    metrics_path: /metrics/synapse/main-process
-    scheme: {{ 'https' if matrix_nginx_proxy_https_enabled else 'http' }}
+    scheme: {{ 'https' if matrix_nginx_proxy_https_enabled|default(true) else 'http' }}
-{% if matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled %}
+{% if matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled|default(true) %}
     basic_auth:
       username: prometheus
       password_file: /path/to/your/passwordfile.pwd
 {% endif %}
     static_configs:
-      - targets: ['{{ matrix_server_fqn_matrix }}:{{ matrix_nginx_proxy_container_https_host_bind_port if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_http_host_bind_port }}']
+      - targets: ['{{ matrix_server_fqn_matrix }}:{{ matrix_nginx_proxy_container_https_host_bind_port|default(443) if matrix_nginx_proxy_https_enabled|default(true) else matrix_nginx_proxy_container_http_host_bind_port|default(80) }}']
         labels:
           job: "master"
           index: "0"
-{% for worker in matrix_nginx_proxy_proxy_synapse_workers_enabled_list %}
+{% for worker in matrix_synapse_workers_enabled_list %}
   - job_name: 'synapse-{{ worker.type }}-{{ worker.instanceId }}'
-    metrics_path: /_synapse-worker-{{ worker.type }}-{{ worker.instanceId }}/metrics
+    metrics_path: /metrics/synapse/worker/{{ worker.type }}-{{ worker.instanceId }}
-    scheme: {{ 'https' if matrix_nginx_proxy_https_enabled else 'http' }}
+    scheme: {{ 'https' if matrix_nginx_proxy_https_enabled|default(true) else 'http' }}
-{% if matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled %}
+{% if matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_enabled|default(true) %}
     basic_auth:
       username: prometheus
       password_file: /path/to/your/passwordfile.pwd
 {% endif %}
     static_configs:
-      - targets: ['{{ matrix_server_fqn_matrix }}:{{ matrix_nginx_proxy_container_https_host_bind_port if matrix_nginx_proxy_https_enabled else matrix_nginx_proxy_container_http_host_bind_port }}']
+      - targets: ['{{ matrix_server_fqn_matrix }}:{{ matrix_nginx_proxy_container_https_host_bind_port|default(443) if matrix_nginx_proxy_https_enabled|default(true) else matrix_nginx_proxy_container_http_host_bind_port|default(80) }}']
         labels:
           job: "{{ worker.type }}"
           index: "{{ worker.instanceId }}"

setup.yml

@@ -47,6 +47,7 @@
     - matrix-dendrite
     - matrix-synapse-admin
     - matrix-prometheus-node-exporter
+    - matrix-prometheus-postgres-exporter
     - matrix-prometheus
     - matrix-grafana
     - matrix-registration
@@ -63,6 +64,5 @@
     - matrix-coturn
     - matrix-aux
     - matrix-postgres-backup
-    - matrix-prometheus-postgres-exporter
     - matrix-backup-borg
     - matrix-common-after