From 9ed2e04d808593c146d81b361baa57c078364dfd Mon Sep 17 00:00:00 2001 From: Slavi Pantaleev Date: Sat, 21 Jan 2023 11:07:04 +0200 Subject: [PATCH] Switch from matrix-prometheus-node-exporter to an external prometheus_node_exporter role --- CHANGELOG.md | 15 +++++ ...configuring-playbook-prometheus-grafana.md | 10 +-- group_vars/matrix_servers | 27 ++++++-- playbooks/matrix.yml | 3 +- requirements.yml | 3 + .../defaults/main.yml | 67 ------------------- .../tasks/main.yml | 26 ------- .../tasks/setup_install.yml | 19 ------ .../tasks/setup_uninstall.yml | 20 ------ ...matrix-prometheus-node-exporter.service.j2 | 44 ------------ .../vars/main.yml | 5 -- .../defaults/main.yml | 15 +++++ .../tasks/main.yml | 19 ++++++ .../inject_into_nginx_proxy.yml | 19 ++---- .../defaults/main.yml | 4 ++ .../tasks/validate_config.yml | 16 +++++ 16 files changed, 104 insertions(+), 208 deletions(-) delete mode 100644 roles/custom/matrix-prometheus-node-exporter/defaults/main.yml delete mode 100644 roles/custom/matrix-prometheus-node-exporter/tasks/main.yml delete mode 100644 roles/custom/matrix-prometheus-node-exporter/tasks/setup_install.yml delete mode 100644 roles/custom/matrix-prometheus-node-exporter/tasks/setup_uninstall.yml delete mode 100644 roles/custom/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 delete mode 100644 roles/custom/matrix-prometheus-node-exporter/vars/main.yml create mode 100644 roles/custom/matrix-prometheus-services-proxy-connect/defaults/main.yml create mode 100644 roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml rename roles/custom/{matrix-prometheus-node-exporter/tasks => matrix-prometheus-services-proxy-connect/tasks/prometheus-node-exporter}/inject_into_nginx_proxy.yml (51%) create mode 100644 roles/custom/matrix_playbook_migration/defaults/main.yml diff --git a/CHANGELOG.md b/CHANGELOG.md index b66cc8ba7..e4607148c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,18 @@ +# 2023-01-21 + +## The matrix-prometheus-node-exporter role lives independently now + +**TLDR**: the `matrix-prometheus-node-exporter` role is now included from another repository. Some variables have been renamed. All functionality remains intact. + +The `matrix-prometheus-node-exporter` role (which configures [Prometheus node exporter](https://github.com/prometheus/node_exporter)) has been extracted from the playbook and now lives in its own repository at https://gitlab.com/etke.cc/roles/prometheus_node_exporter. + +It's still part of the playbook, but is now installed via `ansible-galaxy` (by running `just roles` / `make roles`). Some variables have been renamed (`matrix_prometheus_node_exporter_` -> `prometheus_node_exporter_`, etc.). The playbook will report all variables that you need to rename to get upgraded. All functionality remains intact. + +A new `matrix-prometheus-services-proxy-connect` role was added to the playbook to help integrate the new `prometheus_node_exporter` role with our own services (`matrix-nginx-proxy`) + +Other roles which aren't strictly related to Matrix are likely to follow this fate of moving to their own repositories. Extracting them out allows other Ansible playbooks to make use of these roles easily. + + # 2023-01-13 ## Support for running commands via just diff --git a/docs/configuring-playbook-prometheus-grafana.md b/docs/configuring-playbook-prometheus-grafana.md index 0b785c359..703c03b22 100644 --- a/docs/configuring-playbook-prometheus-grafana.md +++ b/docs/configuring-playbook-prometheus-grafana.md @@ -10,7 +10,7 @@ Remember to add `stats.` to DNS as described in [Configuring DNS](c matrix_prometheus_enabled: true # You can remove this, if unnecessary. -matrix_prometheus_node_exporter_enabled: true +prometheus_node_exporter_enabled: true # You can remove this, if unnecessary. matrix_prometheus_postgres_exporter_enabled: true @@ -40,7 +40,7 @@ The retention policy of Prometheus metrics is [15 days by default](https://prome Name | Description -----|---------- `matrix_prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about. -`matrix_prometheus_node_exporter_enabled`|[Node Exporter](https://prometheus.io/docs/guides/node-exporter/) is an addon of sorts to Prometheus that collects generic system information such as CPU, memory, filesystem, and even system temperatures +`prometheus_node_exporter_enabled`|[Node Exporter](https://prometheus.io/docs/guides/node-exporter/) is an addon of sorts to Prometheus that collects generic system information such as CPU, memory, filesystem, and even system temperatures `matrix_prometheus_postgres_exporter_enabled`|[Postgres Exporter](configuring-playbook-prometheus-postgres.md) is an addon of sorts to expose Postgres database metrics to Prometheus. `matrix_prometheus_nginxlog_exporter_enabled`|[NGINX Log Exporter](configuring-playbook-prometheus-nginxlog.md) is an addon of sorts to expose NGINX logs to Prometheus. `matrix_grafana_enabled`|[Grafana](https://grafana.com/) is the visual component. It shows (on the `stats.` subdomain) the dashboards with the graphs that we're interested in @@ -52,7 +52,7 @@ Name | Description Metrics and resulting graphs can contain a lot of information. This includes system specs but also usage patterns. This applies especially to small personal/family scale homeservers. Someone might be able to figure out when you wake up and go to sleep by looking at the graphs over time. Think about this before enabling anonymous access. And you should really not forget to change your Grafana password. -Most of our docker containers run with limited system access, but the `prometheus-node-exporter` has access to the host network stack and (readonly) root filesystem. This is required to report on them. If you don't like that, you can set `matrix_prometheus_node_exporter_enabled: false` (which is actually the default). You will still get Synapse metrics with this container disabled. Both of the dashboards will always be enabled, so you can still look at historical data after disabling either source. +Most of our docker containers run with limited system access, but the `prometheus-node-exporter` has access to the host network stack and (readonly) root filesystem. This is required to report on them. If you don't like that, you can set `prometheus_node_exporter_enabled: false` (which is actually the default). You will still get Synapse metrics with this container disabled. Both of the dashboards will always be enabled, so you can still look at historical data after disabling either source. ## Collecting metrics to an external Prometheus server @@ -74,8 +74,8 @@ Name | Description `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_raw_content`|Set this to the Basic Authentication credentials (raw `htpasswd` file content) used to protect `/metrics/*`. This htpasswd-file needs to be generated with the `htpasswd` tool and can include multiple username/password pairs. If you only need one credential, use `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_username` and `matrix_nginx_proxy_proxy_matrix_metrics_basic_auth_password` instead. `matrix_synapse_metrics_enabled`|Set this to `true` to make Synapse expose metrics (locally, on the container network) `matrix_synapse_metrics_proxying_enabled`|Set this to `true` to expose Synapse's metrics on `https://matrix.DOMAIN/metrics/synapse/main-process` and `https://matrix.DOMAIN/metrics/synapse/worker/TYPE-ID` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`). Read [below](#collecting-synapse-worker-metrics-to-an-external-prometheus-server) if you're running a Synapse worker setup (`matrix_synapse_workers_enabled: true`). -`matrix_prometheus_node_exporter_enabled`|Set this to `true` to enable the node (general system stats) exporter (locally, on the container network) -`matrix_prometheus_node_exporter_metrics_proxying_enabled`|Set this to `true` to expose the node (general system stats) metrics on `https://matrix.DOMAIN/metrics/node-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) +`prometheus_node_exporter_enabled`|Set this to `true` to enable the node (general system stats) exporter (locally, on the container network) +`matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled`|Set this to `true` to expose the node (general system stats) metrics on `https://matrix.DOMAIN/metrics/node-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) `matrix_prometheus_postgres_exporter_enabled`|Set this to `true` to enable the [Postgres exporter](configuring-playbook-prometheus-postgres.md) (locally, on the container network) `matrix_prometheus_nginxlog_exporter_enabled`|Set this to `true` to enable the [NGINX Log exporter](configuring-playbook-prometheus-nginxlog.md) (locally, on the container network) `matrix_prometheus_postgres_exporter_metrics_proxying_enabled`|Set this to `true` to expose the [Postgres exporter](configuring-playbook-prometheus-postgres.md) metrics on `https://matrix.DOMAIN/metrics/postgres-exporter` (only takes effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`) diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 0e6e3691f..70abde3af 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -301,7 +301,7 @@ devture_systemd_service_manager_services_list_auto: | + ([{'name': 'matrix-prometheus.service', 'priority': 4000, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-core']}] if matrix_prometheus_enabled else []) + - ([{'name': 'matrix-prometheus-node-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-node-exporters']}] if matrix_prometheus_node_exporter_enabled else []) + ([{'name': (prometheus_node_exporter_identifier + '.service'), 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-node-exporters']}] if prometheus_node_exporter_enabled else []) + ([{'name': 'matrix-prometheus-postgres-exporter.service', 'priority': 3900, 'groups': ['matrix', 'monitoring', 'prometheus', 'prometheus-node-exporters']}] if matrix_prometheus_postgres_exporter_enabled else []) + @@ -2927,15 +2927,28 @@ matrix_synapse_admin_container_image_self_build: "{{ matrix_architecture not in ###################################################################### # -# matrix-prometheus-node-exporter +# etke/prometheus_node_exporter # ###################################################################### -matrix_prometheus_node_exporter_enabled: false +prometheus_node_exporter_enabled: false + +prometheus_node_exporter_identifier: matrix-prometheus-node-exporter + +prometheus_node_exporter_base_path: "{{ matrix_base_data_path }}/prometheus-node-exporter" + +prometheus_node_exporter_uid: "{{ matrix_user_uid }}" +prometheus_node_exporter_gid: "{{ matrix_user_gid }}" + +prometheus_node_exporter_server_fqn: "{{ matrix_server_fqn_matrix }}" + +prometheus_node_exporter_container_network: "{{ matrix_docker_network }}" + +prometheus_node_exporter_container_labels_traefik_enabled: false ###################################################################### # -# /matrix-prometheus-node-exporter +# /etke/prometheus_node_exporter # ###################################################################### @@ -2971,8 +2984,8 @@ matrix_prometheus_scraper_synapse_targets: ['matrix-synapse:{{ matrix_synapse_me matrix_prometheus_scraper_synapse_workers_enabled_list: "{{ matrix_synapse_workers_enabled_list }}" matrix_prometheus_scraper_synapse_rules_synapse_tag: "{{ matrix_synapse_docker_image_tag }}" -matrix_prometheus_scraper_node_enabled: "{{ matrix_prometheus_node_exporter_enabled }}" -matrix_prometheus_scraper_node_targets: "{{ ['matrix-prometheus-node-exporter:9100'] if matrix_prometheus_node_exporter_enabled else [] }}" +matrix_prometheus_scraper_node_enabled: "{{ prometheus_node_exporter_enabled }}" +matrix_prometheus_scraper_node_targets: "{{ [prometheus_node_exporter_identifier + ':9100'] if prometheus_node_exporter_enabled else [] }}" matrix_prometheus_scraper_postgres_enabled: "{{ matrix_prometheus_postgres_exporter_enabled }}" matrix_prometheus_scraper_postgres_targets: "{{ ['matrix-prometheus-postgres-exporter:'+ matrix_prometheus_postgres_exporter_port | string] if matrix_prometheus_scraper_postgres_enabled else [] }}" @@ -3030,7 +3043,7 @@ matrix_grafana_dashboard_download_urls: | {{ (matrix_synapse_grafana_dashboard_urls if matrix_homeserver_implementation == 'synapse' and matrix_synapse_metrics_enabled else []) + - (matrix_prometheus_node_exporter_dashboard_urls if matrix_prometheus_node_exporter_enabled else []) + (prometheus_node_exporter_dashboard_urls if prometheus_node_exporter_enabled else []) + (matrix_prometheus_postgres_exporter_dashboard_urls if matrix_prometheus_postgres_exporter_enabled else []) + diff --git a/playbooks/matrix.yml b/playbooks/matrix.yml index 409328aec..5190521e2 100755 --- a/playbooks/matrix.yml +++ b/playbooks/matrix.yml @@ -86,11 +86,12 @@ - custom/matrix-dendrite - custom/matrix-conduit - custom/matrix-synapse-admin - - custom/matrix-prometheus-node-exporter + - galaxy/prometheus_node_exporter - custom/matrix-prometheus-postgres-exporter - custom/matrix-prometheus-nginxlog-exporter - custom/matrix-prometheus - custom/matrix-grafana + - custom/matrix-prometheus-services-proxy-connect - custom/matrix-registration - custom/matrix-client-element - custom/matrix-client-hydrogen diff --git a/requirements.yml b/requirements.yml index 03eed35df..07cab1972 100644 --- a/requirements.yml +++ b/requirements.yml @@ -29,3 +29,6 @@ - src: git+https://github.com/devture/com.devture.ansible.role.playbook_runtime_messages.git version: 9b4b088c62b528b73a9a7c93d3109b091dd42ec6 + +- src: git+https://gitlab.com/etke.cc/roles/prometheus_node_exporter.git + version: v1.5.0-0 diff --git a/roles/custom/matrix-prometheus-node-exporter/defaults/main.yml b/roles/custom/matrix-prometheus-node-exporter/defaults/main.yml deleted file mode 100644 index 070e86164..000000000 --- a/roles/custom/matrix-prometheus-node-exporter/defaults/main.yml +++ /dev/null @@ -1,67 +0,0 @@ ---- -# matrix-prometheus-node-exporter is an Prometheus exporter for machine metrics -# See: https://prometheus.io/docs/guides/node-exporter/ -# Project source code URL: https://github.com/prometheus/node_exporter - -matrix_prometheus_node_exporter_enabled: false - -matrix_prometheus_node_exporter_version: v1.5.0 -matrix_prometheus_node_exporter_docker_image: "{{ matrix_container_global_registry_prefix }}prom/node-exporter:{{ matrix_prometheus_node_exporter_version }}" -matrix_prometheus_node_exporter_docker_image_force_pull: "{{ matrix_prometheus_node_exporter_docker_image.endswith(':latest') }}" - -# A list of extra arguments to pass to the container -matrix_prometheus_node_exporter_container_extra_arguments: [] - -# A list of extra arguments to pass to the node_exporter process -# -# Example: -# matrix_prometheus_node_exporter_process_extra_arguments: -# - "--collector.systemd" -# - "--collector.logind" -# -# Note: the above is just an example. Various collectors may require various tweaks to be able to run. -# Running the systemd collector requires the following `matrix_prometheus_node_exporter_container_extra_arguments`: -# - the socket to be mounted as well (`--mount type=bind,src=/var/run/dbus/system_bus_socket,dst=/var/run/dbus/system_bus_socket,ro,bind-propagation=rslave`) -# - (on AppArmor-based distros) disabling AppArmor protection (`--security-opt apparmor=unconfined`) -matrix_prometheus_node_exporter_process_extra_arguments: [] - -# List of systemd services that matrix-prometheus.service depends on -matrix_prometheus_node_exporter_systemd_required_services_list: ['docker.service'] - -# List of systemd services that matrix-prometheus.service wants -matrix_prometheus_node_exporter_systemd_wanted_services_list: [] - -# Controls whether node-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/node-exporter`. -# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. -# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. -matrix_prometheus_node_exporter_metrics_proxying_enabled: false - -# Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9100 in the container). -# -# Takes an ":" value (e.g. "127.0.0.1:9100"), just a port number or empty string to not expose. -# -# You likely don't need to do this. See `matrix_prometheus_node_exporter_metrics_proxying_enabled`. -# -# Official recommendations are to run this container with `--net=host`, -# but we don't do that, since it: -# - likely exposes the metrics web server way too publicly (before applying https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1008) -# - or listens on a loopback interface only (--net=host and 127.0.0.1:9100), which is not reachable from another container (like `matrix-prometheus`) -# -# Using `--net=host` and binding to Docker's `matrix` bridge network may be a solution to both, -# but that's trickier to accomplish and won't necessarily work (hasn't been tested). -# -# Not using `--net=host` means that our network statistic reports are likely broken (inaccurate), -# because node-exporter can't see all interfaces, etc. -# For now, we'll live with that, until someone develops a better solution. -matrix_prometheus_node_exporter_container_http_host_bind_port: '' - -# If you are supplying your own NGINX proxy but want to use the provided exporters you will have to supply an ":" value for the containers to bind to on your host. -# If matrix_prometheus_node_exporter_container_http_host_bind_port is set to just a port number, this will default to "127.0.0.1:" -# If matrix_prometheus_node_exporter_container_http_host_bind_port is set to an IP that is not 0.0.0.0 and a port, that ":" value will be used -# Otherwise this value will be empty and you will have to manually configure your NGINX config file. (If you are using the config files generated by this playbook, you will have to edit matrix-domain.conf) -matrix_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host: "{{ '127.0.0.1' + matrix_prometheus_node_exporter_container_http_host_bind_port_number_raw if not ':' in matrix_prometheus_node_exporter_container_http_host_bind_port else (matrix_prometheus_node_exporter_container_http_host_bind_port if matrix_prometheus_node_exporter_container_http_host_bind_port.split(':')[0] != '0.0.0.0' else '') }}" - -# matrix_prometheus_node_exporter_dashboard_urls contains a list of URLs with Grafana dashboard definitions. -# If the Grafana role is enabled, these dashboards will be downloaded. -matrix_prometheus_node_exporter_dashboard_urls: - - https://raw.githubusercontent.com/rfrail3/grafana-dashboards/master/prometheus/node-exporter-full.json diff --git a/roles/custom/matrix-prometheus-node-exporter/tasks/main.yml b/roles/custom/matrix-prometheus-node-exporter/tasks/main.yml deleted file mode 100644 index 6a0cd8307..000000000 --- a/roles/custom/matrix-prometheus-node-exporter/tasks/main.yml +++ /dev/null @@ -1,26 +0,0 @@ ---- - -- block: - - when: matrix_prometheus_node_exporter_enabled | bool and matrix_prometheus_node_exporter_metrics_proxying_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/inject_into_nginx_proxy.yml" - tags: - - setup-all - - setup-nginx-proxy - - install-all - - install-nginx-proxy - -- block: - - when: matrix_prometheus_node_exporter_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_install.yml" - tags: - - setup-all - - setup-prometheus-node-exporter - - install-all - - install-prometheus-node-exporter - -- block: - - when: not matrix_prometheus_node_exporter_enabled | bool - ansible.builtin.include_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" - tags: - - setup-all - - setup-prometheus-node-exporter diff --git a/roles/custom/matrix-prometheus-node-exporter/tasks/setup_install.yml b/roles/custom/matrix-prometheus-node-exporter/tasks/setup_install.yml deleted file mode 100644 index 3c0af96b9..000000000 --- a/roles/custom/matrix-prometheus-node-exporter/tasks/setup_install.yml +++ /dev/null @@ -1,19 +0,0 @@ ---- - -- name: Ensure matrix-prometheus-node-exporter image is pulled - community.docker.docker_image: - name: "{{ matrix_prometheus_node_exporter_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_prometheus_node_exporter_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_prometheus_node_exporter_docker_image_force_pull }}" - register: result - retries: "{{ devture_playbook_help_container_retries_count }}" - delay: "{{ devture_playbook_help_container_retries_delay }}" - until: result is not failed - -- name: Ensure matrix-prometheus-node-exporter.service installed - ansible.builtin.template: - src: "{{ role_path }}/templates/systemd/matrix-prometheus-node-exporter.service.j2" - dest: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-node-exporter.service" - mode: 0644 - register: matrix_prometheus_node_exporter_systemd_service_result diff --git a/roles/custom/matrix-prometheus-node-exporter/tasks/setup_uninstall.yml b/roles/custom/matrix-prometheus-node-exporter/tasks/setup_uninstall.yml deleted file mode 100644 index b5ef4f0f7..000000000 --- a/roles/custom/matrix-prometheus-node-exporter/tasks/setup_uninstall.yml +++ /dev/null @@ -1,20 +0,0 @@ ---- - -- name: Check existence of matrix-prometheus-node-exporter service - ansible.builtin.stat: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-node-exporter.service" - register: matrix_prometheus_node_exporter_service_stat - -- when: matrix_prometheus_node_exporter_service_stat.stat.exists | bool - block: - - name: Ensure matrix-prometheus-node-exporter is stopped - ansible.builtin.service: - name: matrix-prometheus-node-exporter - state: stopped - enabled: false - daemon_reload: true - - - name: Ensure matrix-prometheus-node-exporter.service doesn't exist - ansible.builtin.file: - path: "{{ devture_systemd_docker_base_systemd_path }}/matrix-prometheus-node-exporter.service" - state: absent diff --git a/roles/custom/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 b/roles/custom/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 deleted file mode 100644 index 82aca453f..000000000 --- a/roles/custom/matrix-prometheus-node-exporter/templates/systemd/matrix-prometheus-node-exporter.service.j2 +++ /dev/null @@ -1,44 +0,0 @@ -#jinja2: lstrip_blocks: "True" -[Unit] -Description=matrix-prometheus-node-exporter -{% for service in matrix_prometheus_node_exporter_systemd_required_services_list %} -Requires={{ service }} -After={{ service }} -{% endfor %} -{% for service in matrix_prometheus_node_exporter_systemd_wanted_services_list %} -Wants={{ service }} -{% endfor %} -DefaultDependencies=no - -[Service] -Type=simple -Environment="HOME={{ devture_systemd_docker_base_systemd_unit_home_path }}" -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-prometheus-node-exporter 2>/dev/null || true' -ExecStartPre=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-prometheus-node-exporter 2>/dev/null || true' - - -ExecStart={{ devture_systemd_docker_base_host_command_docker }} run --rm --name matrix-prometheus-node-exporter \ - --log-driver=none \ - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ - --cap-drop=ALL \ - --read-only \ - {% for arg in matrix_prometheus_node_exporter_container_extra_arguments %} - {{ arg }} \ - {% endfor %} - --network={{ matrix_docker_network }} \ - {% if matrix_prometheus_node_exporter_container_http_host_bind_port %} - -p {{ matrix_prometheus_node_exporter_container_http_host_bind_port }}:9100 \ - {% endif %} - --pid=host \ - --mount type=bind,src=/,dst=/host,ro,bind-propagation=rslave \ - {{ matrix_prometheus_node_exporter_docker_image }} \ - --path.rootfs=/host {{ matrix_prometheus_node_exporter_process_extra_arguments|join(' ') }} - -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} kill matrix-prometheus-node-exporter 2>/dev/null || true' -ExecStop=-{{ devture_systemd_docker_base_host_command_sh }} -c '{{ devture_systemd_docker_base_host_command_docker }} rm matrix-prometheus-node-exporter 2>/dev/null || true' -Restart=always -RestartSec=30 -SyslogIdentifier=matrix-prometheus-node-exporter - -[Install] -WantedBy=multi-user.target diff --git a/roles/custom/matrix-prometheus-node-exporter/vars/main.yml b/roles/custom/matrix-prometheus-node-exporter/vars/main.yml deleted file mode 100644 index 952dc2050..000000000 --- a/roles/custom/matrix-prometheus-node-exporter/vars/main.yml +++ /dev/null @@ -1,5 +0,0 @@ ---- - -# `matrix_prometheus_node_exporter_container_http_host_bind_port_number_raw` contains the raw port number extracted from `matrix_prometheus_node_exporter_container_http_host_bind_port`, -# which can contain values like this: ('1234', '127.0.0.1:1234', '0.0.0.0:1234') -matrix_prometheus_node_exporter_container_http_host_bind_port_number_raw: "{{ '' if matrix_prometheus_node_exporter_container_http_host_bind_port == '' else (matrix_prometheus_node_exporter_container_http_host_bind_port.split(':')[1] if ':' in matrix_prometheus_node_exporter_container_http_host_bind_port else matrix_prometheus_node_exporter_container_http_host_bind_port) }}" diff --git a/roles/custom/matrix-prometheus-services-proxy-connect/defaults/main.yml b/roles/custom/matrix-prometheus-services-proxy-connect/defaults/main.yml new file mode 100644 index 000000000..5799d0535 --- /dev/null +++ b/roles/custom/matrix-prometheus-services-proxy-connect/defaults/main.yml @@ -0,0 +1,15 @@ +--- +# matrix-prometheus-services-proxy-connect is a role which helps integrate +# the various Prometheus roles, which may live outside this Matrix playbook, into it. + + +# Controls whether node-exporter metrics should be proxied (exposed) on `matrix.DOMAIN/metrics/node-exporter`. +# This will only work take effect if `matrix_nginx_proxy_proxy_matrix_metrics_enabled: true`. +# See the `matrix-nginx-proxy` role for details about enabling `matrix_nginx_proxy_proxy_matrix_metrics_enabled`. +matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled: false + +# If you are supplying your own NGINX proxy but want to use the provided exporters you will have to supply an ":" value for the containers to bind to on your host. +# If prometheus_node_exporter_container_http_host_bind_port is set to just a port number, this will default to "127.0.0.1:" +# If prometheus_node_exporter_container_http_host_bind_port is set to an IP that is not 0.0.0.0 and a port, that ":" value will be used +# Otherwise this value will be empty and you will have to manually configure your NGINX config file. (If you are using the config files generated by this playbook, you will have to edit matrix-domain.conf) +matrix_prometheus_services_proxy_connect_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host: "{{ '127.0.0.1' + prometheus_node_exporter_container_http_host_bind_port_number_raw if not ':' in prometheus_node_exporter_container_http_host_bind_port else (prometheus_node_exporter_container_http_host_bind_port if prometheus_node_exporter_container_http_host_bind_port.split(':')[0] != '0.0.0.0' else '') }}" diff --git a/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml new file mode 100644 index 000000000..7f35a318d --- /dev/null +++ b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/main.yml @@ -0,0 +1,19 @@ +--- + +- name: Fail if matrix-nginx-proxy role executed before matrix-prometheus-services-proxy-connect + ansible.builtin.fail: + msg: >- + Trying to append node-exporter's reverse-proxying configuration to matrix-nginx-proxy, + but it's pointless since the matrix-nginx-proxy role had already executed. + To fix this, please change the order of roles in your playbook, + so that the matrix-nginx-proxy role would run after the matrix-prometheus-services-proxy-connect role. + when: matrix_nginx_proxy_role_executed | default(False) | bool + +- block: + - when: prometheus_node_exporter_enabled | bool and matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled | bool + ansible.builtin.include_tasks: "{{ role_path }}/tasks/prometheus-node-exporter/inject_into_nginx_proxy.yml" + tags: + - setup-all + - setup-nginx-proxy + - install-all + - install-nginx-proxy diff --git a/roles/custom/matrix-prometheus-node-exporter/tasks/inject_into_nginx_proxy.yml b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/prometheus-node-exporter/inject_into_nginx_proxy.yml similarity index 51% rename from roles/custom/matrix-prometheus-node-exporter/tasks/inject_into_nginx_proxy.yml rename to roles/custom/matrix-prometheus-services-proxy-connect/tasks/prometheus-node-exporter/inject_into_nginx_proxy.yml index 24e05ed8d..4c86e8aab 100644 --- a/roles/custom/matrix-prometheus-node-exporter/tasks/inject_into_nginx_proxy.yml +++ b/roles/custom/matrix-prometheus-services-proxy-connect/tasks/prometheus-node-exporter/inject_into_nginx_proxy.yml @@ -1,25 +1,16 @@ --- -- name: Fail if matrix-nginx-proxy role already executed - ansible.builtin.fail: - msg: >- - Trying to append node-exporter's reverse-proxying configuration to matrix-nginx-proxy, - but it's pointless since the matrix-nginx-proxy role had already executed. - To fix this, please change the order of roles in your playbook, - so that the matrix-nginx-proxy role would run after the matrix-prometheus-node-exporter role. - when: matrix_nginx_proxy_role_executed | default(False) | bool - - name: Generate node-exporter metrics proxying configuration for matrix-nginx-proxy (matrix.DOMAIN/metrics/node-exporter) ansible.builtin.set_fact: - matrix_prometheus_node_exporter_nginx_metrics_configuration_block: | + matrix_prometheus_services_proxy_connect_node_exporter_nginx_metrics_configuration_block: | location /metrics/node-exporter { {% if matrix_nginx_proxy_enabled | default(False) %} {# Use the embedded DNS resolver in Docker containers to discover the service #} resolver 127.0.0.11 valid=5s; - set $backend "matrix-prometheus-node-exporter:9100"; + set $backend "{{ prometheus_node_exporter_identifier }}:9100"; proxy_pass http://$backend/metrics; - {% elif matrix_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host != '' %} - proxy_pass http://{{ matrix_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host }}/metrics; + {% elif matrix_prometheus_services_proxy_connect_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host != '' %} + proxy_pass http://{{ matrix_prometheus_services_proxy_connect_prometheus_node_exporter_matrix_nginx_proxy_not_enabled_proxy_pass_host }}/metrics; {% else %} return 404 "matrix-nginx-proxy is disabled and no host port was bound to the container, so metrics are unavailable"; {% endif %} @@ -31,5 +22,5 @@ {{ matrix_nginx_proxy_proxy_matrix_metrics_additional_system_location_configuration_blocks | default([]) + - [matrix_prometheus_node_exporter_nginx_metrics_configuration_block] + [matrix_prometheus_services_proxy_connect_node_exporter_nginx_metrics_configuration_block] }} diff --git a/roles/custom/matrix_playbook_migration/defaults/main.yml b/roles/custom/matrix_playbook_migration/defaults/main.yml new file mode 100644 index 000000000..b553160a9 --- /dev/null +++ b/roles/custom/matrix_playbook_migration/defaults/main.yml @@ -0,0 +1,4 @@ +--- + +# Controls if (`matrix_prometheus_node_exporter` -> `prometheus_node_exporter`) validation will run. +matrix_playbook_migration_matrix_prometheus_node_exporter_migration_validation_enabled: true diff --git a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml index f4ce2bdca..5d22e4bfe 100644 --- a/roles/custom/matrix_playbook_migration/tasks/validate_config.yml +++ b/roles/custom/matrix_playbook_migration/tasks/validate_config.yml @@ -34,6 +34,7 @@ - {'old': 'matrix_systemd_services_list', 'new': 'devture_systemd_service_manager_services_list_additional'} - {'old': 'matrix_common_after_systemd_service_start_wait_for_timeout_seconds', 'new': 'devture_systemd_service_manager_up_verification_delay_seconds'} - {'old': 'matrix_systemd_services_autostart_enabled', 'new': 'devture_systemd_service_manager_services_autostart_enabled'} + - {'old': 'matrix_prometheus_node_exporter_metrics_proxying_enabled', 'new': 'matrix_prometheus_services_proxy_connect_prometheus_node_exporter_metrics_proxying_enabled'} - name: (Deprecation) Catch and report matrix_postgres variables ansible.builtin.fail: @@ -48,3 +49,18 @@ The following variables in your configuration need to be renamed: {{ vars | dict2items | selectattr('key', 'match', 'matrix_postgres_.*') | map (attribute='key') | join(', ') }} when: "vars | dict2items | selectattr('key', 'match', 'matrix_postgres_.*') | list | items2dict" + +- when: matrix_playbook_migration_matrix_prometheus_node_exporter_migration_validation_enabled | bool + block: + - ansible.builtin.set_fact: + matrix_playbook_migration_matrix_prometheus_node_exporter_migration_vars: |- + {{ vars | dict2items | selectattr('key', 'match', 'matrix_prometheus_node_exporter_.*') | list | items2dict }} + + - name: (Deprecation) Catch and report matrix_prometheus_node_exporter variables + ansible.builtin.fail: + msg: >- + The matrix-prometheus-node-exporter role that used to be part of this playbook has been replaced by https://gitlab.com/etke.cc/roles/prometheus_node_exporter. + The new role is compatible with the old one, but uses different names for its variables. + Please change your configuration (vars.yml) to rename all variables (`matrix_prometheus_node_exporter_` -> `prometheus_node_exporter_`). + We found usage of the following variables: {{ matrix_playbook_migration_matrix_prometheus_node_exporter_migration_vars.keys() | join(', ') }} + when: "matrix_playbook_migration_matrix_prometheus_node_exporter_migration_vars | length > 0"