merge upstream

README.md

@@ -130,7 +130,7 @@ When updating the playbook, refer to [the changelog](CHANGELOG.md) to catch up w
 
 - Matrix room: [#matrix-docker-ansible-deploy:devture.com](https://matrix.to/#/#matrix-docker-ansible-deploy:devture.com)
 
-- IRC channel: `#matrix-docker-ansible-deploy` on the [Freenode](https://freenode.net/) IRC network (irc.freenode.net)
+- IRC channel: `#matrix-docker-ansible-deploy` on the [Libera Chat](https://libera.chat/) IRC network (irc.libera.chat:6697)
 
 - GitHub issues: [spantaleev/matrix-docker-ansible-deploy/issues](https://github.com/spantaleev/matrix-docker-ansible-deploy/issues)
 

docs/configuring-playbook-bot-mjolnir.md

@@ -4,8 +4,6 @@ The playbook can install and configure the [Mjolnir](https://github.com/matrix-o
 
 See the project's [documentation](https://github.com/matrix-org/mjolnir) to learn what it does and why it might be useful to you.
 
-Note: the playbook does not currently support the Mjolnir Synapse module. The playbook does support another antispam module, see [Setting up Synapse Simple Antispam](configuring-playbook-synapse-simple-antispam.md).
-
 
 ## 1. Register the bot account
 
@@ -90,8 +88,21 @@ matrix_bot_mjolnir_access_token: "ACCESS_TOKEN_FROM_STEP_2_GOES_HERE"
 matrix_bot_mjolnir_management_room: "ROOM_ID_FROM_STEP_4_GOES_HERE"
 ```
 
+## 6. Adding mjolnir synapse antispam module (optional)
 
-## 6. Installing
+Add the following configuration to your `inventory/host_vars/matrix.DOMAIN/vars.yml` file (adapt to your needs):
+
+
+```yaml
+matrix_synapse_ext_spam_checker_mjolnir_antispam_enabled: true
+matrix_synapse_ext_spam_checker_mjolnir_antispam_config_block_invites: true
+matrix_synapse_ext_spam_checker_mjolnir_antispam_config_block_messages: false
+matrix_synapse_ext_spam_checker_mjolnir_antispam_config_block_usernames: false
+matrix_synapse_ext_spam_checker_mjolnir_antispam_config_ban_lists: []
+```
+
+
+## 7. Installing
 
 After configuring the playbook, run the [installation](installing.md) command:
 

docs/configuring-playbook-prometheus-postgres.md

@@ -0,0 +1,29 @@
+# Enabling metrics and graphs for Postgres (optional)
+
+Expanding on the metrics exposed by the [synapse exporter and the node exporter](configuring-playbook-prometheus-grafana.md), the playbook enables the [postgres exporter](https://github.com/prometheus-community/postgres_exporter) that exposes more detailed information about what's happening on your postgres database.
+
+You can enable this with the following settings in your configuration file (`inventory/host_vars/matrix.<your-domain>/vars.yml`):
+
+
+```yaml
+matrix_prometheus_postgres_exporter_enabled: true
+
+# the role creates a postgres user as credential. You can configure these if required:
+matrix_prometheus_postgres_exporter_database_username: 'matrix_prometheus_postgres_exporter'
+matrix_prometheus_postgres_exporter_database_password: 'some-password'
+
+```
+
+## What does it do?
+
+Name | Description
+-----|----------
+`matrix_prometheus_postgres_exporter_enabled`|Enable the postgres prometheus exporter. This sets up the docker container, connects it to the database and adds a 'job' to the prometheus config which tells prometheus about this new exporter. The default is 'false'
+`matrix_prometheus_postgres_exporter_database_username`| The 'username' for the user that the exporter uses to connect to the database. The default is 'matrix_prometheus_postgres_exporter'
+`matrix_prometheus_postgres_exporter_database_password`| The 'password' for the user that the exporter uses to connect to the database.
+
+
+## More information
+
+- [The PostgresSQL dashboard](https://grafana.com/grafana/dashboards/9628) (generic postgres dashboard)
+

docs/maintenance-migrating.md

@@ -1,3 +1,7 @@
+> **Note**: This migration guide is applicable if you migrate from one server to another server having the same CPU architecture (e.g. both servers being `amd64`). 
+> 
+> If you're trying to migrate between different architectures (e.g. `amd64` --> `arm64`), simply copying the complete `/matrix` directory is not possible as it would move the raw PostgreSQL data between different architectures. In this specific case, you can use the guide below as a reference, but you would also need to dump the database on your current server and import it properly on the new server. See our [Backing up PostgreSQL](maintenance-postgres.md#backing-up-postgresql) docs for help with PostgreSQL backup/restore.
+
 # Migrating to new server
 
 1. Prepare by lowering DNS TTL for your domains (`matrix.DOMAIN`, etc.), so that DNS record changes (step 4 below) would happen faster, leading to less downtime

docs/maintenance-postgres.md

@@ -99,6 +99,8 @@ Example: `--extra-vars="postgres_dump_name=matrix-postgres-dump.sql"`
 
 PostgreSQL can be tuned to make it run faster. This is done by passing extra arguments to Postgres with the `matrix_postgres_process_extra_arguments` variable. You should use a website like https://pgtune.leopard.in.ua/ or information from https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server to determine what Postgres settings you should change.
 
+**Note**: the configuration generator at https://pgtune.leopard.in.ua/ adds spaces around the `=` sign, which is invalid. You'll need to remove it manually (`max_connections = 300` -> `max_connections=300`)
+
 ### Here are some examples:
 
 These are not recommended values and they may not work well for you. This is just to give you an idea of some of the options that can be set. If you are an experienced PostgreSQL admin feel free to update this documentation with better examples.
@@ -106,11 +108,33 @@ These are not recommended values and they may not work well for you. This is jus
 Here is an example config for a small 2 core server with 4GB of RAM and SSD storage:
 ```
 matrix_postgres_process_extra_arguments: [
-  "-c 'shared_buffers=128MB'",
-  "-c 'effective_cache_size=2304MB'",
-  "-c 'effective_io_concurrency=100'",
-  "-c 'random_page_cost=2.0'",
-  "-c 'min_wal_size=500MB'",
+  "-c shared_buffers=128MB",
+  "-c effective_cache_size=2304MB",
+  "-c effective_io_concurrency=100",
+  "-c random_page_cost=2.0",
+  "-c min_wal_size=500MB",
+]
+```
+
+Here is an example config for a 4 core server with 8GB of RAM on a Virtual Private Server (VPS); the paramters have been configured using https://pgtune.leopard.in.ua with the following setup: PostgreSQL version 12, OS Type: Linux, DB Type: Mixed type of application, Data Storage: SSD storage:
+```
+matrix_postgres_process_extra_arguments: [
+  "-c max_connections=100",
+  "-c shared_buffers=2GB",
+  "-c effective_cache_size=6GB",
+  "-c maintenance_work_mem=512MB",
+  "-c checkpoint_completion_target=0.9",
+  "-c wal_buffers=16MB",
+  "-c default_statistics_target=100",
+  "-c random_page_cost=1.1",
+  "-c effective_io_concurrency=200",
+  "-c work_mem=5242kB",
+  "-c min_wal_size=1GB",
+  "-c max_wal_size=4GB",
+  "-c max_worker_processes=4",
+  "-c max_parallel_workers_per_gather=2",
+  "-c max_parallel_workers=4",
+  "-c max_parallel_maintenance_workers=2",
 ]
 ```
 

group_vars/matrix_servers

@@ -1466,6 +1466,13 @@ matrix_postgres_additional_databases: |
       'username': matrix_sygnal_database_username,
       'password': matrix_sygnal_database_password,
     }] if (matrix_sygnal_enabled and matrix_sygnal_database_engine == 'postgres' and matrix_sygnal_database_hostname == 'matrix-postgres') else [])
+     +
+    ([{
+      'name': matrix_prometheus_postgres_exporter_database_name,
+      'username': matrix_prometheus_postgres_exporter_database_username,
+      'password': matrix_prometheus_postgres_exporter_database_password,
+    }] if (matrix_prometheus_postgres_exporter_enabled and matrix_prometheus_postgres_exporter_database_hostname == 'matrix-postgres') else [])
+   
    }}
 
 matrix_postgres_import_roles_to_ignore: |
@@ -1766,6 +1773,10 @@ matrix_prometheus_scraper_synapse_rules_synapse_tag: "{{ matrix_synapse_docker_i
 matrix_prometheus_scraper_node_enabled: "{{ matrix_prometheus_node_exporter_enabled }}"
 matrix_prometheus_scraper_node_targets: "{{ ['matrix-prometheus-node-exporter:9100'] if matrix_prometheus_node_exporter_enabled else [] }}"
 
+matrix_prometheus_scraper_postgres_enabled: "{{ matrix_prometheus_postgres_exporter_enabled }}"
+matrix_prometheus_scraper_postgres_targets: "{{ ['matrix-prometheus-postgres-exporter:'+ matrix_prometheus_postgres_exporter_port|string] if matrix_prometheus_scraper_postgres_enabled else [] }}"
+
+
 ######################################################################
 #
 # /matrix-prometheus
@@ -1773,6 +1784,27 @@ matrix_prometheus_scraper_node_targets: "{{ ['matrix-prometheus-node-exporter:91
 ######################################################################
 
 
+######################################################################
+#
+# matrix-prometheus-postgres-exporter
+#
+######################################################################
+
+matrix_prometheus_postgres_exporter_enabled: false
+matrix_prometheus_postgres_exporter_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'prometheus.pg.db') | to_uuid }}"
+
+matrix_prometheus_postgres_exporter_systemd_required_services_list: |
+  {{
+    ['docker.service']
+    +
+    (['matrix-postgres.service'] if matrix_postgres_enabled else [])
+  }}
+
+######################################################################
+#
+# /matrix-prometheus-postgres-exporter
+#
+######################################################################
 
 ######################################################################
 #
@@ -1787,6 +1819,14 @@ matrix_grafana_enabled: false
 # Grafana's HTTP port to the local host.
 matrix_grafana_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else '127.0.0.1:3000' }}"
 
+matrix_grafana_dashboard_download_urls_all: |
+  {{
+    matrix_grafana_dashboard_download_urls
+    +
+    (matrix_prometheus_postgres_exporter_dashboard_urls if matrix_prometheus_postgres_exporter_enabled else [])
+  }}
+
+
 ######################################################################
 #
 # /matrix-grafana

roles/matrix-awx/surveys/configure_element.json.j2

@@ -61,18 +61,6 @@
       "new_question": true,
       "variable": "matrix_client_element_registration_enabled",
       "type": "multiplechoice"
-    },
-    {
-      "question_name": "Set Element Subdomain",
-      "question_description": "Sets the subdomain of the Element web-client, you should only specify the subdomain, not the base domain you've already set. (Eg: 'element' for element.example.org) Note that if you change this value you'll need to reconfigure your DNS.",
-      "required": false,
-      "min": 0,
-      "max": 2048,
-      "default": "{{ element_subdomain }}",
-      "choices": "",
-      "new_question": true,
-      "variable": "element_subdomain",
-      "type": "text"
     }
   ]
 }

roles/matrix-awx/surveys/configure_element_subdomain.json.j2

@@ -0,0 +1,18 @@
+{
+  "name": "Configure Element Subdomain",
+  "description": "Configure Element clients subdomain location. (Eg: 'element' for element.example.org)",
+  "spec": [
+    {
+      "question_name": "Set Element Subdomain",
+      "question_description": "Sets the subdomain of the Element web-client, you should only specify the subdomain, not the base domain you've already set. (Eg: 'element' for element.example.org) Note that if you change this value you'll need to reconfigure your DNS.",
+      "required": false,
+      "min": 0,
+      "max": 2048,
+      "default": "{{ element_subdomain }}",
+      "choices": "",
+      "new_question": true,
+      "variable": "element_subdomain",
+      "type": "text"
+    }
+  ]
+}

roles/matrix-awx/surveys/configure_synapse.json.j2

@@ -107,7 +107,7 @@
       "default": "",
       "choices": "",
       "new_question": true,
-      "variable": "matrix_synapse_registration_shared_secret",
+      "variable": "ext_matrix_synapse_registration_shared_secret",
       "type": "password"
     },
     {

roles/matrix-awx/tasks/main.yml

@@ -98,6 +98,15 @@
   tags:
     - setup-client-element
 
+# Additional playbook to set the variable file during Element configuration
+- include_tasks: 
+    file: "set_variables_element_subdomain.yml"
+    apply:
+      tags: setup-client-element-subdomain
+  when: run_setup|bool and matrix_awx_enabled|bool
+  tags:
+    - setup-client-element-subdomain
+
 # Additional playbook to set the variable file during Synapse configuration
 - include_tasks: 
     file: "set_variables_synapse.yml"

roles/matrix-awx/tasks/set_variables_element.yml

@@ -12,7 +12,6 @@
     'matrix_client_element_brand': '{{ matrix_client_element_brand }}'
     'matrix_client_element_default_theme': '{{ matrix_client_element_default_theme }}'
     'matrix_client_element_registration_enabled': '{{ matrix_client_element_registration_enabled }}'
-    'matrix_server_fqn_element': "{{ element_subdomain }}.{{ matrix_domain }}"
 
 - name: Set fact for 'https' string
   set_fact:

roles/matrix-awx/tasks/set_variables_element_subdomain.yml

@@ -0,0 +1,49 @@
+
+- name: Record Element-Web variables locally on AWX
+  delegate_to: 127.0.0.1
+  lineinfile:
+    path: '{{ awx_cached_matrix_vars }}'
+    regexp: "^#? *{{ item.key | regex_escape() }}:"
+    line: "{{ item.key }}: {{ item.value }}"
+    insertafter: '# Element Settings Start'
+  with_dict:
+    'matrix_server_fqn_element': "{{ element_subdomain }}.{{ matrix_domain }}"
+    
+- name: Save new 'Configure Element Subdomain' survey.json to the AWX tower, template
+  delegate_to: 127.0.0.1
+  template:
+    src: 'roles/matrix-awx/surveys/configure_element_subdomain.json.j2'
+    dest: '/var/lib/awx/projects/clients/{{ member_id }}/{{ subscription_id }}/configure_element_subdomain.json'
+
+- name: Copy new 'Configure Element Subdomain' survey.json to target machine
+  copy:
+    src: '/var/lib/awx/projects/clients/{{ member_id }}/{{ subscription_id }}/configure_element_subdomain.json'
+    dest: '/matrix/awx/configure_element_subdomain.json'
+    mode: '0660'
+
+- name: Collect AWX admin token the hard way!
+  delegate_to: 127.0.0.1
+  shell: |
+      curl -sku {{ tower_username }}:{{ tower_password }} -H "Content-Type: application/json" -X POST -d '{"description":"Tower CLI", "application":null, "scope":"write"}' https://{{ tower_host }}/api/v2/users/1/personal_tokens/ | jq '.token' | sed -r 's/\"//g'
+  register: tower_token
+  no_log: True
+
+- name: Recreate 'Configure Element Subdomain' job template
+  delegate_to: 127.0.0.1
+  awx.awx.tower_job_template:
+    name: "{{ matrix_domain }} - 1 - Configure Element Subdomain"
+    description: "Configure Element clients subdomain location. (Eg: 'element' for element.example.org)"
+    extra_vars: "{{ lookup('file', '/var/lib/awx/projects/clients/{{ member_id }}/{{ subscription_id }}/extra_vars.json') }}"
+    job_type: run
+    job_tags: "start,setup-all,setup-client-element-subdomain"
+    inventory: "{{ member_id }}"
+    project: "{{ member_id }} - Matrix Docker Ansible Deploy"
+    playbook: setup.yml
+    credential: "{{ member_id }} - AWX SSH Key"
+    survey_enabled: true
+    survey_spec: "{{ lookup('file', '/var/lib/awx/projects/clients/{{ member_id }}/{{ subscription_id }}/configure_element_subdomain.json') }}"
+    state: present
+    verbosity: 1
+    tower_host: "https://{{ tower_host }}"
+    tower_oauthtoken: "{{ tower_token.stdout }}"
+    validate_certs: yes

roles/matrix-awx/tasks/set_variables_synapse.yml

@@ -59,8 +59,8 @@
     line: "{{ item.key }}: {{ item.value }}"
     insertafter: '# Synapse Settings Start'
   with_dict:
-    'matrix_synapse_registration_shared_secret': '{{ matrix_synapse_registration_shared_secret }}'
-  when: matrix_synapse_registration_shared_secret|length > 0
+    'matrix_synapse_registration_shared_secret': '{{ ext_matrix_synapse_registration_shared_secret }}'
+  when: ext_matrix_synapse_registration_shared_secret|length > 0
 
 - name: Record registations_require_3pid extra variable if true
   delegate_to: 127.0.0.1

roles/matrix-bridge-appservice-irc/defaults/main.yml

@@ -7,7 +7,7 @@ matrix_appservice_irc_container_self_build: false
 matrix_appservice_irc_docker_repo: "https://github.com/matrix-org/matrix-appservice-irc.git"
 matrix_appservice_irc_docker_src_files_path: "{{ matrix_base_data_path }}/appservice-irc/docker-src"
 
-matrix_appservice_irc_version: release-0.25.0
+matrix_appservice_irc_version: release-0.26.0
 matrix_appservice_irc_docker_image: "{{ matrix_container_global_registry_prefix }}matrixdotorg/matrix-appservice-irc:{{ matrix_appservice_irc_version }}"
 matrix_appservice_irc_docker_image_force_pull: "{{ matrix_appservice_irc_docker_image.endswith(':latest') }}"
 

roles/matrix-bridge-appservice-irc/tasks/migrate_nedb_to_postgres.yml

@@ -26,10 +26,16 @@
   become: false
   when: "matrix_postgres_service_start_result.changed|bool"
 
+- name: Check existence of matrix-appservice-irc service
+  stat:
+    path: "{{ matrix_systemd_path }}/matrix-appservice-irc.service"
+  register: matrix_appservice_irc_service_stat
+
 - name: Ensure matrix-appservice-irc is stopped
   service:
     name: matrix-appservice-irc
     state: stopped
+  when: "matrix_appservice_irc_service_stat.stat.exists"
 
 - name: Import appservice-irc NeDB database into Postgres
   command:

roles/matrix-bridge-heisenbridge/templates/systemd/matrix-heisenbridge.service.j2

@@ -22,7 +22,7 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-heisenbridge \
           --cap-drop=ALL \
           --network={{ matrix_docker_network }} \
           {% if matrix_heisenbridge_identd_enabled %}
-          -p 113:113 \
+          -p 113:13113 \
           {% endif %}
           -v {{ matrix_heisenbridge_base_path }}:/config:z \
           {% for arg in matrix_heisenbridge_container_extra_arguments %}
@@ -31,6 +31,7 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-heisenbridge \
           {{ matrix_heisenbridge_docker_image }} \
           {% if matrix_heisenbridge_identd_enabled %}
           --identd \
+          --identd-port 13113 \
           {% endif %}
           {% if matrix_heisenbridge_owner %}
           -o {{ matrix_heisenbridge_owner }} \

roles/matrix-grafana/tasks/setup.yml

@@ -64,7 +64,7 @@
     mode: 0440
     owner: "{{ matrix_user_username }}"
     group: "{{ matrix_user_groupname }}"
-  with_items: "{{ matrix_grafana_dashboard_download_urls }}"
+  with_items: "{{ matrix_grafana_dashboard_download_urls_all }}"
   when: matrix_grafana_enabled|bool
 
 - name: Ensure matrix-grafana.service installed

roles/matrix-nginx-proxy/defaults/main.yml

@@ -1,5 +1,5 @@
 matrix_nginx_proxy_enabled: true
-matrix_nginx_proxy_version: 1.20.0-alpine
+matrix_nginx_proxy_version: 1.21.0-alpine
 
 # We use an official nginx image, which we fix-up to run unprivileged.
 # An alternative would be an `nginxinc/nginx-unprivileged` image, but
@@ -357,6 +357,18 @@ matrix_nginx_proxy_self_check_validate_certificates: true
 # so we default to not following redirects as well.
 matrix_nginx_proxy_self_check_well_known_matrix_client_follow_redirects: none
 
+# For OCSP purposes, we need to define a resolver at the `server{}` level or `http{}` level (we do the latter).
+#
+# Otherwise, we get warnings like this:
+# > [warn] 22#22: no resolver defined to resolve r3.o.lencr.org while requesting certificate status, responder: r3.o.lencr.org, certificate: "/matrix/ssl/config/live/.../fullchain.pem"
+#
+# We point it to the internal Docker resolver, which likely delegates to nameservers defined in `/etc/resolv.conf`.
+#
+# When nginx proxy is disabled, our configuration is likely used by non-containerized nginx, so can't use the internal Docker resolver.
+# Pointing `resolver` to some public DNS server might be an option, but for now we impose DNS servers on people.
+# It might also be that no such warnings occur when not running in a container.
+matrix_nginx_proxy_http_level_resolver: "{{ '127.0.0.11' if matrix_nginx_proxy_enabled else '' }}"
+
 # By default, this playbook automatically retrieves and auto-renews
 # free SSL certificates from Let's Encrypt.
 #

roles/matrix-nginx-proxy/tasks/ssl/setup_ssl_lets_encrypt.yml

@@ -34,7 +34,7 @@
       template:
         src: "{{ role_path }}/templates/usr-local-bin/matrix-ssl-lets-encrypt-certificates-renew.j2"
         dest: "{{ matrix_local_bin_path }}/matrix-ssl-lets-encrypt-certificates-renew"
-        mode: 0750
+        mode: 0755
 
     - name: Ensure SSL renewal systemd units installed
       template:

roles/matrix-nginx-proxy/templates/nginx/conf.d/nginx-http.conf.j2

@@ -4,6 +4,11 @@
 #
 # Thus, we ensure a larger bucket size value is used.
 server_names_hash_bucket_size 64;
+
+{% if matrix_nginx_proxy_http_level_resolver %}
+	resolver {{ matrix_nginx_proxy_http_level_resolver }};
+{% endif %}
+
 {% for configuration_block in matrix_nginx_proxy_proxy_http_additional_server_configuration_blocks %}
 	{{- configuration_block }}
 {% endfor %}

roles/matrix-postgres/tasks/setup_postgres.yml

@@ -77,14 +77,14 @@
   template:
     src: "{{ role_path }}/templates/usr-local-bin/matrix-postgres-cli.j2"
     dest: "{{ matrix_local_bin_path }}/matrix-postgres-cli"
-    mode: 0750
+    mode: 0755
   when: matrix_postgres_enabled|bool
 
 - name: Ensure matrix-change-user-admin-status script created
   template:
     src: "{{ role_path }}/templates/usr-local-bin/matrix-change-user-admin-status.j2"
     dest: "{{ matrix_local_bin_path }}/matrix-change-user-admin-status"
-    mode: 0750
+    mode: 0755
   when: matrix_postgres_enabled|bool
 
 - name: (Migration) Ensure old matrix-make-user-admin script deleted
@@ -97,7 +97,7 @@
   template:
     src: "{{ role_path }}/templates/usr-local-bin/matrix-postgres-update-user-password-hash.j2"
     dest: "{{ matrix_local_bin_path }}/matrix-postgres-update-user-password-hash"
-    mode: 0750
+    mode: 0755
   when: matrix_postgres_enabled|bool
 
 - name: Ensure matrix-postgres.service installed

roles/matrix-prometheus-postgres-exporter/defaults/main.yml

@@ -0,0 +1,49 @@
+# matrix-prometheus-postgres-exporter is an Prometheus exporter for postgres metrics
+# See: https://github.com/prometheus-community/postgres_exporter
+
+matrix_prometheus_postgres_exporter_enabled: false
+
+matrix_prometheus_postgres_exporter_version: v0.9.0
+matrix_prometheus_postgres_exporter_port: 9187
+
+matrix_prometheus_postgres_exporter_docker_image: "quay.io/prometheuscommunity/postgres-exporter:{{ matrix_prometheus_postgres_exporter_version }}"
+matrix_prometheus_postgres_exporter_docker_image_force_pull: "{{ matrix_prometheus_postgres_exporter_docker_image.endswith(':latest') }}"
+
+# A list of extra arguments to pass to the container
+matrix_prometheus_postgres_exporter_container_extra_arguments: ["-e PG_EXPORTER_AUTO_DISCOVER_DATABASES=true",
+ "-e PG_EXPORTER_WEB_LISTEN_ADDRESS=\":{{matrix_prometheus_postgres_exporter_port}}\"",
+ "-e DATA_SOURCE_NAME=\"postgresql://{{matrix_prometheus_postgres_exporter_database_username}}:{{matrix_prometheus_postgres_exporter_database_password}}@{{matrix_prometheus_postgres_exporter_database_hostname}}:5432/{{matrix_prometheus_postgres_exporter_database_name}}?sslmode=disable\"" ]
+
+# List of systemd services that matrix-prometheus-postgres-exporter.service depends on
+matrix_prometheus_postgres_exporter_systemd_required_services_list: ['docker.service']
+
+# List of systemd services that matrix-prometheus-postgres-exporter.service wants
+matrix_prometheus_postgres_exporter_systemd_wanted_services_list: []
+
+# details for connecting to the database
+matrix_prometheus_postgres_exporter_database_username: 'matrix_prometheus_postgres_exporter'
+matrix_prometheus_postgres_exporter_database_password: 'some-password'
+matrix_prometheus_postgres_exporter_database_hostname: 'matrix-postgres'
+matrix_prometheus_postgres_exporter_database_port: 5432
+matrix_prometheus_postgres_exporter_database_name: 'matrix_prometheus_postgres_exporter'
+
+
+# Controls whether the matrix-prometheus container exposes its HTTP port (tcp/9100 in the container).
+#
+# Takes an "<ip>:<port>" value (e.g. "127.0.0.1:9100"), or empty string to not expose.
+#
+# Official recommendations are to run this container with `--net=host`,
+# but we don't do that, since it:
+# - likely exposes the metrics web server way too publicly (before applying https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1008)
+# - or listens on a loopback interface only (--net=host and 127.0.0.1:9100), which is not reachable from another container (like `matrix-prometheus`)
+#
+# Using `--net=host` and binding to Docker's `matrix` bridge network may be a solution to both,
+# but that's trickier to accomplish and won't necessarily work (hasn't been tested).
+#
+# Not using `--net=host` means that our network statistic reports are likely broken (inaccurate),
+# because node-exporter can't see all interfaces, etc.
+# For now, we'll live with that, until someone develops a better solution.
+matrix_prometheus_postgres_exporter_container_http_host_bind_port: ''
+
+matrix_prometheus_postgres_exporter_dashboard_urls:
+- "https://grafana.com/api/dashboards/9628/revisions/7/download"

roles/matrix-prometheus-postgres-exporter/tasks/init.yml

@@ -0,0 +1,5 @@
+- set_fact:
+    matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-prometheus-postgres-exporter.service'] }}"
+  when: matrix_prometheus_postgres_exporter_enabled|bool
+
+

roles/matrix-prometheus-postgres-exporter/tasks/main.yml

@@ -0,0 +1,8 @@
+- import_tasks: "{{ role_path }}/tasks/init.yml"
+  tags:
+    - always
+
+- import_tasks: "{{ role_path }}/tasks/setup.yml"
+  tags:
+    - setup-all
+    - setup-prometheus-postgres-exporter

roles/matrix-prometheus-postgres-exporter/tasks/setup.yml

@@ -0,0 +1,54 @@
+---
+
+#
+# Tasks related to setting up matrix-prometheus-postgres-exporter
+#
+
+- name: Ensure matrix-prometheus-postgres-exporter image is pulled
+  docker_image:
+    name: "{{ matrix_prometheus_postgres_exporter_docker_image }}"
+    source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}"
+    force_source: "{{ matrix_prometheus_postgres_exporter_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}"
+    force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_prometheus_postgres_exporter_docker_image_force_pull }}"
+  when: "matrix_prometheus_postgres_exporter_enabled|bool"
+
+- name: Ensure matrix-prometheus-postgres-exporter.service installed
+  template:
+    src: "{{ role_path }}/templates/systemd/matrix-prometheus-postgres-exporter.service.j2"
+    dest: "{{ matrix_systemd_path }}/matrix-prometheus-postgres-exporter.service"
+    mode: 0644
+  register: matrix_prometheus_postgres_exporter_systemd_service_result
+  when: matrix_prometheus_postgres_exporter_enabled|bool
+
+- name: Ensure systemd reloaded after matrix-prometheus.service installation
+  service:
+    daemon_reload: yes
+  when: "matrix_prometheus_postgres_exporter_enabled|bool and matrix_prometheus_postgres_exporter_systemd_service_result.changed"
+
+#
+# Tasks related to getting rid of matrix-prometheus-postgres-exporter (if it was previously enabled)
+#
+
+- name: Check existence of matrix-prometheus-postgres-exporter service
+  stat:
+    path: "{{ matrix_systemd_path }}/matrix-prometheus-postgres-exporter.service"
+  register: matrix_prometheus_postgres_exporter_service_stat
+
+- name: Ensure matrix-prometheus-postgres-exporter is stopped
+  service:
+    name: matrix-prometheus-postgres-exporter
+    state: stopped
+    daemon_reload: yes
+  register: stopping_result
+  when: "not matrix_prometheus_postgres_exporter_enabled|bool and matrix_prometheus_postgres_exporter_service_stat.stat.exists"
+
+- name: Ensure matrix-prometheus-postgres-exporter.service doesn't exist
+  file:
+    path: "{{ matrix_systemd_path }}/matrix-prometheus-postgres-exporter.service"
+    state: absent
+  when: "not matrix_prometheus_postgres_exporter_enabled|bool and matrix_prometheus_postgres_exporter_service_stat.stat.exists"
+
+- name: Ensure systemd reloaded after matrix-prometheus-postgres-exporter.service removal
+  service:
+    daemon_reload: yes
+  when: "not matrix_prometheus_postgres_exporter_enabled|bool and matrix_prometheus_postgres_exporter_service_stat.stat.exists"

roles/matrix-prometheus-postgres-exporter/templates/systemd/matrix-prometheus-postgres-exporter.service.j2

@@ -0,0 +1,42 @@
+#jinja2: lstrip_blocks: "True"
+[Unit]
+Description=matrix-prometheus-postgres-exporter
+{% for service in matrix_prometheus_postgres_exporter_systemd_required_services_list %}
+Requires={{ service }}
+After={{ service }}
+{% endfor %}
+{% for service in matrix_prometheus_postgres_exporter_systemd_wanted_services_list %}
+Wants={{ service }}
+{% endfor %}
+DefaultDependencies=no
+
+[Service]
+Type=simple
+Environment="HOME={{ matrix_systemd_unit_home_path }}"
+ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-prometheus-postgres-exporter 2>/dev/null'
+ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-prometheus-postgres-exporter 2>/dev/null'
+
+
+ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-prometheus-postgres-exporter \
+			--log-driver=none \
+			--user={{ matrix_user_uid }}:{{ matrix_user_gid }} \
+			--cap-drop=ALL \
+			--read-only \
+			{% for arg in matrix_prometheus_postgres_exporter_container_extra_arguments %}
+			{{ arg }} \
+			{% endfor %}
+			--network={{ matrix_docker_network }} \
+			{% if matrix_prometheus_postgres_exporter_container_http_host_bind_port %}
+			-p {{ matrix_prometheus_postgres_exporter_container_http_host_bind_port }}:{{matrix_prometheus_postgres_exporter_port}} \
+			{% endif %}
+			--pid=host \
+			{{ matrix_prometheus_postgres_exporter_docker_image }} 
+
+ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-prometheus-postgres-exporter 2>/dev/null'
+ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-prometheus-postgres-exporter 2>/dev/null'
+Restart=always
+RestartSec=30
+SyslogIdentifier=matrix-prometheus-postgres-exporter
+
+[Install]
+WantedBy=multi-user.target

roles/matrix-prometheus/defaults/main.yml

@@ -3,7 +3,7 @@
 
 matrix_prometheus_enabled: false
 
-matrix_prometheus_version: v2.27.0
+matrix_prometheus_version: v2.27.1
 matrix_prometheus_docker_image: "{{ matrix_container_global_registry_prefix }}prom/prometheus:{{ matrix_prometheus_version }}"
 matrix_prometheus_docker_image_force_pull: "{{ matrix_prometheus_docker_image.endswith(':latest') }}"
 

roles/matrix-prometheus/templates/prometheus.yml.j2

@@ -38,3 +38,9 @@ scrape_configs:
     static_configs:
       - targets: {{ matrix_prometheus_scraper_node_targets|to_json }}
   {% endif %}
+
+  {% if matrix_prometheus_scraper_postgres_enabled %}
+  - job_name: postgres
+    static_configs:
+      - targets: {{ matrix_prometheus_scraper_postgres_targets|to_json }}
+  {% endif %}

roles/matrix-synapse-admin/defaults/main.yml

@@ -8,7 +8,7 @@ matrix_synapse_admin_container_self_build_repo: "https://github.com/Awesome-Tech
 
 matrix_synapse_admin_docker_src_files_path: "{{ matrix_base_data_path }}/synapse-admin/docker-src"
 
-matrix_synapse_admin_version: latest
+matrix_synapse_admin_version: 0.8.1
 matrix_synapse_admin_docker_image: "{{ matrix_synapse_admin_docker_image_name_prefix }}awesometechnologies/synapse-admin:{{ matrix_synapse_admin_version }}"
 matrix_synapse_admin_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_admin_container_self_build else matrix_container_global_registry_prefix }}"
 matrix_synapse_admin_docker_image_force_pull: "{{ matrix_synapse_admin_docker_image.endswith(':latest') }}"

roles/matrix-synapse/defaults/main.yml

@@ -497,6 +497,8 @@ matrix_synapse_ext_password_provider_ldap_attributes_name: "cn"
 matrix_synapse_ext_password_provider_ldap_bind_dn: ""
 matrix_synapse_ext_password_provider_ldap_bind_password: ""
 matrix_synapse_ext_password_provider_ldap_filter: ""
+matrix_synapse_ext_password_provider_ldap_active_directory: false
+matrix_synapse_ext_password_provider_ldap_default_domain: ""
 
 # Enable this to activate the Synapse Antispam spam-checker module.
 # See: https://github.com/t2bot/synapse-simple-antispam
@@ -505,6 +507,27 @@ matrix_synapse_ext_spam_checker_synapse_simple_antispam_git_repository_url: "htt
 matrix_synapse_ext_spam_checker_synapse_simple_antispam_git_version: "923ca5c85b08f157181721abbae50dd89c31e4b5"
 matrix_synapse_ext_spam_checker_synapse_simple_antispam_config_blocked_homeservers: []
 
+# Enable this to activate the Mjolnir Antispam spam-checker module.
+# See: https://github.com/matrix-org/mjolnir#synapse-module
+matrix_synapse_ext_spam_checker_mjolnir_antispam_enabled: false
+matrix_synapse_ext_spam_checker_mjolnir_antispam_git_repository_url: "https://github.com/matrix-org/mjolnir"
+matrix_synapse_ext_spam_checker_mjolnir_antispam_git_version: "70f353fbbad0af469b1001080dea194d512b2815"
+matrix_synapse_ext_spam_checker_mjolnir_antispam_config_block_invites: true
+# Flag messages sent by servers/users in the ban lists as spam. Currently
+# this means that spammy messages will appear as empty to users. Default
+# false.
+matrix_synapse_ext_spam_checker_mjolnir_antispam_config_block_messages: false
+# Remove users from the user directory search by filtering matrix IDs and
+# display names by the entries in the user ban list. Default false.
+matrix_synapse_ext_spam_checker_mjolnir_antispam_config_block_usernames: false
+# The room IDs of the ban lists to honour. Unlike other parts of Mjolnir,
+# this list cannot be room aliases or permalinks. This server is expected
+# to already be joined to the room - Mjolnir will not automatically join
+# these rooms.
+# ["!roomid:example.org"]
+matrix_synapse_ext_spam_checker_mjolnir_antispam_config_ban_lists: []
+
+
 matrix_s3_media_store_enabled: false
 matrix_s3_media_store_custom_endpoint_enabled: false
 matrix_s3_goofys_docker_image: "ewoutp/goofys:latest"

roles/matrix-synapse/tasks/ext/mjolnir-antispam/setup.yml

@@ -0,0 +1,7 @@
+---
+
+- import_tasks: "{{ role_path }}/tasks/ext/mjolnir-antispam/setup_install.yml"
+  when: matrix_synapse_ext_spam_checker_mjolnir_antispam_enabled|bool
+
+- import_tasks: "{{ role_path }}/tasks/ext/mjolnir-antispam/setup_uninstall.yml"
+  when: "not matrix_synapse_ext_spam_checker_mjolnir_antispam_enabled|bool"

roles/matrix-synapse/tasks/ext/mjolnir-antispam/setup_install.yml

@@ -0,0 +1,52 @@
+---
+
+- name: Ensure git installed (RedHat)
+  yum:
+    name:
+      - git
+    state: present
+    update_cache: no
+  when: "ansible_os_family == 'RedHat'"
+
+- name: Ensure git installed (Debian)
+  apt:
+    name:
+      - git
+    state: present
+    update_cache: no
+  when: "ansible_os_family == 'Debian'"
+
+- name: Ensure git installed (Archlinux)
+  pacman:
+    name:
+      - git
+    state: present
+    update_cache: no
+  when: "ansible_distribution == 'Archlinux'"
+
+- name: Clone mjolnir-antispam git repository
+  git:
+    repo: "{{ matrix_synapse_ext_spam_checker_mjolnir_antispam_git_repository_url }}"
+    version: "{{ matrix_synapse_ext_spam_checker_mjolnir_antispam_git_version }}"
+    dest: "{{ matrix_synapse_ext_path }}/mjolnir"
+  become: true
+  become_user: "{{ matrix_user_username }}"
+
+- set_fact:
+    matrix_synapse_spam_checker: >
+      {{ matrix_synapse_spam_checker }}
+      +
+      [{
+        "module": "mjolnir.AntiSpam",
+        "config": {
+          "block_invites": {{ matrix_synapse_ext_spam_checker_mjolnir_antispam_config_block_invites }},
+          "block_messages": {{ matrix_synapse_ext_spam_checker_mjolnir_antispam_config_block_messages }},
+          "block_usernames": {{ matrix_synapse_ext_spam_checker_mjolnir_antispam_config_block_usernames }},
+          "ban_lists": {{ matrix_synapse_ext_spam_checker_mjolnir_antispam_config_ban_lists }}
+        }
+      }]
+
+    matrix_synapse_container_extra_arguments: >
+      {{ matrix_synapse_container_extra_arguments|default([]) }}
+      +
+      ["--mount type=bind,src={{ matrix_synapse_ext_path }}/mjolnir/synapse_antispam/mjolnir,dst={{ matrix_synapse_in_container_python_packages_path }}/mjolnir,ro"]

roles/matrix-synapse/tasks/ext/mjolnir-antispam/setup_uninstall.yml

@@ -0,0 +1,6 @@
+---
+
+- name: Ensure mjolnir-antispam doesn't exist
+  file:
+    path: "{{ matrix_synapse_ext_path }}/mjolnir"
+    state: absent

roles/matrix-synapse/tasks/ext/setup.yml

@@ -7,3 +7,5 @@
 - import_tasks: "{{ role_path }}/tasks/ext/ldap-auth/setup.yml"
 
 - import_tasks: "{{ role_path }}/tasks/ext/synapse-simple-antispam/setup.yml"
+
+- import_tasks: "{{ role_path }}/tasks/ext/mjolnir-antispam/setup.yml"

roles/matrix-synapse/tasks/synapse/setup_install.yml

@@ -106,4 +106,4 @@
   template:
     src: "{{ role_path }}/templates/synapse/usr-local-bin/matrix-synapse-register-user.j2"
     dest: "{{ matrix_local_bin_path }}/matrix-synapse-register-user"
-    mode: 0750
+    mode: 0755

roles/matrix-synapse/templates/synapse/homeserver.yaml.j2

@@ -2596,6 +2596,8 @@ password_providers:
       uri: {{ matrix_synapse_ext_password_provider_ldap_uri|string|to_json }}
       start_tls: {{ matrix_synapse_ext_password_provider_ldap_start_tls|to_json }}
       base: {{ matrix_synapse_ext_password_provider_ldap_base|string|to_json }}
+      active_directory: {{ matrix_synapse_ext_password_provider_ldap_active_directory|to_json }}
+      default_domain: {{ matrix_synapse_ext_password_provider_ldap_default_domain|string|to_json }}
       attributes:
         uid: {{ matrix_synapse_ext_password_provider_ldap_attributes_uid|string|to_json }}
         mail: {{ matrix_synapse_ext_password_provider_ldap_attributes_mail|string|to_json }}

setup.yml

@@ -54,5 +54,5 @@
     - matrix-coturn
     - matrix-aux
     - matrix-postgres-backup
+    - matrix-prometheus-postgres-exporter
     - matrix-common-after
-