finallycoffee/matrix-docker-ansible-deploy
5
1
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Move roles/matrix* to roles/custom/matrix*

Browse Source 
This paves the way for installing other roles into `roles/galaxy` using `ansible-galaxy`,
similar to how it's done in:

- https://github.com/spantaleev/gitea-docker-ansible-deploy
- https://github.com/spantaleev/nextcloud-docker-ansible-deploy

In the near future, we'll be removing a lot of the shared role code from here
and using upstream roles for it. Some of the core `matrix-*` roles have
already been extracted out into other reusable roles:

- https://github.com/devture/com.devture.ansible.role.postgres
- https://github.com/devture/com.devture.ansible.role.systemd_docker_base
- https://github.com/devture/com.devture.ansible.role.timesync
- https://github.com/devture/com.devture.ansible.role.vars_preserver
- https://github.com/devture/com.devture.ansible.role.playbook_runtime_messages
- https://github.com/devture/com.devture.ansible.role.playbook_help

We just need to migrate to those.
This commit is contained in:
Slavi Pantaleev
2022-11-03 09:11:29 +02:00
parent 6c131138ad
commit 410a915a8a
722 changed files with 148 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files

133
roles/custom/matrix-postgres/defaults/main.yml Normal file
View File

@ -0,0 +1,133 @@
---
# Project source code URL: https://github.com/postgres/postgres
# Controls if the Postgres server managed by the playbook is enabled.
# You can turn it off and use an external Postgres server by setting this to `false`.
# Doing this has various downsides. See `docs/configuring-playbook-external-postgres.md` to learn more.
matrix_postgres_enabled: true
matrix_postgres_connection_hostname: "matrix-postgres"
matrix_postgres_connection_port: 5432
matrix_postgres_connection_username: "matrix"
matrix_postgres_connection_password: ""
matrix_postgres_db_name: "matrix"
matrix_postgres_base_path: "{{ matrix_base_data_path }}/postgres"
matrix_postgres_data_path: "{{ matrix_postgres_base_path }}/data"
# matrix_postgres_systemd_services_to_stop_for_maintenance_list specifies the list of systemd services to stop before vacuuming or upgrading.
# These services will be restarted after the operation completes.
matrix_postgres_systemd_services_to_stop_for_maintenance_list: []
matrix_postgres_architecture: amd64
# matrix_postgres_docker_image_suffix controls whether we use Alpine-based images (`-alpine`) or the normal Debian-based images.
# Alpine-based Postgres images are smaller and we usually prefer them, but they don't work on ARM32 (tested on a Raspberry Pi 3 running Raspbian 10.7).
# On ARM32, `-alpine` images fail with the following error:
# > LOG: startup process (PID 37) was terminated by signal 11: Segmentation fault
matrix_postgres_docker_image_suffix: "{{ '-alpine' if matrix_postgres_architecture in ['amd64', 'arm64'] else '' }}"
matrix_postgres_docker_image_v9: "{{ matrix_container_global_registry_prefix }}postgres:9.6.24{{ matrix_postgres_docker_image_suffix }}"
matrix_postgres_docker_image_v10: "{{ matrix_container_global_registry_prefix }}postgres:10.22{{ matrix_postgres_docker_image_suffix }}"
matrix_postgres_docker_image_v11: "{{ matrix_container_global_registry_prefix }}postgres:11.17{{ matrix_postgres_docker_image_suffix }}"
matrix_postgres_docker_image_v12: "{{ matrix_container_global_registry_prefix }}postgres:12.12{{ matrix_postgres_docker_image_suffix }}"
matrix_postgres_docker_image_v13: "{{ matrix_container_global_registry_prefix }}postgres:13.8{{ matrix_postgres_docker_image_suffix }}"
matrix_postgres_docker_image_v14: "{{ matrix_container_global_registry_prefix }}postgres:14.5{{ matrix_postgres_docker_image_suffix }}"
matrix_postgres_docker_image_v15: "{{ matrix_container_global_registry_prefix }}postgres:15.0{{ matrix_postgres_docker_image_suffix }}"
matrix_postgres_docker_image_latest: "{{ matrix_postgres_docker_image_v15 }}"
# This variable is assigned at runtime. Overriding its value has no effect.
matrix_postgres_docker_image_to_use: '{{ matrix_postgres_docker_image_latest }}'
matrix_postgres_docker_image_force_pull: "{{ matrix_postgres_docker_image_to_use.endswith(':latest') }}"
# A list of extra arguments to pass to the container
matrix_postgres_container_extra_arguments: []
# A list of extra arguments to pass to the postgres process
# e.g. "-c 'max_connections=200'"
matrix_postgres_process_extra_arguments: []
# Controls whether the matrix-postgres container exposes a port (tcp/5432 in the
# container) that can be used to access the database from outside the container (e.g. with psql)
#
# psql postgresql://username:password@localhost:<port>/database_name
#
# Takes an "<ip>:<port>" or "<port>" value (e.g. "127.0.0.1:5432"), or empty string to not expose.
matrix_postgres_container_postgres_bind_port: ""
# A list of additional (databases and their credentials) to create.
#
# Example:
# matrix_postgres_additional_databases:
# - name: matrix_appservice_discord
# username: matrix_appservice_discord
# password: some_password
# - name: matrix_appservice_slack
# username: matrix_appservice_slack
# password: some_password
matrix_postgres_additional_databases: []
# A list of roles/users to avoid creating when importing (or upgrading) the database.
# If a dump file contains the roles and they've also been created beforehand (see `matrix_postgres_additional_databases`),
# importing would fail.
# We either need to not create them or to ignore the `CREATE ROLE` statements in the dump.
matrix_postgres_import_roles_to_ignore: |
{{
(
[matrix_postgres_connection_username]
+
matrix_postgres_additional_databases|map(attribute='username') | list
) | unique
}}
# When importing an existing Postgres database (when restoring a backup) or when doing a Postgres upgrade (which dumps & restores), we'd like to avoid:
# - creating users (`CREATE ROLE ..`)
# - updating passwords for users (`ALTER ROLE matrix WITH SUPERUSER INHERIT NOCREATEROLE NOCREATEDB LOGIN NOREPLICATION NOBYPASSRLS PASSWORD 'md5...`)
#
# Both of these operations are done by the playbook anyway.
# Updating passwords is especially undesirable, because older versions hash passwords using md5 and export them as md5 hashes in the dump file,
# which is unsupported by default by newer Postgres versions (v14+).
# When users are created and passwords are set by the playbook, they end up hashed as `scram-sha-256` on Postgres v14+.
# If an md5-hashed password is restored on top, Postgres v14+ will refuse to authenticate users with it by default.
#
# We also allow for the role name to be quoted, which is rare, but might happen for role names which are special keywords (e.g. `default`).
matrix_postgres_import_roles_ignore_regex: "^(CREATE|ALTER) ROLE \\\"?({{ matrix_postgres_import_roles_to_ignore | join('|') }})\\\"?(;| WITH)" # noqa jinja[spacing]
# A list of databases to avoid creating when importing (or upgrading) the database.
# If a dump file contains the databases and they've also been created beforehand (see `matrix_postgres_additional_databases`),
# importing would fail.
# We either need to not create them or to ignore the `CREATE DATABASE` statements in the dump.
matrix_postgres_import_databases_to_ignore: |
{{
(
[matrix_postgres_db_name]
+
matrix_postgres_additional_databases|map(attribute='name') | list
) | unique
}}
# We also allow for the database name to be quoted, which is rare, but might happen for database names which are special keywords (e.g. `default`).
matrix_postgres_import_databases_ignore_regex: "^CREATE DATABASE \\\"?({{ matrix_postgres_import_databases_to_ignore | join('|') }})\\\"?\\s" # noqa jinja[spacing]
# The number of seconds to wait after starting `matrix-postgres.service`
# and before trying to run queries for creating additional databases/users against it.
#
# For most (subsequent) runs, Postgres would already be running, so no waiting will be happening at all.
#
# On ARM, we wait some more. ARM32 devices are especially known for being slow.
# ARM64 likely don't need such a long delay, but it doesn't hurt too much having it.
matrix_postgres_additional_databases_postgres_start_wait_timeout_seconds: "{{ 45 if matrix_postgres_architecture in ['arm32', 'arm64'] else 15 }}"
matrix_postgres_pgloader_container_image_self_build: false
matrix_postgres_pgloader_container_image_self_build_repo: "https://github.com/illagrenan/pgloader-docker.git"
matrix_postgres_pgloader_container_image_self_build_repo_branch: "v{{ matrix_postgres_pgloader_docker_image_tag }}"
matrix_postgres_pgloader_container_image_self_build_src_path: "{{ matrix_postgres_base_path }}/pgloader-container-src"
# We use illagrenan/pgloader, instead of the more official dimitri/pgloader image,
# because the official one only provides a `latest` tag.
matrix_postgres_pgloader_docker_image: "{{ matrix_postgres_pgloader_docker_image_name_prefix }}illagrenan/pgloader:{{ matrix_postgres_pgloader_docker_image_tag }}"
matrix_postgres_pgloader_docker_image_name_prefix: "{{ 'localhost/' if matrix_postgres_pgloader_container_image_self_build else matrix_container_global_registry_prefix }}"
matrix_postgres_pgloader_docker_image_tag: "3.6.2"
matrix_postgres_pgloader_docker_image_force_pull: "{{ matrix_postgres_pgloader_docker_image.endswith(':latest') }}"

75
roles/custom/matrix-postgres/tasks/detect_existing_postgres_version.yml Normal file
View File

@ -0,0 +1,75 @@
---
# This utility aims to determine if there is some existing Postgres version in use or not.
# If there is, it also tries to detect the Docker image that corresponds to that version.
#
# This utility is intentionally not in `tasks/util`, because if it were, it wouldn't be possible
# to include it in other roles via the import_role module: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/import_role_module.html
- name: Fail detection if expectation fails (Postgres not enabled)
ansible.builtin.fail:
msg: "Trying to detect the version of the built-in Postgres server, but Postgres installation is not enabled (`matrix_postgres_enabled: false`)"
when: not matrix_postgres_enabled
- name: Initialize Postgres version determination variables (default to empty)
ansible.builtin.set_fact:
matrix_postgres_detection_pg_version_path: "{{ matrix_postgres_data_path }}/PG_VERSION"
matrix_postgres_detected_existing: false
matrix_postgres_detected_version: ""
matrix_postgres_detected_version_corresponding_docker_image: ""
- name: Determine existing Postgres version (check PG_VERSION file)
ansible.builtin.stat:
path: "{{ matrix_postgres_detection_pg_version_path }}"
register: result_pg_version_stat
- ansible.builtin.set_fact:
matrix_postgres_detected_existing: true
when: "result_pg_version_stat.stat.exists"
- name: Determine existing Postgres version (read PG_VERSION file)
ansible.builtin.slurp:
src: "{{ matrix_postgres_detection_pg_version_path }}"
register: result_pg_version
when: matrix_postgres_detected_existing | bool
- name: Determine existing Postgres version (make sense of PG_VERSION file)
ansible.builtin.set_fact:
matrix_postgres_detected_version: "{{ result_pg_version['content'] | b64decode | replace('\n', '') }}"
when: matrix_postgres_detected_existing | bool
- name: Determine corresponding Docker image to detected version (assume default of latest)
ansible.builtin.set_fact:
matrix_postgres_detected_version_corresponding_docker_image: "{{ matrix_postgres_docker_image_latest }}"
when: "matrix_postgres_detected_version != ''"
- name: Determine corresponding Docker image to detected version (use 9.x, if detected)
ansible.builtin.set_fact:
matrix_postgres_detected_version_corresponding_docker_image: "{{ matrix_postgres_docker_image_v9 }}"
when: "matrix_postgres_detected_version.startswith('9.')"
- name: Determine corresponding Docker image to detected version (use 10.x, if detected)
ansible.builtin.set_fact:
matrix_postgres_detected_version_corresponding_docker_image: "{{ matrix_postgres_docker_image_v10 }}"
when: "matrix_postgres_detected_version == '10' or matrix_postgres_detected_version.startswith('10.')"
- name: Determine corresponding Docker image to detected version (use 11.x, if detected)
ansible.builtin.set_fact:
matrix_postgres_detected_version_corresponding_docker_image: "{{ matrix_postgres_docker_image_v11 }}"
when: "matrix_postgres_detected_version == '11' or matrix_postgres_detected_version.startswith('11.')"
- name: Determine corresponding Docker image to detected version (use 12.x, if detected)
ansible.builtin.set_fact:
matrix_postgres_detected_version_corresponding_docker_image: "{{ matrix_postgres_docker_image_v12 }}"
when: "matrix_postgres_detected_version == '12' or matrix_postgres_detected_version.startswith('12.')"
- name: Determine corresponding Docker image to detected version (use 13.x, if detected)
ansible.builtin.set_fact:
matrix_postgres_detected_version_corresponding_docker_image: "{{ matrix_postgres_docker_image_v13 }}"
when: "matrix_postgres_detected_version == '13' or matrix_postgres_detected_version.startswith('13.')"
- name: Determine corresponding Docker image to detected version (use 14.x, if detected)
ansible.builtin.set_fact:
matrix_postgres_detected_version_corresponding_docker_image: "{{ matrix_postgres_docker_image_v14 }}"
when: "matrix_postgres_detected_version == '14' or matrix_postgres_detected_version.startswith('14.')"

102
roles/custom/matrix-postgres/tasks/import_generic_sqlite_db.yml Normal file
View File

@ -0,0 +1,102 @@
---
# Pre-checks
- name: Fail if Postgres not enabled
ansible.builtin.fail:
msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot import."
when: "not matrix_postgres_enabled | bool"
- name: Fail if playbook called incorrectly
ansible.builtin.fail:
msg: "The `sqlite_database_path` variable needs to be provided to this playbook, via --extra-vars"
when: "sqlite_database_path is not defined or sqlite_database_path.startswith('<')"
- name: Check if the provided SQLite database file exists
ansible.builtin.stat:
path: "{{ sqlite_database_path }}"
register: sqlite_database_path_stat_result
- name: Fail if provided SQLite database file doesn't exist
ansible.builtin.fail:
msg: "File cannot be found on the server at {{ sqlite_database_path }}"
when: "not sqlite_database_path_stat_result.stat.exists"
# We either expect `postgres_db_connection_string` specifying a full Postgres database connection string,
# or `postgres_connection_string_variable_name`, specifying a name of a variable, which contains a valid connection string.
- when: 'postgres_connection_string_variable_name is defined'
block:
- name: Fail if postgres_connection_string_variable_name points to an undefined variable
ansible.builtin.fail:
msg: "postgres_connection_string_variable_name is defined, but there is no variable with the name `{{ postgres_connection_string_variable_name }}`"
when: "postgres_connection_string_variable_name not in vars"
- name: Get Postgres connection string from variable
ansible.builtin.set_fact:
postgres_db_connection_string: "{{ lookup('vars', postgres_connection_string_variable_name) }}"
- name: Fail if playbook called incorrectly
ansible.builtin.fail:
msg: >-
Either a `postgres_db_connection_string` variable or a `postgres_connection_string_variable_name` needs to be provided to this playbook, via `--extra-vars`.
Example: `--extra-vars="postgres_db_connection_string=postgresql://username:password@localhost:<port>/database_name"` or `--extra-vars="postgres_connection_string_variable_name=matrix_appservice_discord_database_connString"`
when: "postgres_db_connection_string is not defined or not postgres_db_connection_string.startswith('postgresql://')"
# Defaults
- name: Set postgres_start_wait_time, if not provided
ansible.builtin.set_fact:
postgres_start_wait_time: 15
when: "postgres_start_wait_time | default('') == ''"
# Actual import work
- name: Ensure matrix-postgres is started
ansible.builtin.service:
name: matrix-postgres
state: started
daemon_reload: true
register: matrix_postgres_service_start_result
- name: Wait a bit, so that Postgres can start
ansible.builtin.wait_for:
timeout: "{{ postgres_start_wait_time }}"
delegate_to: 127.0.0.1
become: false
when: "matrix_postgres_service_start_result.changed | bool"
- name: Import SQLite database from {{ sqlite_database_path }} into Postgres # noqa name[template]
ansible.builtin.command:
cmd: >-
{{ matrix_host_command_docker }} run
--rm
--user={{ matrix_user_uid }}:{{ matrix_user_gid }}
--cap-drop=ALL
--network={{ matrix_docker_network }}
--mount type=bind,src={{ sqlite_database_path }},dst=/in.db,ro
--entrypoint=/bin/sh
{{ matrix_postgres_pgloader_docker_image }}
-c
'pgloader /in.db {{ postgres_db_connection_string }}'
register: matrix_postgres_import_generic_sqlite_db_import_result
changed_when: matrix_postgres_import_generic_sqlite_db_import_result.rc == 0
- name: Archive SQLite database ({{ sqlite_database_path }} -> {{ sqlite_database_path }}.backup) # noqa name[template]
ansible.builtin.command:
cmd: "mv {{ sqlite_database_path }} {{ sqlite_database_path }}.backup"
register: matrix_postgres_import_generic_sqlite_db_move_result
changed_when: matrix_postgres_import_generic_sqlite_db_move_result.rc == 0
- name: Inject result
ansible.builtin.set_fact:
matrix_playbook_runtime_results: |
{{
matrix_playbook_runtime_results | default([])
+
[
"NOTE: Your SQLite database file has been imported into Postgres. The original file has been moved from `{{ sqlite_database_path }}` to `{{ sqlite_database_path }}.backup`. When you've confirmed that the import went well and everything works, you should be able to safely delete this file."
]
}}

114
roles/custom/matrix-postgres/tasks/import_postgres.yml Normal file
View File

@ -0,0 +1,114 @@
---
# Pre-checks
- name: Fail if Postgres not enabled
ansible.builtin.fail:
msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot import."
when: "not matrix_postgres_enabled | bool"
- name: Fail if playbook called incorrectly
ansible.builtin.fail:
msg: "The `server_path_postgres_dump` variable needs to be provided to this playbook, via --extra-vars"
when: "server_path_postgres_dump is not defined or server_path_postgres_dump.startswith('<')"
- name: Check if the provided Postgres dump file exists
ansible.builtin.stat:
path: "{{ server_path_postgres_dump }}"
register: result_server_path_postgres_dump_stat
- name: Fail if provided Postgres dump file doesn't exists
ansible.builtin.fail:
msg: "File cannot be found on the server at {{ server_path_postgres_dump }}"
when: "not result_server_path_postgres_dump_stat.stat.exists"
# Defaults
- name: Set postgres_start_wait_time, if not provided
ansible.builtin.set_fact:
postgres_start_wait_time: 15
when: "postgres_start_wait_time | default('') == ''"
- name: Set postgres_import_wait_time, if not provided
ansible.builtin.set_fact:
postgres_import_wait_time: "{{ 7 * 86400 }}"
when: "postgres_import_wait_time | default('') == ''"
# By default, we connect and import into the main (`matrix`) database.
# Single-database dumps for Synapse may wish to import into `synapse` instead.
- name: Set postgres_default_import_database, if not provided
ansible.builtin.set_fact:
postgres_default_import_database: "{{ matrix_postgres_db_name }}"
when: "postgres_default_import_database | default('') == ''"
# Actual import work
- name: Ensure matrix-postgres is started
ansible.builtin.service:
name: matrix-postgres
state: started
daemon_reload: true
register: matrix_postgres_import_start_result
- name: Wait a bit, so that Postgres can start
when: matrix_postgres_import_start_result.changed | bool
ansible.builtin.wait_for:
timeout: "{{ postgres_start_wait_time }}"
delegate_to: 127.0.0.1
become: false
- ansible.builtin.import_tasks: tasks/detect_existing_postgres_version.yml
- name: Abort, if no existing Postgres version detected
ansible.builtin.fail:
msg: "Could not find existing Postgres installation"
when: "not matrix_postgres_detected_existing | bool"
# Starting the database container had automatically created the default
# role (`matrix_postgres_connection_username`) and database (`matrix_postgres_db_name`).
# The dump most likely contains those same entries and would try to re-create them, leading to errors.
# We need to skip over those lines.
- name: Generate Postgres database import command
ansible.builtin.set_fact:
matrix_postgres_import_command: >-
{{ matrix_host_command_docker }} run --rm --name matrix-postgres-import
--log-driver=none
--user={{ matrix_user_uid }}:{{ matrix_user_gid }}
--cap-drop=ALL
--network={{ matrix_docker_network }}
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql
--mount type=bind,src={{ server_path_postgres_dump }},dst=/{{ server_path_postgres_dump | basename }},ro
--entrypoint=/bin/sh
{{ matrix_postgres_docker_image_latest }}
-c "cat /{{ server_path_postgres_dump | basename }} |
{{ 'gunzip |' if server_path_postgres_dump.endswith('.gz') else '' }}
grep -vE '{{ matrix_postgres_import_roles_ignore_regex }}' |
grep -vE '{{ matrix_postgres_import_databases_ignore_regex }}' |
psql -v ON_ERROR_STOP=1 -h matrix-postgres --dbname={{ postgres_default_import_database }}"
tags:
- skip_ansible_lint
# This is a hack.
# See: https://ansibledaily.com/print-to-standard-output-without-escaping/
#
# We want to run `debug: msg=".."`, but that dumps it as JSON and escapes double quotes within it,
# which ruins the command (`matrix_postgres_import_command`)
- name: Note about Postgres importing alternative
ansible.builtin.set_fact:
dummy: true
with_items:
- >-
Importing Postgres database using the following command: `{{ matrix_postgres_import_command }}`.
If this crashes, you can stop Postgres (`systemctl stop matrix-postgres`),
delete its existing data (`rm -rf {{ matrix_postgres_data_path }}/*`), start it again (`systemctl start matrix-postgres`)
and manually run the above import command directly on the server.
- name: Perform Postgres database import
ansible.builtin.command:
cmd: "{{ matrix_postgres_import_command }}"
async: "{{ postgres_import_wait_time }}"
poll: 10
register: matrix_postgres_import_postgres_command_result
failed_when: not matrix_postgres_import_postgres_command_result.finished or matrix_postgres_import_postgres_command_result.rc != 0
changed_when: matrix_postgres_import_postgres_command_result.finished and matrix_postgres_import_postgres_command_result.rc == 0

89
roles/custom/matrix-postgres/tasks/import_synapse_sqlite_db.yml Normal file
View File

@ -0,0 +1,89 @@
---
# Pre-checks
- name: Fail if Postgres not enabled
ansible.builtin.fail:
msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot import."
when: "not matrix_postgres_enabled | bool"
- name: Fail if playbook called incorrectly
ansible.builtin.fail:
msg: "The `server_path_homeserver_db` variable needs to be provided to this playbook, via --extra-vars"
when: "server_path_homeserver_db is not defined or server_path_homeserver_db.startswith('<')"
- name: Check if the provided SQLite homeserver.db file exists
ansible.builtin.stat:
path: "{{ server_path_homeserver_db }}"
register: result_server_path_homeserver_db_stat
- name: Fail if provided SQLite homeserver.db file doesn't exist
ansible.builtin.fail:
msg: "File cannot be found on the server at {{ server_path_homeserver_db }}"
when: "not result_server_path_homeserver_db_stat.stat.exists"
# Defaults
- name: Set postgres_start_wait_time, if not provided
ansible.builtin.set_fact:
postgres_start_wait_time: 15
when: "postgres_start_wait_time | default('') == ''"
# Actual import work
- name: Ensure matrix-postgres is stopped
ansible.builtin.service:
name: matrix-postgres
state: stopped
daemon_reload: true
- name: Ensure postgres data is wiped out
ansible.builtin.file:
path: "{{ matrix_postgres_data_path }}"
state: absent
- name: Ensure postgres data path exists
ansible.builtin.file:
path: "{{ matrix_postgres_data_path }}"
state: directory
mode: 0700
owner: "{{ matrix_user_username }}"
group: "{{ matrix_user_groupname }}"
- name: Ensure matrix-postgres is started
ansible.builtin.service:
name: matrix-postgres
state: restarted
daemon_reload: true
- name: Wait a bit, so that Postgres can start
ansible.builtin.wait_for:
timeout: "{{ postgres_start_wait_time }}"
delegate_to: 127.0.0.1
become: false
# We don't use the `docker_container` module, because using it with `cap_drop` requires
# a very recent version, which is not available for a lot of people yet.
#
# Also, some old `docker_container` versions were buggy and would leave containers behind
# on failure, which we had to work around to allow retries (by re-running the playbook).
- name: Import SQLite database into Postgres
ansible.builtin.command:
cmd: |
docker run
--rm
--name=matrix-synapse-migrate
--log-driver=none
--user={{ matrix_user_uid }}:{{ matrix_user_gid }}
--cap-drop=ALL
--network={{ matrix_docker_network }}
--entrypoint=python
--mount type=bind,src={{ matrix_synapse_config_dir_path }},dst=/data
--mount type=bind,src={{ matrix_synapse_config_dir_path }},dst=/matrix-media-store-parent/media-store
--mount type=bind,src={{ server_path_homeserver_db }},dst=/{{ server_path_homeserver_db | basename }}
{{ matrix_synapse_docker_image_final }}
/usr/local/bin/synapse_port_db --sqlite-database /{{ server_path_homeserver_db | basename }} --postgres-config /data/homeserver.yaml
register: matrix_postgres_import_synapse_sqlite_db_result
changed_when: matrix_postgres_import_synapse_sqlite_db_result.rc == 0

5
roles/custom/matrix-postgres/tasks/init.yml Normal file
View File

@ -0,0 +1,5 @@
---
- ansible.builtin.set_fact:
matrix_systemd_services_list: "{{ matrix_systemd_services_list + ['matrix-postgres.service'] }}"
when: matrix_postgres_enabled | bool

45
roles/custom/matrix-postgres/tasks/main.yml Normal file
View File

@ -0,0 +1,45 @@
---
- ansible.builtin.import_tasks: "{{ role_path }}/tasks/init.yml"
tags:
- always
- ansible.builtin.import_tasks: "{{ role_path }}/tasks/validate_config.yml"
when: "run_setup | bool and matrix_postgres_enabled | bool"
tags:
- setup-all
- setup-postgres
- ansible.builtin.import_tasks: "{{ role_path }}/tasks/setup_postgres.yml"
when: run_setup | bool
tags:
- setup-all
- setup-postgres
- ansible.builtin.import_tasks: "{{ role_path }}/tasks/import_postgres.yml"
when: run_postgres_import | bool
tags:
- import-postgres
# The `run_postgres_import_sqlite_db` variable had better be renamed to be consistent,
# but that's a breaking change which may cause trouble for people.
- ansible.builtin.import_tasks: "{{ role_path }}/tasks/import_synapse_sqlite_db.yml"
when: run_postgres_import_sqlite_db | bool
tags:
- import-synapse-sqlite-db
# Perhaps we need a new variable here, instead of `run_postgres_import_sqlite_db`.
- ansible.builtin.import_tasks: "{{ role_path }}/tasks/import_generic_sqlite_db.yml"
when: run_postgres_import_sqlite_db | bool
tags:
- import-generic-sqlite-db
- ansible.builtin.import_tasks: "{{ role_path }}/tasks/upgrade_postgres.yml"
when: run_postgres_upgrade | bool
tags:
- upgrade-postgres
- ansible.builtin.import_tasks: "{{ role_path }}/tasks/run_vacuum.yml"
when: run_postgres_vacuum | bool
tags:
- run-postgres-vacuum

176
roles/custom/matrix-postgres/tasks/migrate_db_to_postgres.yml Normal file
View File

@ -0,0 +1,176 @@
---
- name: Fail if Postgres not enabled
ansible.builtin.fail:
msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot migrate."
when: "not matrix_postgres_enabled | bool"
- name: Fail if util called incorrectly (missing matrix_postgres_db_migration_request)
ansible.builtin.fail:
msg: "The `matrix_postgres_db_migration_request` variable needs to be provided to this util."
when: "matrix_postgres_db_migration_request is not defined"
- name: Fail if util called incorrectly (invalid matrix_postgres_db_migration_request)
ansible.builtin.fail:
msg: "The `matrix_postgres_db_migration_request` variable needs to contain `{{ item }}`."
with_items:
- src
- dst
- caller
- engine_variable_name
- systemd_services_to_stop
when: "item not in matrix_postgres_db_migration_request"
- name: Check if the provided source database file exists
ansible.builtin.stat:
path: "{{ matrix_postgres_db_migration_request.src }}"
register: matrix_postgres_db_migration_request_src_stat_result
- name: Fail if provided source database file doesn't exist
ansible.builtin.fail:
msg: "File cannot be found on the server at {{ matrix_postgres_db_migration_request.src }}"
when: "not matrix_postgres_db_migration_request_src_stat_result.stat.exists"
- when: "matrix_postgres_pgloader_container_image_self_build | bool"
block:
- name: Ensure pgloader repository is present on self-build
ansible.builtin.git:
repo: "{{ matrix_postgres_pgloader_container_image_self_build_repo }}"
dest: "{{ matrix_postgres_pgloader_container_image_self_build_src_path }}"
version: "{{ matrix_postgres_pgloader_container_image_self_build_repo_branch }}"
force: "yes"
become: true
become_user: "{{ matrix_user_username }}"
register: matrix_postgres_pgloader_git_pull_results
# If `stable` is used, we hit an error when processing /opt/src/pgloader/build/quicklisp/dists/quicklisp/software/uax-15-20201220-git/data/CompositionExclusions.txt:
# > the octet sequence #(194) cannot be decoded
#
# The issue is described here and is not getting fixed for months: https://github.com/dimitri/pgloader/pull/1179
#
# Although we're not using the dimitri/pgloader image, the one we're using suffers from the same problem.
- name: Switch pgloader base image from Debian stable (likely 10.x/Buster) to Bullseye
ansible.builtin.lineinfile:
path: "{{ matrix_postgres_pgloader_container_image_self_build_src_path }}/Dockerfile"
regexp: "{{ item.match }}"
line: "{{ item.replace }}"
with_items:
- match: '^FROM debian:stable-slim as builder$'
replace: 'FROM debian:bullseye-slim as builder'
- match: '^FROM debian:stable-slim$'
replace: 'FROM debian:bullseye-slim'
- name: Ensure pgloader Docker image is built
community.docker.docker_image:
name: "{{ matrix_postgres_pgloader_docker_image }}"
source: build
force_source: "{{ matrix_postgres_pgloader_git_pull_results.changed if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}"
force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_postgres_pgloader_git_pull_results.changed }}"
build:
dockerfile: Dockerfile
path: "{{ matrix_postgres_pgloader_container_image_self_build_src_path }}"
pull: true
- name: Ensure pgloader Docker image is pulled
community.docker.docker_image:
name: "{{ matrix_postgres_pgloader_docker_image }}"
source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}"
force_source: "{{ matrix_postgres_pgloader_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}"
force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_postgres_pgloader_docker_image_force_pull }}"
when: "not matrix_postgres_pgloader_container_image_self_build"
# Defaults
- name: Set postgres_start_wait_time, if not provided
ansible.builtin.set_fact:
postgres_start_wait_time: 15
when: "postgres_start_wait_time | default('') == ''"
# Actual import work
# matrix-postgres is most likely started already
- name: Ensure matrix-postgres is started
ansible.builtin.service:
name: matrix-postgres
state: started
daemon_reload: true
register: matrix_postgres_service_start_result
- name: Wait a bit, so that Postgres can start
ansible.builtin.wait_for:
timeout: "{{ postgres_start_wait_time }}"
delegate_to: 127.0.0.1
become: false
when: "matrix_postgres_service_start_result.changed | bool"
# We only stop services here, leaving it to the caller to start them later.
#
# We can't start them, because they probably need to be reconfigured too (changing the configuration from using SQLite to Postgres, etc.),
# before starting.
#
# Since the caller will be starting them, it might make sense to leave stopping to it as well.
# However, we don't do it, because it's simpler having it here, and it also gets to happen only if we'll be doing an import.
# If we bailed out (somewhere above), nothing would have gotten stopped. It's nice to leave this running in such cases.
- name: Ensure systemd services blocking the database import are stopped
ansible.builtin.service:
name: "{{ item }}"
state: stopped
failed_when: false
with_items: "{{ matrix_postgres_db_migration_request.systemd_services_to_stop }}"
- name: Import {{ matrix_postgres_db_migration_request.engine_old }} database from {{ matrix_postgres_db_migration_request.src }} into Postgres # noqa name[template]
ansible.builtin.command:
cmd: >-
{{ matrix_host_command_docker }} run
--rm
--user={{ matrix_user_uid }}:{{ matrix_user_gid }}
--cap-drop=ALL
--network={{ matrix_docker_network }}
--mount type=bind,src={{ matrix_postgres_db_migration_request.src }},dst=/in.db,ro
--entrypoint=/bin/sh
{{ matrix_postgres_pgloader_docker_image }}
-c
'pgloader {{ matrix_postgres_db_migration_request.pgloader_options | default([]) | join(' ') }} /in.db {{ matrix_postgres_db_migration_request.dst }}'
register: matrix_postgres_migrate_db_to_postgres_import_result
changed_when: matrix_postgres_migrate_db_to_postgres_import_result.rc == 0
- when: "matrix_postgres_db_migration_request.additional_psql_statements_list | default([]) | length > 0"
block:
- ansible.builtin.import_role:
name: custom/matrix-postgres
tasks_from: detect_existing_postgres_version
- ansible.builtin.set_fact:
matrix_postgres_docker_image_to_use: "{{ matrix_postgres_docker_image_latest if matrix_postgres_detected_version_corresponding_docker_image == '' else matrix_postgres_detected_version_corresponding_docker_image }}"
- name: Execute additional Postgres SQL migration statements
ansible.builtin.command:
cmd: >-
{{ matrix_host_command_docker }} run
--rm
--user={{ matrix_user_uid }}:{{ matrix_user_gid }}
--cap-drop=ALL
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql
--network={{ matrix_docker_network }}
{{ matrix_postgres_docker_image_to_use }}
psql --host=matrix-postgres --dbname={{ matrix_postgres_db_migration_request.additional_psql_statements_db_name }} --command='{{ item }}'
with_items: "{{ matrix_postgres_db_migration_request.additional_psql_statements_list }}"
register: matrix_postgres_migrate_db_to_postgres_additional_queries_result
changed_when: matrix_postgres_migrate_db_to_postgres_additional_queries_result.rc == 0
- name: Archive {{ matrix_postgres_db_migration_request.engine_old }} database ({{ matrix_postgres_db_migration_request.src }} -> {{ matrix_postgres_db_migration_request.src }}.backup) # noqa name[template]
ansible.builtin.command:
cmd: "mv {{ matrix_postgres_db_migration_request.src }} {{ matrix_postgres_db_migration_request.src }}.backup"
register: matrix_postgres_migrate_db_to_postgres_move_result
changed_when: matrix_postgres_migrate_db_to_postgres_move_result.rc == 0
- name: Inject result
ansible.builtin.set_fact:
matrix_playbook_runtime_results: |
{{
matrix_playbook_runtime_results | default([])
+
[
"NOTE: Your {{ matrix_postgres_db_migration_request.engine_old }} database file has been imported into Postgres. The original database file has been moved from `{{ matrix_postgres_db_migration_request.src }}` to `{{ matrix_postgres_db_migration_request.src }}.backup`. When you've confirmed that the import went well and everything works, you should be able to safely delete this file."
]
}}

78
roles/custom/matrix-postgres/tasks/migrate_postgres_data_directory.yml Normal file
View File

@ -0,0 +1,78 @@
---
# We used to store Postgres data directly under `/matrix/postgres` (what is now considered `matrix_postgres_base_path`).
#
# From now on, we expect to store Postgres data one directory below now (`/matrix/postgres/data` - `matrix_postgres_data_path`).
# We wish to use the base directory for other purposes (storing environment variable files, etc.).
# Mixing those with the Postgres data is no good and it leads to Postgres's `initdb` complaining to initialize
# a database in a non-empty directory.
#
# For this reason, we store the Postgres data in `/matrix/postgres/data` and need to relocate any installations
# which still store it in the parent directory (`/matrix/postgres`).
#
# This utility is intentionally not in `tasks/util`, because if it were, it wouldn't be possible
# to include it in other roles via the import_role module: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/import_role_module.html
- name: Check if old Postgres data directory is used
ansible.builtin.stat:
path: "{{ matrix_postgres_base_path }}/PG_VERSION"
register: result_pg_old_data_dir_stat
- name: Warn if old Postgres data directory detected
ansible.builtin.debug:
msg: >
Found that you have Postgres data in `{{ matrix_postgres_base_path }}`.
From now on, Postgres data is supposed to be stored in `{{ matrix_postgres_data_path }}` instead.
We'll stop Postgres and relocate the files there for you.
when: "result_pg_old_data_dir_stat.stat.exists"
# We should stop Postgres first, before building a list of files,
# as to ignore any `postmaster.pid` files, etc.
- name: Ensure matrix-postgres is stopped
ansible.builtin.service:
name: matrix-postgres
state: stopped
daemon_reload: true
when: "result_pg_old_data_dir_stat.stat.exists"
- name: Find files and directories in old Postgres data path
ansible.builtin.find:
paths: "{{ matrix_postgres_base_path }}"
file_type: any
excludes: ["data"]
register: "result_pg_old_data_dir_find"
when: "result_pg_old_data_dir_stat.stat.exists"
- name: Ensure new Postgres data path exists
ansible.builtin.file:
path: "{{ matrix_postgres_data_path }}"
state: directory
mode: 0700
owner: "{{ matrix_user_username }}"
group: "{{ matrix_user_groupname }}"
when: "result_pg_old_data_dir_stat.stat.exists"
- when: "result_pg_old_data_dir_stat.stat.exists"
block:
- name: Relocate Postgres data files from old directory to new
ansible.builtin.command:
cmd: "mv {{ item.path }} {{ matrix_postgres_data_path }}/{{ item.path | basename }}"
with_items: "{{ result_pg_old_data_dir_find.files }}"
register: matrix_postgres_migrate_postgres_data_directory_move_result
changed_when: matrix_postgres_migrate_postgres_data_directory_move_result.rc == 0
# Intentionally not starting matrix-postgres here.
# It likely needs to be updated to point to the new directory.
# In fact, let's even get rid of the outdated service, to ensure no one will start it
# and have it initialize a new database.
- name: Ensure outdated matrix-postgres.service doesn't exist
ansible.builtin.file:
path: "{{ matrix_systemd_path }}/matrix-postgres.service"
state: absent
when: "result_pg_old_data_dir_stat.stat.exists"
- name: Ensure systemd reloaded after getting rid of outdated matrix-postgres.service
ansible.builtin.service:
daemon_reload: true
when: "result_pg_old_data_dir_stat.stat.exists"

96
roles/custom/matrix-postgres/tasks/run_vacuum.yml Normal file
View File

@ -0,0 +1,96 @@
---
# Pre-checks
- name: Fail if Postgres not enabled
ansible.builtin.fail:
msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot run vacuum."
when: "not matrix_postgres_enabled | bool"
# Defaults
- name: Set postgres_start_wait_time, if not provided
ansible.builtin.set_fact:
postgres_start_wait_time: 15
when: "postgres_start_wait_time | default('') == ''"
- name: Set postgres_vacuum_wait_time, if not provided
ansible.builtin.set_fact:
postgres_vacuum_wait_time: "{{ 7 * 86400 }}"
when: "postgres_vacuum_wait_time | default('') == ''"
# Actual vacuuming work
- name: Ensure matrix-postgres is started
ansible.builtin.service:
name: matrix-postgres
state: started
daemon_reload: true
register: matrix_postgres_vacuum_start_result
- name: Wait a bit, so that Postgres can start
when: matrix_postgres_vacuum_start_result.changed | bool
ansible.builtin.wait_for:
timeout: "{{ postgres_start_wait_time }}"
delegate_to: 127.0.0.1
become: false
- ansible.builtin.import_tasks: tasks/detect_existing_postgres_version.yml
- name: Abort, if no existing Postgres version detected
ansible.builtin.fail:
msg: "Could not find existing Postgres installation"
when: "not matrix_postgres_detected_existing | bool"
- name: Generate Postgres database vacuum command
ansible.builtin.set_fact:
matrix_postgres_vacuum_command: >-
{{ matrix_host_command_docker }} run --rm --name matrix-postgres-synapse-vacuum
--user={{ matrix_user_uid }}:{{ matrix_user_gid }}
--cap-drop=ALL
--network={{ matrix_docker_network }}
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql
{{ matrix_postgres_docker_image_latest }}
psql -v ON_ERROR_STOP=1 -h matrix-postgres {{ matrix_synapse_database_database }} -c 'VACUUM FULL VERBOSE'
- name: Note about Postgres vacuum alternative
ansible.builtin.debug:
msg: >-
Running vacuum with the following Postgres ansible.builtin.command: `{{ matrix_postgres_vacuum_command }}`.
If this crashes, you can stop all processes (`systemctl stop matrix-*`),
start Postgres only (`systemctl start matrix-postgres`)
and manually run the above command directly on the server.
- name: Populate service facts
ansible.builtin.service_facts:
- ansible.builtin.set_fact:
matrix_postgres_synapse_was_running: "{{ ansible_facts.services['matrix-synapse.service'] | default(none) is not none and ansible_facts.services['matrix-synapse.service'].state == 'running' }}"
- name: Ensure services are stopped
ansible.builtin.service:
name: "{{ item }}"
state: stopped
daemon_reload: true
with_items: "{{ matrix_postgres_systemd_services_to_stop_for_maintenance_list }}"
- name: Run Postgres vacuum command
ansible.builtin.command: "{{ matrix_postgres_vacuum_command }}"
async: "{{ postgres_vacuum_wait_time }}"
poll: 10
register: matrix_postgres_synapse_vacuum_result
failed_when: not matrix_postgres_synapse_vacuum_result.finished or matrix_postgres_synapse_vacuum_result.rc != 0
changed_when: matrix_postgres_synapse_vacuum_result.finished and matrix_postgres_synapse_vacuum_result.rc == 0
# Intentionally show the results
- ansible.builtin.debug:
var: "matrix_postgres_synapse_vacuum_result"
- name: Ensure services are started
ansible.builtin.service:
name: "{{ item }}"
state: started
daemon_reload: true
with_items: "{{ matrix_postgres_systemd_services_to_stop_for_maintenance_list }}"

215
roles/custom/matrix-postgres/tasks/setup_postgres.yml Normal file
View File

@ -0,0 +1,215 @@
---
#
# Tasks related to setting up an internal postgres server
#
- ansible.builtin.import_tasks: "{{ role_path }}/tasks/migrate_postgres_data_directory.yml"
when: matrix_postgres_enabled | bool
- ansible.builtin.import_tasks: "{{ role_path }}/tasks/detect_existing_postgres_version.yml"
when: matrix_postgres_enabled | bool
# If we have found an existing version (installed from before), we use its corresponding Docker image.
# If not, we install using the latest Postgres.
#
# Upgrading is supposed to be performed separately and explicitly (see `upgrade_postgres.yml`).
- ansible.builtin.set_fact:
matrix_postgres_docker_image_to_use: "{{ matrix_postgres_docker_image_latest if matrix_postgres_detected_version_corresponding_docker_image == '' else matrix_postgres_detected_version_corresponding_docker_image }}"
when: matrix_postgres_enabled | bool
- name: Abort if on an unsupported Postgres version
ansible.builtin.fail:
msg: "You're on Postgres {{ matrix_postgres_detected_version }}, which is no longer supported. To upgrade, see docs/maintenance-postgres.md"
when: "matrix_postgres_enabled | bool and matrix_postgres_detected_version.startswith('9.')"
- name: Inject warning if on an old version of Postgres
ansible.builtin.set_fact:
matrix_playbook_runtime_results: |
{{
matrix_playbook_runtime_results | default([])
+
[
"NOTE: Your setup is on an old Postgres version ({{ matrix_postgres_docker_image_to_use }}), while {{ matrix_postgres_docker_image_latest }} is supported. You can upgrade using --tags=upgrade-postgres"
]
}}
when: "matrix_postgres_enabled | bool and matrix_postgres_docker_image_to_use != matrix_postgres_docker_image_latest"
# Even if we don't run the internal server, we still need this for running the CLI
- name: Ensure postgres Docker image is pulled
community.docker.docker_image:
name: "{{ matrix_postgres_docker_image_to_use }}"
source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}"
force_source: "{{ matrix_postgres_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}"
force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_postgres_docker_image_force_pull }}"
when: matrix_postgres_enabled | bool
register: result
retries: "{{ matrix_container_retries_count }}"
delay: "{{ matrix_container_retries_delay }}"
until: result is not failed
- name: Ensure Postgres paths exist
ansible.builtin.file:
path: "{{ item }}"
state: directory
mode: 0700
owner: "{{ matrix_user_username }}"
group: "{{ matrix_user_groupname }}"
with_items:
- "{{ matrix_postgres_base_path }}"
- "{{ matrix_postgres_data_path }}"
when: matrix_postgres_enabled | bool
# We do this as a separate task, because:
# - we'd like to do it for the data path only, not for the base path (which contains root-owned environment variable files we'd like to leave as-is)
# - we need to do it without `mode`, or we risk making certain `.conf` and other files's executable bit to flip to true
- name: Ensure Postgres data path ownership is correct
ansible.builtin.file:
path: "{{ matrix_postgres_data_path }}"
state: directory
owner: "{{ matrix_user_username }}"
group: "{{ matrix_user_groupname }}"
recurse: true
when: matrix_postgres_enabled | bool
- name: Ensure Postgres environment variables file created
ansible.builtin.template:
src: "{{ role_path }}/templates/{{ item }}.j2"
dest: "{{ matrix_postgres_base_path }}/{{ item }}"
owner: "{{ matrix_user_username }}"
group: "{{ matrix_user_groupname }}"
mode: 0640
with_items:
- "env-postgres-psql"
- "env-postgres-server"
when: matrix_postgres_enabled | bool
- name: Ensure matrix-postgres-cli script created
ansible.builtin.template:
src: "{{ role_path }}/templates/usr-local-bin/matrix-postgres-cli.j2"
dest: "{{ matrix_local_bin_path }}/matrix-postgres-cli"
mode: 0755
when: matrix_postgres_enabled | bool
- name: Ensure matrix-postgres-cli-non-interactive script created
ansible.builtin.template:
src: "{{ role_path }}/templates/usr-local-bin/matrix-postgres-cli-non-interactive.j2"
dest: "{{ matrix_local_bin_path }}/matrix-postgres-cli-non-interactive"
mode: 0755
when: matrix_postgres_enabled | bool
- name: Ensure matrix-change-user-admin-status script created
ansible.builtin.template:
src: "{{ role_path }}/templates/usr-local-bin/matrix-change-user-admin-status.j2"
dest: "{{ matrix_local_bin_path }}/matrix-change-user-admin-status"
mode: 0755
when: matrix_postgres_enabled | bool
- name: (Migration) Ensure old matrix-make-user-admin script deleted
ansible.builtin.file:
path: "{{ matrix_local_bin_path }}/matrix-make-user-admin"
state: absent
when: matrix_postgres_enabled | bool
- name: Ensure matrix-postgres-update-user-password-hash script created
ansible.builtin.template:
src: "{{ role_path }}/templates/usr-local-bin/matrix-postgres-update-user-password-hash.j2"
dest: "{{ matrix_local_bin_path }}/matrix-postgres-update-user-password-hash"
mode: 0755
when: matrix_postgres_enabled | bool
- name: Ensure matrix-postgres.service installed
ansible.builtin.template:
src: "{{ role_path }}/templates/systemd/matrix-postgres.service.j2"
dest: "{{ matrix_systemd_path }}/matrix-postgres.service"
mode: 0644
register: matrix_postgres_systemd_service_result
when: matrix_postgres_enabled | bool
- name: Ensure systemd reloaded after matrix-postgres.service installation
ansible.builtin.service:
daemon_reload: true
when: "matrix_postgres_enabled | bool and matrix_postgres_systemd_service_result.changed"
- ansible.builtin.include_tasks:
file: "{{ role_path }}/tasks/util/create_additional_databases.yml"
apply:
tags:
- always
when: "matrix_postgres_enabled | bool and matrix_postgres_additional_databases|length > 0"
- name: Check existence of matrix-postgres backup data path
ansible.builtin.stat:
path: "{{ matrix_postgres_data_path }}-auto-upgrade-backup"
register: matrix_postgres_data_backup_path_stat
when: "matrix_postgres_enabled | bool"
- name: Inject warning if backup data remains
ansible.builtin.set_fact:
matrix_playbook_runtime_results: |
{{
matrix_playbook_runtime_results | default([])
+
[
"NOTE: You have some Postgres backup data in `{{ matrix_postgres_data_path }}-auto-upgrade-backup`, which was created during the last major Postgres update you ran. If your setup works well after this upgrade, feel free to delete this whole directory."
]
}}
when: "matrix_postgres_enabled | bool and matrix_postgres_data_backup_path_stat.stat.exists"
#
# Tasks related to getting rid of the internal postgres server (if it was previously enabled)
#
- name: Check existence of matrix-postgres service
ansible.builtin.stat:
path: "{{ matrix_systemd_path }}/matrix-postgres.service"
register: matrix_postgres_service_stat
when: "not matrix_postgres_enabled | bool"
- name: Ensure matrix-postgres is stopped
ansible.builtin.service:
name: matrix-postgres
state: stopped
daemon_reload: true
when: "not matrix_postgres_enabled | bool and matrix_postgres_service_stat.stat.exists"
- name: Ensure matrix-postgres.service doesn't exist
ansible.builtin.file:
path: "{{ matrix_systemd_path }}/matrix-postgres.service"
state: absent
when: "not matrix_postgres_enabled | bool and matrix_postgres_service_stat.stat.exists"
- name: Ensure systemd reloaded after matrix-postgres.service removal
ansible.builtin.service:
daemon_reload: true
when: "not matrix_postgres_enabled | bool and matrix_postgres_service_stat.stat.exists"
- name: Check existence of matrix-postgres local data path
ansible.builtin.stat:
path: "{{ matrix_postgres_data_path }}"
register: matrix_postgres_data_path_stat
when: "not matrix_postgres_enabled | bool"
# We just want to notify the user. Deleting data is too destructive.
- name: Inject warning if matrix-postgres local data remains
ansible.builtin.set_fact:
matrix_playbook_runtime_results: |
{{
matrix_playbook_runtime_results | default([])
+
[
"NOTE: You are not using a local PostgreSQL database, but some old data remains from before in `{{ matrix_postgres_data_path }}`. Feel free to delete it."
]
}}
when: "not matrix_postgres_enabled | bool and matrix_postgres_data_path_stat.stat.exists"
- name: Remove Postgres scripts
ansible.builtin.file:
path: "{{ matrix_local_bin_path }}/{{ item }}"
state: absent
with_items:
- matrix-postgres-cli
- matrix-change-user-admin-status
- matrix-postgres-update-user-password-hash
when: "not matrix_postgres_enabled | bool"

188
roles/custom/matrix-postgres/tasks/upgrade_postgres.yml Normal file
View File

@ -0,0 +1,188 @@
---
- name: Set default postgres_dump_dir, if not provided
ansible.builtin.set_fact:
postgres_dump_dir: "/tmp"
when: "postgres_dump_dir | default('') == ''"
- name: Set postgres_dump_name, if not provided
ansible.builtin.set_fact:
postgres_dump_name: "matrix-postgres-dump.sql.gz"
when: "postgres_dump_name | default('') == ''"
- name: Set postgres_auto_upgrade_backup_data_path, if not provided
ansible.builtin.set_fact:
postgres_auto_upgrade_backup_data_path: "{{ matrix_postgres_data_path }}-auto-upgrade-backup"
when: "postgres_auto_upgrade_backup_data_path | default('') == ''"
- name: Set postgres_start_wait_time, if not provided
ansible.builtin.set_fact:
postgres_start_wait_time: 15
when: "postgres_start_wait_time | default('') == ''"
- name: Set postgres_force_upgrade, if not provided
ansible.builtin.set_fact:
postgres_force_upgrade: false
when: "postgres_force_upgrade | default('') == ''"
- name: Fail, if trying to upgrade external Postgres database
ansible.builtin.fail:
msg: "Your configuration indicates that you're not using Postgres from this role. There is nothing to upgrade."
when: "not matrix_postgres_enabled | bool"
- name: Check Postgres auto-upgrade backup data directory
ansible.builtin.stat:
path: "{{ postgres_auto_upgrade_backup_data_path }}"
register: result_auto_upgrade_path
- name: Abort, if existing Postgres auto-upgrade data path detected
ansible.builtin.fail:
msg: "Detected that a left-over {{ postgres_auto_upgrade_backup_data_path }} exists. You should rename it to {{ matrix_postgres_data_path }} if the previous upgrade went wrong, or delete it if it went well."
when: "result_auto_upgrade_path.stat.exists"
- ansible.builtin.import_tasks: tasks/detect_existing_postgres_version.yml
- name: Abort, if no existing Postgres version detected
ansible.builtin.fail:
msg: "Could not find existing Postgres installation"
when: "not matrix_postgres_detected_existing | bool"
- name: Abort, if already at latest Postgres version
ansible.builtin.fail:
msg: "You are already running the latest Postgres version supported ({{ matrix_postgres_docker_image_latest }}). Nothing to do"
when: "matrix_postgres_detected_version_corresponding_docker_image == matrix_postgres_docker_image_latest and not postgres_force_upgrade"
- ansible.builtin.debug:
msg: "Upgrading database from {{ matrix_postgres_detected_version_corresponding_docker_image }} to {{ matrix_postgres_docker_image_latest }}"
- name: Ensure services are stopped
ansible.builtin.service:
name: "{{ item }}"
state: stopped
daemon_reload: true
with_items: "{{ matrix_postgres_systemd_services_to_stop_for_maintenance_list }}"
- name: Ensure matrix-postgres is started
ansible.builtin.service:
name: matrix-postgres
state: started
daemon_reload: true
- name: Wait a bit, so that Postgres can start
ansible.builtin.wait_for:
timeout: "{{ postgres_start_wait_time }}"
delegate_to: 127.0.0.1
become: false
# We dump all databases, roles, etc.
#
# Because we'll be importing into a new container which initializes the default
# role (`matrix_postgres_connection_username`) and database (`matrix_postgres_db_name`) by itself on startup,
# we need to remove these from the dump, or we'll get errors saying these already exist.
- name: Perform Postgres database dump
ansible.builtin.command:
cmd: >-
{{ matrix_host_command_docker }} run --rm --name matrix-postgres-dump
--log-driver=none
--user={{ matrix_user_uid }}:{{ matrix_user_gid }}
--network={{ matrix_docker_network }}
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql
--entrypoint=/bin/sh
--mount type=bind,src={{ postgres_dump_dir }},dst=/out
{{ matrix_postgres_detected_version_corresponding_docker_image }}
-c "pg_dumpall -h matrix-postgres
{{ '| gzip -c ' if postgres_dump_name.endswith('.gz') else '' }}
> /out/{{ postgres_dump_name }}"
register: matrix_postgres_upgrade_postgres_dump_command_result
changed_when: matrix_postgres_upgrade_postgres_dump_command_result.rc == 0
tags:
- skip_ansible_lint
- name: Ensure matrix-postgres is stopped
ansible.builtin.service:
name: matrix-postgres
state: stopped
- name: Rename existing Postgres data directory
ansible.builtin.command:
cmd: "mv {{ matrix_postgres_data_path }} {{ postgres_auto_upgrade_backup_data_path }}"
register: matrix_postgres_upgrade_postgres_move_command_result
changed_when: matrix_postgres_upgrade_postgres_move_command_result.rc == 0
- ansible.builtin.debug:
msg: "NOTE: Your Postgres data directory has been moved from `{{ matrix_postgres_data_path }}` to `{{ postgres_auto_upgrade_backup_data_path }}`. In the event of failure, you can move it back and run the playbook with --tags=setup-postgres to restore operation."
- ansible.builtin.import_tasks: tasks/setup_postgres.yml
- name: Ensure matrix-postgres autoruns and is restarted
ansible.builtin.service:
name: matrix-postgres
enabled: true
state: restarted
daemon_reload: true
- name: Wait a bit, so that Postgres can start
ansible.builtin.wait_for:
timeout: "{{ postgres_start_wait_time }}"
delegate_to: 127.0.0.1
become: false
# Starting the database container had automatically created the default
# role (`matrix_postgres_connection_username`) and database (`matrix_postgres_db_name`).
# The dump most likely contains those same entries and would try to re-create them, leading to errors.
# We need to skip over those lines.
- name: Generate Postgres database import command
ansible.builtin.set_fact:
matrix_postgres_import_command: >-
{{ matrix_host_command_docker }} run --rm --name matrix-postgres-import
--log-driver=none
--user={{ matrix_user_uid }}:{{ matrix_user_gid }}
--cap-drop=ALL
--network={{ matrix_docker_network }}
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql
--entrypoint=/bin/sh
--mount type=bind,src={{ postgres_dump_dir }},dst=/in,ro
{{ matrix_postgres_docker_image_latest }}
-c "cat /in/{{ postgres_dump_name }} |
{{ 'gunzip |' if postgres_dump_name.endswith('.gz') else '' }}
grep -vE '{{ matrix_postgres_import_roles_ignore_regex }}' |
grep -vE '{{ matrix_postgres_import_databases_ignore_regex }}' |
psql -v ON_ERROR_STOP=1 -h matrix-postgres"
tags:
- skip_ansible_lint
# This is a hack.
# See: https://ansibledaily.com/print-to-standard-output-without-escaping/
#
# We want to run `debug: msg=".."`, but that dumps it as JSON and escapes double quotes within it,
# which ruins the command (`matrix_postgres_import_command`)
- name: Note about Postgres importing
ansible.builtin.set_fact:
dummy: true
with_items:
- >-
Importing Postgres database using the following command: `{{ matrix_postgres_import_command }}`.
If this crashes, you can stop Postgres (`systemctl stop matrix-postgres`),
delete the new database data (`rm -rf {{ matrix_postgres_data_path }}`)
and restore the automatically-made backup (`mv {{ postgres_auto_upgrade_backup_data_path }} {{ matrix_postgres_data_path }}`).
- name: Perform Postgres database import
ansible.builtin.command:
cmd: "{{ matrix_postgres_import_command }}"
register: matrix_postgres_upgrade_postgres_import_command_result
changed_when: matrix_postgres_upgrade_postgres_import_command_result.rc == 0
- name: Delete Postgres database dump file
ansible.builtin.file:
path: "{{ postgres_dump_dir }}/{{ postgres_dump_name }}"
state: absent
- name: Ensure services are started
ansible.builtin.service:
name: "{{ item }}"
state: started
daemon_reload: true
with_items: "{{ matrix_postgres_systemd_services_to_stop_for_maintenance_list }}"
- ansible.builtin.debug:
msg: "NOTE: Your old Postgres data directory is preserved at `{{ postgres_auto_upgrade_backup_data_path }}`. You might want to get rid of it once you've confirmed that all is well."

41
roles/custom/matrix-postgres/tasks/util/create_additional_database.yml Normal file
View File

@ -0,0 +1,41 @@
---
# It'd be better if this is belonged to `validate_config.yml`, but it would have to be some loop-within-a-loop there,
# and that's ugly. We also don't expect this to catch errors often. It's more of a defensive last-minute check.
- name: Fail if additional database data appears invalid
ansible.builtin.fail:
msg: "Additional database definition ({{ additional_db }} lacks a required key: {{ item }}"
when: "item not in additional_db"
with_items: "{{ ['name', 'username', 'password'] }}"
# The SQL statements that we'll run against Postgres are stored in a file that others can't read.
# This file will be mounted into the container and fed to Postgres.
# This way, we avoid passing sensitive data around in CLI commands that other users on the system can see.
- name: Create additional database initialization SQL file for {{ additional_db.name }}
ansible.builtin.template:
src: "{{ role_path }}/templates/sql/init-additional-db-user-and-role.sql.j2"
dest: "/tmp/matrix-postgres-init-additional-db-user-and-role.sql"
mode: 0600
owner: "{{ matrix_user_uid }}"
group: "{{ matrix_user_gid }}"
- name: Execute Postgres additional database initialization SQL file for {{ additional_db.name }}
ansible.builtin.command:
cmd: >-
{{ matrix_host_command_docker }} run
--rm
--user={{ matrix_user_uid }}:{{ matrix_user_gid }}
--cap-drop=ALL
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql
--network {{ matrix_docker_network }}
--mount type=bind,src=/tmp/matrix-postgres-init-additional-db-user-and-role.sql,dst=/matrix-postgres-init-additional-db-user-and-role.sql,ro
--entrypoint=/bin/sh
{{ matrix_postgres_docker_image_to_use }}
-c
'psql -h {{ matrix_postgres_connection_hostname }} --file=/matrix-postgres-init-additional-db-user-and-role.sql'
changed_when: true
- name: Delete additional database initialization SQL file for {{ additional_db.name }}
ansible.builtin.file:
path: /tmp/matrix-postgres-init-additional-db-user-and-role.sql
state: absent

23
roles/custom/matrix-postgres/tasks/util/create_additional_databases.yml Normal file
View File

@ -0,0 +1,23 @@
---
- name: Ensure matrix-postgres is started
ansible.builtin.service:
name: matrix-postgres
state: started
daemon_reload: true
register: matrix_postgres_service_start_result
- name: Wait a bit, so that Postgres can start
ansible.builtin.wait_for:
timeout: "{{ matrix_postgres_additional_databases_postgres_start_wait_timeout_seconds }}"
delegate_to: 127.0.0.1
become: false
when: "matrix_postgres_service_start_result.changed | bool"
- name: Create additional Postgres user and database
ansible.builtin.include_tasks: "{{ role_path }}/tasks/util/create_additional_database.yml"
with_items: "{{ matrix_postgres_additional_databases }}"
loop_control:
loop_var: additional_db
# Suppress logging to avoid dumping the credentials to the shell
no_log: true

39
roles/custom/matrix-postgres/tasks/validate_config.yml Normal file
View File

@ -0,0 +1,39 @@
---
- name: (Deprecation) Warn about matrix_postgres_use_external usage
ansible.builtin.fail:
msg: >
The `matrix_postgres_use_external` variable defined in your configuration is not used by this playbook anymore!
You'll need to adapt to the new way of using an external Postgres server.
It's a combination of `matrix_postgres_enabled: false` and specifying Postgres connection
details in a few `matrix_postgres_connection_` variables.
See the "Using an external PostgreSQL server (optional)" documentation page.
when: "'matrix_postgres_use_external' in vars"
# This is separate (from the other required variables below),
# because we'd like to have a friendlier message for our existing users.
- name: Fail if matrix_postgres_connection_password not defined
ansible.builtin.fail:
msg: >-
The playbook no longer has a default Postgres password defined in the `matrix_postgres_connection_password` variable, among lots of other Postgres changes.
You need to perform multiple manual steps to resolve this.
See our changelog for more details:
https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#breaking-change-postgres-changes-that-require-manual-intervention
when: "matrix_postgres_connection_password == ''"
- name: Fail if required Postgres settings not defined
ansible.builtin.fail:
msg: >-
You need to define a required configuration setting (`{{ item }}`).
when: "vars[item] == ''"
with_items:
- "matrix_postgres_connection_hostname"
- "matrix_postgres_connection_port"
- "matrix_postgres_connection_username"
- "matrix_postgres_connection_password"
- "matrix_postgres_db_name"
- name: Fail if Postgres password length exceeded
ansible.builtin.fail:
msg: "The maximum `matrix_postgres_connection_password` length is 99 characters"
when: "matrix_postgres_connection_password | length > 99"

8
roles/custom/matrix-postgres/templates/env-postgres-psql.j2 Normal file
View File

@ -0,0 +1,8 @@
#jinja2: lstrip_blocks: "True"
PGUSER={{ matrix_postgres_connection_username }}
PGPASSWORD={{ matrix_postgres_connection_password }}
PGDATABASE={{ matrix_postgres_db_name }}
# Prevent errors like this:
# > could not save history to file "//.psql_history": Permission denied
# .. due to Postgres not being able to write to the filesystem.
PSQL_HISTORY=/dev/null

7
roles/custom/matrix-postgres/templates/env-postgres-server.j2 Normal file
View File

@ -0,0 +1,7 @@
#jinja2: lstrip_blocks: "True"
POSTGRES_USER={{ matrix_postgres_connection_username }}
POSTGRES_PASSWORD={{ matrix_postgres_connection_password }}
POSTGRES_DB={{ matrix_postgres_db_name }}
# Synapse refuses to run if collation is not C.
# See https://github.com/matrix-org/synapse/issues/6722
POSTGRES_INITDB_ARGS=--lc-collate C --lc-ctype C --encoding UTF8

19
roles/custom/matrix-postgres/templates/sql/init-additional-db-user-and-role.sql.j2 Normal file
View File

@ -0,0 +1,19 @@
-- `CREATE USER` does not support `IF NOT EXISTS`, so we use this workaround to prevent an error and raise a notice instead.
-- Seen here: https://stackoverflow.com/a/49858797
DO $$
BEGIN
CREATE USER "{{ additional_db.username }}";
EXCEPTION WHEN DUPLICATE_OBJECT THEN
RAISE NOTICE 'not creating user "{{ additional_db.username }}", since it already exists';
END
$$;
-- This is useful for initial user creation (since we don't assign a password above) and for handling subsequent password changes
-- TODO - we should escape quotes in the password.
ALTER ROLE "{{ additional_db.username }}" PASSWORD '{{ additional_db.password }}';
-- This will generate an error on subsequent execution
CREATE DATABASE "{{ additional_db.name }}" WITH LC_CTYPE 'C' LC_COLLATE 'C' OWNER "{{ additional_db.username }}";
-- This is useful for changing the database owner subsequently
ALTER DATABASE "{{ additional_db.name }}" OWNER TO "{{ additional_db.username }}";

46
roles/custom/matrix-postgres/templates/systemd/matrix-postgres.service.j2 Normal file
View File

@ -0,0 +1,46 @@
#jinja2: lstrip_blocks: "True"
[Unit]
Description=Matrix Postgres server
After=docker.service
Requires=docker.service
DefaultDependencies=no
[Service]
Type=simple
Environment="HOME={{ matrix_systemd_unit_home_path }}"
ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-postgres 2>/dev/null || true'
ExecStartPre=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-postgres 2>/dev/null || true'
# We need /dev/shm to be larger than the default to allow VACUUM to work.
# See:
# - https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1362
# - https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1268
ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-postgres \
--log-driver=none \
--user={{ matrix_user_uid }}:{{ matrix_user_gid }} \
--cap-drop=ALL \
--read-only \
--tmpfs=/tmp:rw,noexec,nosuid,size=100m \
--tmpfs=/run/postgresql:rw,noexec,nosuid,size=100m \
--shm-size=256m \
--network={{ matrix_docker_network }} \
{% if matrix_postgres_container_postgres_bind_port %}
-p {{ matrix_postgres_container_postgres_bind_port }}:5432 \
{% endif %}
--env-file={{ matrix_postgres_base_path }}/env-postgres-server \
--mount type=bind,src={{ matrix_postgres_data_path }},dst=/var/lib/postgresql/data \
--mount type=bind,src=/etc/passwd,dst=/etc/passwd,ro \
{% for arg in matrix_postgres_container_extra_arguments %}
{{ arg }} \
{% endfor %}
{{ matrix_postgres_docker_image_to_use }} \
postgres {{ matrix_postgres_process_extra_arguments|join(' ') }}
ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} kill matrix-postgres 2>/dev/null || true'
ExecStop=-{{ matrix_host_command_sh }} -c '{{ matrix_host_command_docker }} rm matrix-postgres 2>/dev/null || true'
Restart=always
RestartSec=30
SyslogIdentifier=matrix-postgres
[Install]
WantedBy=multi-user.target

19
roles/custom/matrix-postgres/templates/usr-local-bin/matrix-change-user-admin-status.j2 Normal file
View File

@ -0,0 +1,19 @@
#jinja2: lstrip_blocks: "True"
#!/bin/bash
if [ $# -ne 2 ]; then
echo "Usage: "$0" <username> <0/1>"
echo "Usage: 0 = non-admin"
echo "Usage: 1 = admin"
exit 1
fi
docker run \
-i \
--rm \
--user={{ matrix_user_uid }}:{{ matrix_user_gid }} \
--cap-drop=ALL \
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql \
--network {{ matrix_docker_network }} \
{{ matrix_postgres_docker_image_to_use }} \
psql -h {{ matrix_postgres_connection_hostname }} --dbname={{ matrix_synapse_database_database }} -c "UPDATE users set admin=$2 WHERE name like '@$1:{{ matrix_domain }}'"

12
roles/custom/matrix-postgres/templates/usr-local-bin/matrix-postgres-cli-non-interactive.j2 Normal file
View File

@ -0,0 +1,12 @@
#jinja2: lstrip_blocks: "True"
#!/bin/bash
docker run \
--rm \
--user={{ matrix_user_uid }}:{{ matrix_user_gid }} \
--cap-drop=ALL \
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql \
--network {{ matrix_docker_network }} \
{{ matrix_postgres_docker_image_to_use }} \
psql -h {{ matrix_postgres_connection_hostname }} \
"$@"

13
roles/custom/matrix-postgres/templates/usr-local-bin/matrix-postgres-cli.j2 Normal file
View File

@ -0,0 +1,13 @@
#jinja2: lstrip_blocks: "True"
#!/bin/bash
docker run \
-it \
--rm \
--user={{ matrix_user_uid }}:{{ matrix_user_gid }} \
--cap-drop=ALL \
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql \
--network {{ matrix_docker_network }} \
{{ matrix_postgres_docker_image_to_use }} \
psql -h {{ matrix_postgres_connection_hostname }} \
"$@"

16
roles/custom/matrix-postgres/templates/usr-local-bin/matrix-postgres-update-user-password-hash.j2 Normal file
View File

@ -0,0 +1,16 @@
#jinja2: lstrip_blocks: "True"
#!/bin/bash
if [ $# -ne 2 ]; then
echo "Usage: "$0" <username> <password_hash>"
exit 1
fi
docker run \
--rm \
--user={{ matrix_user_uid }}:{{ matrix_user_gid }} \
--cap-drop=ALL \
--env-file={{ matrix_postgres_base_path }}/env-postgres-psql \
--network {{ matrix_docker_network }} \
{{ matrix_postgres_docker_image_to_use }} \
psql -h {{ matrix_postgres_connection_hostname }} --dbname={{ matrix_synapse_database_database }} -c "UPDATE users set password_hash='$2' WHERE name = '@$1:{{ matrix_domain }}'"