diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 000000000..bc3c8e757 --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1,4 @@ +# These are supported funding model platforms + +# https://liberapay.com/s.pantaleev/ +liberapay: s.pantaleev diff --git a/CHANGELOG.md b/CHANGELOG.md index c350d2356..c4a586e8b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,91 @@ +# 2020-12-23 + +## The big move to all-on-Postgres (potentially dangerous) + +**TLDR**: all your bridges (and other services) will likely be auto-migrated from SQLite/nedb to Postgres, hopefully without trouble. You can opt-out (see how below), if too worried about breakage. + +Until now, we've only used Postgres as a database for Synapse. All other services (bridges, bots, etc.) were kept simple and used a file-based database (SQLite or nedb). + +Since [this huge pull request](https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/740), **all of our services now use Postgres by default**. Thanks to [Johanna Dorothea Reichmann](https://github.com/jdreichmann) for starting the work on it and for providing great input! + +Moving all services to Postgres brings a few **benefits** to us: + +- **improved performance** +- **improved compatibility**. Most bridges are deprecating SQLite/nedb support or offer less features when not on Postgres. +- **easier backups**. It's still some effort to take a proper backup (Postgres dump + various files, keys), but a Postgres dump now takes you much further. +- we're now **more prepared to introduce other services** that need a Postgres database - [Dendrite](https://github.com/matrix-org/dendrite), the [mautrix-signal](https://github.com/tulir/mautrix-signal) bridge (existing [pull request](https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/686)), etc. + +### Key takeway + +- existing installations that use an [external Postgres](https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/docs/configuring-playbook-external-postgres.md) server should be unaffected (they remain on SQLite/nedb for all services, except Synapse) + +- for existing installations which use our integrated Postgres database server (`matrix-postgres`, which is the default), **we automatically migrate data** from SQLite/nedb to Postgres and **archive the database files** (`something.db` -> `something.db.backup`), so you can restore them if you need to go back (see how below). + +### Opting-out of the Postgres migration + +This is a **very large and somewhat untested change** (potentially dangerous), so **if you're not feeling confident/experimental, opt-out** of it for now. Still, it's the new default and what we (and various bridges) will focus on going forward, so don't stick to old ways for too long. + +You can remain on SQLite/nedb (at least for now) by adding a variable like this to your `vars.yml` file for each service you use: `matrix_COMPONENT_database_engine: sqlite` (e.g. `matrix_mautrix_facebook_database_engine: sqlite`). + +Some services (like `appservice-irc` and `appservice-slack`) don't use SQLite, so use `nedb`, instead of `sqlite` for them. + +### Going back to SQLite/nedb if things went wrong + +If you went with the Postgres migration and it went badly for you (some bridge not working as expected or not working at all), do this: + +- stop all services (`ansible-playbook -i inventory/hosts setup.yml --tags=stop`) +- SSH into the server and rename the old database files (`something.db.backup` -> `something.db`). Example: `mv /matrix/mautrix-facebook/data/mautrix-facebook.db.backup /matrix/mautrix-facebook/data/mautrix-facebook.db` +- switch the affected service back to SQLite (e.g. `matrix_mautrix_facebook_database_engine: sqlite`). Some services (like `appservice-irc` and `appservice-slack`) don't use SQLite, so use `nedb`, instead of `sqlite` for them. +- re-run the playbook (`ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start`) +- [get in touch](README.md#support) with us + + +# 2020-12-11 + +## synapse-janitor support removed + +We've removed support for the unmaintained [synapse-janitor](https://github.com/xwiki-labs/synapse_scripts) script. There's been past reports of it corrupting the Synapse database. Since there hasn't been any new development on it and it doesn't seem too useful nowadays, there's no point in including it in the playbook. + +If you need to clean up or compact your database, consider using the Synapse Admin APIs directly. See our [Synapse maintenance](docs/maintenance-synapse.md) and [Postgres maintenance](docs/maintenance-postgres.md) documentation pages for more details. + + +## Docker 20.10 is here + +(No need to do anything special in relation to this. Just something to keep in mind) + +Docker 20.10 got released recently and your server will likely get it the next time you update. + +This is the first major Docker update in a long time and it packs a lot of changes. +Some of them introduced some breakage for us initially (see [here](https://github.com/spantaleev/matrix-docker-ansible-deploy/commit/d08b27784f222effcbce2abf924bf07bbe0893be) and [here](https://github.com/spantaleev/matrix-docker-ansible-deploy/commit/7593d969e316cc0144bce378a5be58c76c2c37ee)), but it should be all good now. + + +# 2020-12-08 + +## openid APIs exposed by default on the federation port when federation disabled + +We've changed some defaults. People running with our default configuration (federation enabled), are not affected at all. + +If you are running an unfederated server (`matrix_synapse_federation_enabled: false`), this may be of interest to you. + +When federation is disabled, but ma1sd or Dimension are enabled, we'll now expose the `openid` APIs on the federation port. +These APIs are necessary for some ma1sd features to work. If you'd like to prevent this, you can: `matrix_synapse_federation_port_openid_resource_required: false`. + + +# 2020-11-27 + +## Recent Jitsi updates may require configuration changes + +We've recently [updated from Jitsi build 4857 to build 5142](https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/719), which brings a lot of configuration changes. + +**If you use our default Jitsi settings, you won't have to do anything.** + +People who have [fine-tuned Jitsi](docs/configuring-playbook-jitsi.md#optional-fine-tune-jitsi) may find that some options got renamed now, others are gone and yet others still need to be defined in another way. + +The next time you run the playbook [installation](docs/installing.md) command, our validation logic will tell you if you're using some variables like that and will recommend a migration path for each one. + +Additionally, we've recently disabled transcriptions (`matrix_jitsi_enable_transcriptions: false`) and recording (`matrix_jitsi_enable_recording: false`) by default. These features did not work anyway, because we don't install the required dependencies for them (Jigasi and Jibri, respectively). If you've been somehow pointing your Jitsi installation to some manually installed Jigasi/Jibri service, you may need to toggle these flags back to enabled to have transcriptions and recordings working. + + # 2020-11-23 ## Breaking change matrix-sms-bridge diff --git a/README.md b/README.md index 1a2079377..ac2b7ca96 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ +[![Support room on Matrix](https://img.shields.io/matrix/matrix-docker-ansible-deploy:devture.com.svg?label=%23matrix-docker-ansible-deploy%3Adevture.com&logo=matrix&style=for-the-badge&server_fqdn=matrix.devture.com)](https://matrix.to/#/#matrix-docker-ansible-deploy:devture.com) [![donate](https://liberapay.com/assets/widgets/donate.svg)](https://liberapay.com/s.pantaleev/donate) + # Matrix (An open network for secure, decentralized communication) server setup using Ansible and Docker ## Purpose diff --git a/docs/README.md b/docs/README.md index 8d22477ea..8c95eff3b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,13 +10,13 @@ - [Installing](installing.md) -- **Importing data from another Synapse server installation** +- **Importing data from another server installation** - - [Importing an existing SQLite database (from another installation)](importing-sqlite.md) (optional) + - [Importing an existing SQLite database (from another Synapse installation)](importing-synapse-sqlite.md) (optional) - [Importing an existing Postgres database (from another installation)](importing-postgres.md) (optional) - - [Importing `media_store` data files from an existing installation](importing-media-store.md) (optional) + - [Importing `media_store` data files from an existing Synapse installation](importing-synapse-media-store.md) (optional) - [Registering users](registering-users.md) diff --git a/docs/configuring-playbook-bridge-appservice-discord.md b/docs/configuring-playbook-bridge-appservice-discord.md index 13a5174bb..f3efc555d 100644 --- a/docs/configuring-playbook-bridge-appservice-discord.md +++ b/docs/configuring-playbook-bridge-appservice-discord.md @@ -22,10 +22,10 @@ matrix_appservice_discord_client_id: "YOUR DISCORD APP CLIENT ID" matrix_appservice_discord_bot_token: "YOUR DISCORD APP BOT TOKEN" ``` -4. If you've already installed Matrix services using the playbook before, you'll need to re-run it (`--tags=setup-all,start`). If not, proceed with [configuring other playbook services](configuring-playbook.md) and then with [Installing](installing.md). Get back to this guide once ready. -5. Retrieve Discord invite link from the `{{ matrix_appservice_discord_config_path }}/invite_link` file on the server (this defaults to `/matrix/appservice-discord/config/invite_link`). You need to peek at the file on the server via SSH, etc., because it's not available via HTTP(S). -6. Invite the Bot to Discord servers you wish to bridge. Administrator permission is recommended. -7. Room addresses follow this syntax: `#_discord_guildid_channelid`. You can easily find the guild and channel ids by logging into Discord in a browser and opening the desired channel. The URL will have this format: `discordapp.com/channels/guild_id/channel_id`. Once you have figured out the appropriate room addrss, you can join by doing `/join #_discord_guildid_channelid` in your Matrix client. +5. If you've already installed Matrix services using the playbook before, you'll need to re-run it (`--tags=setup-all,start`). If not, proceed with [configuring other playbook services](configuring-playbook.md) and then with [Installing](installing.md). Get back to this guide once ready. +6. Retrieve Discord invite link from the `{{ matrix_appservice_discord_config_path }}/invite_link` file on the server (this defaults to `/matrix/appservice-discord/config/invite_link`). You need to peek at the file on the server via SSH, etc., because it's not available via HTTP(S). +7. Invite the Bot to Discord servers you wish to bridge. Administrator permission is recommended. +8. Room addresses follow this syntax: `#_discord_guildid_channelid`. You can easily find the guild and channel ids by logging into Discord in a browser and opening the desired channel. The URL will have this format: `discordapp.com/channels/guild_id/channel_id`. Once you have figured out the appropriate room addrss, you can join by doing `/join #_discord_guildid_channelid` in your Matrix client. Other configuration options are available via the `matrix_appservice_discord_configuration_extension_yaml` variable. diff --git a/docs/configuring-playbook-dimension.md b/docs/configuring-playbook-dimension.md index 604e6aa61..d5f0a9e62 100644 --- a/docs/configuring-playbook-dimension.md +++ b/docs/configuring-playbook-dimension.md @@ -3,6 +3,9 @@ **[Dimension](https://dimension.t2bot.io) can only be installed after Matrix services are installed and running.** If you're just installing Matrix services for the first time, please continue with the [Configuration](configuring-playbook.md) / [Installation](installing.md) flow and come back here later. +**Note**: enabling Dimension, means that the `openid` API endpoints will be exposed on the Matrix Federation port (usually `8448`), even if [federation](configuring-playbook-federation.md) is disabled. It's something to be aware of, especially in terms of firewall whitelisting (make sure port `8448` is accessible). + + ## Prerequisites This playbook now supports running [Dimension](https://dimension.t2bot.io) in both a federated and an [unfederated](https://github.com/turt2live/matrix-dimension/blob/master/docs/unfederated.md) environment. This is handled automatically based on the value of `matrix_synapse_federation_enabled`. @@ -48,7 +51,7 @@ To get an access token for the Dimension user, you can follow one of two options 3. Copy the highlighted text to your configuration. 4. Close the private browsing session. **Do not log out**. Logging out will invalidate the token, making it not work. -*With CURL* +*With CURL* ``` curl -X POST --header 'Content-Type: application/json' -d '{ diff --git a/docs/configuring-playbook-federation.md b/docs/configuring-playbook-federation.md index 1e4ad61e3..2e6410ec0 100644 --- a/docs/configuring-playbook-federation.md +++ b/docs/configuring-playbook-federation.md @@ -37,3 +37,13 @@ matrix_synapse_federation_enabled: false ``` With that, your server's users will only be able to talk among themselves, but not to anyone who is on another server. + +**Disabling federation does not necessarily disable the federation port** (`8448`). Services like [Dimension](configuring-playbook-dimension.md) and [ma1sd](configuring-playbook-ma1sd.md) normally rely on `openid` APIs exposed on that port. Even if you disable federation and only if necessary, we may still be exposing the federation port and serving the `openid` APIs there. To override this and completely disable Synapse's federation port use: + +```yaml +# This stops the federation port on the Synapse side (normally `matrix-synapse:8048` on the container network). +matrix_synapse_federation_port_enabled: false + +# This removes the `8448` virtual host from the matrix-nginx-proxy reverse-proxy server. +matrix_nginx_proxy_proxy_matrix_federation_api_enabled: false +``` diff --git a/docs/configuring-playbook-jitsi.md b/docs/configuring-playbook-jitsi.md index 62edc578f..f72241e12 100644 --- a/docs/configuring-playbook-jitsi.md +++ b/docs/configuring-playbook-jitsi.md @@ -99,7 +99,7 @@ matrix_jitsi_web_custom_config_extension: | config.disableAudioLevels = true; - # Limit the number of video feeds forwarded to each client + // Limit the number of video feeds forwarded to each client config.channelLastN = 4; matrix_jitsi_web_config_resolution_width_ideal_and_max: 480 diff --git a/docs/configuring-playbook-ma1sd.md b/docs/configuring-playbook-ma1sd.md index 03208337f..70c507cba 100644 --- a/docs/configuring-playbook-ma1sd.md +++ b/docs/configuring-playbook-ma1sd.md @@ -4,7 +4,9 @@ By default, this playbook configures an [ma1sd](https://github.com/ma1uta/ma1sd) This server is private by default, potentially at the expense of user discoverability. -ma1sd is a fork of [mxisd](https://github.com/kamax-io/mxisd) which was pronounced end of life 2019-06-21. +*ma1sd is a fork of [mxisd](https://github.com/kamax-io/mxisd) which was pronounced end of life 2019-06-21.* + +**Note**: enabling ma1sd (which is also the default), means that the `openid` API endpoints will be exposed on the Matrix Federation port (usually `8448`), even if [federation](configuring-playbook-federation.md) is disabled. It's something to be aware of, especially in terms of firewall whitelisting (make sure port `8448` is accessible). ## Disabling ma1sd @@ -50,6 +52,9 @@ To use the [Registration](https://github.com/ma1uta/ma1sd/blob/master/docs/featu - `matrix_ma1sd_configuration_extension_yaml` - to configure ma1sd as required. See the [Registration feature's docs](https://github.com/ma1uta/ma1sd/blob/master/docs/features/registration.md) for inspiration. Also see the [Additional features](#additional-features) section below to learn more about how to use `matrix_ma1sd_configuration_extension_yaml`. +**Note**: For this to work, either the homeserver needs to [federate](configuring-playbook-federation.md) or the `openid` APIs need to exposed on the federation port. When federation is disabled and ma1sd is enabled, we automatically expose the `openid` APIs (only!) on the federation port. Make sure the federation port (usually `https://matrix.DOMAIN:8448`) is whitelisted in your firewall (even if you don't actually use/need federation). + + ## Authentication [Authentication](https://github.com/ma1uta/ma1sd/blob/master/docs/features/authentication.md) provides the possibility to use your own [Identity Stores](https://github.com/ma1uta/ma1sd/blob/master/docs/stores/README.md) (for example LDAP) to authenticate users on your Homeserver. The following configuration can be used to authenticate against an LDAP server: diff --git a/docs/configuring-playbook-nginx.md b/docs/configuring-playbook-nginx.md index 945864e97..5693c5695 100644 --- a/docs/configuring-playbook-nginx.md +++ b/docs/configuring-playbook-nginx.md @@ -24,7 +24,6 @@ matrix_nginx_proxy_proxy_matrix_nginx_status_allowed_addresses: - 1.1.1.1 ``` - ## Synapse + OpenID Connect for Single-Sign-On If you want to use OpenID Connect as an SSO provider (as per the [Synapse OpenID docs](https://github.com/matrix-org/synapse/blob/develop/docs/openid.md)), you need to use the following configuration (in your `vars.yml` file) to instruct nginx to forward `/_synapse/oidc` to Synapse: @@ -32,3 +31,11 @@ If you want to use OpenID Connect as an SSO provider (as per the [Synapse OpenID ```yaml matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_synapse_oidc_api_enabled: true ``` + +## Disable Nginx access logs + +This will disable the access logging for nginx. + +```yaml +matrix_nginx_proxy_access_log_enabled: false +``` diff --git a/docs/configuring-playbook-own-webserver.md b/docs/configuring-playbook-own-webserver.md index 8c671df4d..c930da405 100644 --- a/docs/configuring-playbook-own-webserver.md +++ b/docs/configuring-playbook-own-webserver.md @@ -144,7 +144,7 @@ matrix_nginx_proxy_container_extra_arguments: - '--label "traefik.enable=true"' # The Nginx proxy container will receive traffic from these subdomains - - '--label "traefik.http.routers.matrix-nginx-proxy.rule=Host(`{{ matrix_server_fqn_matrix }}`,`{{ matrix_server_fqn_element }}`,`{{ matrix_server_fqn_dimension }},`{{ matrix_server_fqn_jitsi }}`)"' + - '--label "traefik.http.routers.matrix-nginx-proxy.rule=Host(`{{ matrix_server_fqn_matrix }}`,`{{ matrix_server_fqn_element }}`,`{{ matrix_server_fqn_dimension }}`,`{{ matrix_server_fqn_jitsi }}`)"' # (The 'web-secure' entrypoint must bind to port 443 in Traefik config) - '--label "traefik.http.routers.matrix-nginx-proxy.entrypoints=web-secure"' diff --git a/docs/configuring-playbook-ssl-certificates.md b/docs/configuring-playbook-ssl-certificates.md index 5b5c7cdc1..7f05a5b25 100644 --- a/docs/configuring-playbook-ssl-certificates.md +++ b/docs/configuring-playbook-ssl-certificates.md @@ -67,6 +67,7 @@ By default, it obtains certificates for: - possibly for `element.`, unless you have disabled the [Element client component](configuring-playbook-client-element.md) using `matrix_client_element_enabled: false` - possibly for `riot.`, if you have explicitly enabled Riot to Element redirection (for background compatibility) using `matrix_nginx_proxy_proxy_riot_compat_redirect_enabled: true` - possibly for `dimension.`, if you have explicitly [set up Dimension](configuring-playbook-dimension.md). +- possibly for `jitsi.`, if you have explicitly [set up Jitsi](configuring-playbook-jitsi.md). - possibly for your base domain (``), if you have explicitly configured [Serving the base domain](configuring-playbook-base-domain-serving.md) If you are hosting other domains on the Matrix machine, you can make the playbook obtain and renew certificates for those other domains too. @@ -80,6 +81,7 @@ matrix_ssl_domains_to_obtain_certificates_for: - '{{ matrix_server_fqn_matrix }}' - '{{ matrix_server_fqn_element }}' - '{{ matrix_server_fqn_dimension }}' + - '{{ matrix_server_fqn_jitsi }}' - '{{ matrix_domain }}' ``` diff --git a/docs/importing-postgres.md b/docs/importing-postgres.md index a88067e1c..0dd75cb2e 100644 --- a/docs/importing-postgres.md +++ b/docs/importing-postgres.md @@ -1,7 +1,7 @@ # Importing an existing Postgres database from another installation (optional) -Run this if you'd like to import your database from a previous installation of Synapse. -(don't forget to import your `media_store` files as well - see [the importing-media-store guide](importing-media-store.md)). +Run this if you'd like to import your database from a previous installation. +(don't forget to import your Synapse `media_store` files as well - see [the importing-synape-media-store guide](importing-synapse-media-store.md)). ## Prerequisites diff --git a/docs/importing-media-store.md b/docs/importing-synapse-media-store.md similarity index 92% rename from docs/importing-media-store.md rename to docs/importing-synapse-media-store.md index 0d86370b4..0ba7bacbc 100644 --- a/docs/importing-media-store.md +++ b/docs/importing-synapse-media-store.md @@ -1,4 +1,4 @@ -# Importing `media_store` data files from an existing installation (optional) +# Importing `media_store` data files from an existing Synapse installation (optional) Run this if you'd like to import your `media_store` files from a previous installation of Synapse. @@ -17,6 +17,6 @@ As an alternative, you can perform a manual restore using the [AWS CLI tool](htt Run this command (make sure to replace `` with a path on your server): - ansible-playbook -i inventory/hosts setup.yml --extra-vars='server_path_media_store=' --tags=import-media-store + ansible-playbook -i inventory/hosts setup.yml --extra-vars='server_path_media_store=' --tags=import-synapse-media-store **Note**: `` must be a file path to a `media_store` directory on the server (not on your local machine!). diff --git a/docs/importing-sqlite.md b/docs/importing-synapse-sqlite.md similarity index 86% rename from docs/importing-sqlite.md rename to docs/importing-synapse-sqlite.md index 9e3a910d1..aade22261 100644 --- a/docs/importing-sqlite.md +++ b/docs/importing-synapse-sqlite.md @@ -1,7 +1,7 @@ -# Importing an existing SQLite database from another installation (optional) +# Importing an existing SQLite database from another Synapse installation (optional) Run this if you'd like to import your database from a previous default installation of Synapse. -(don't forget to import your `media_store` files as well - see [the importing-media-store guide](importing-media-store.md)). +(don't forget to import your `media_store` files as well - see [the importing-synapse-media-store guide](importing-synapse-media-store.md)). While this playbook always sets up PostgreSQL, by default a Synapse installation would run using an SQLite database. @@ -18,7 +18,7 @@ Before doing the actual import, **you need to upload your SQLite database file t Run this command (make sure to replace `` with a file path on your server): - ansible-playbook -i inventory/hosts setup.yml --extra-vars='server_path_homeserver_db=' --tags=import-sqlite-db + ansible-playbook -i inventory/hosts setup.yml --extra-vars='server_path_homeserver_db=' --tags=import-synapse-sqlite-db **Notes**: diff --git a/docs/installing.md b/docs/installing.md index 43758ffb2..a2ce1371f 100644 --- a/docs/installing.md +++ b/docs/installing.md @@ -21,11 +21,11 @@ Feel free to **re-run this setup command any time** you think something is off w After installing, but before starting the services, you may want to do additional things like: -- [Importing an existing SQLite database (from another installation)](importing-sqlite.md) (optional) +- [Importing an existing SQLite database (from another Synapse installation)](importing-synapse-sqlite.md) (optional) - [Importing an existing Postgres database (from another installation)](importing-postgres.md) (optional) -- [Importing `media_store` data files from an existing installation](importing-media-store.md) (optional) +- [Importing `media_store` data files from an existing Synapse installation](importing-synapse-media-store.md) (optional) ## Starting the services diff --git a/docs/maintenance-synapse.md b/docs/maintenance-synapse.md index 799982990..143238c1f 100644 --- a/docs/maintenance-synapse.md +++ b/docs/maintenance-synapse.md @@ -4,14 +4,11 @@ This document shows you how to perform various maintenance tasks related to the Table of contents: -- [Purging unused data with synapse-janitor](#purging-unused-data-with-synapse-janitor), for when you wish to delete unused data from the Synapse database - - [Purging old data with the Purge History API](#purging-old-data-with-the-purge-history-api), for when you wish to delete in-use (but old) data from the Synapse database - [Synapse maintenance](#synapse-maintenance) - [Purging old data with the Purge History API](#purging-old-data-with-the-purge-history-api) - [Compressing state with rust-synapse-compress-state](#compressing-state-with-rust-synapse-compress-state) - - [Purging unused data with synapse-janitor](#purging-unused-data-with-synapse-janitor) - [Browse and manipulate the database](#browse-and-manipulate-the-database) - [Browse and manipulate the database](#browse-and-manipulate-the-database), for when you really need to take matters into your own hands @@ -57,27 +54,6 @@ If you need to adjust this, pass: `--extra-vars='matrix_synapse_rust_synapse_com After state compression, you may wish to run a [`FULL` Postgres `VACUUM`](./maintenance-postgres.md#vacuuming-postgresql). -## Purging unused data with synapse-janitor - -**NOTE**: There are [reports](https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/465) that **synapse-janitor is dangerous to use and causes database corruption**. You may wish to refrain from using it. - -When you **leave** and **forget** a room, Synapse can clean up its data, but currently doesn't. -This **unused and unreachable data** remains in your database forever. - -There are external tools (like [synapse-janitor](https://github.com/xwiki-labs/synapse_scripts)), which are meant to solve this problem. - -To ask the playbook to run synapse-janitor, execute: - -```bash -ansible-playbook -i inventory/hosts setup.yml --tags=run-postgres-synapse-janitor,start -``` - -**Note**: this will automatically stop Synapse temporarily and restart it later. - -Running synapse-janitor potentially deletes a lot of data from the Postgres database. -You may wish to run a [`FULL` Postgres `VACUUM`](./maintenance-postgres.md#vacuuming-postgresql) after that. - - ## Browse and manipulate the database When the [matrix admin API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api) and the other tools do not provide a more convenient way, having a look at synapse's postgresql database can satisfy a lot of admins' needs. diff --git a/docs/prerequisites.md b/docs/prerequisites.md index 4356081b5..488f6c6fa 100644 --- a/docs/prerequisites.md +++ b/docs/prerequisites.md @@ -10,7 +10,7 @@ This playbook doesn't support running on ARM (see [this issue](https://github.co - `root` access to your server (or a user capable of elevating to `root` via `sudo`). -- [Python](https://www.python.org/) being installed on the server. Most distributions install Python by default, but some don't (e.g. Ubuntu 18.04) and require manual installation (something like `apt-get install python`). +- [Python](https://www.python.org/) being installed on the server. Most distributions install Python by default, but some don't (e.g. Ubuntu 18.04) and require manual installation (something like `apt-get install python3`). On some distros, Ansible may incorrectly [detect the Python version](https://docs.ansible.com/ansible/latest/reference_appendices/interpreter_discovery.html) (2 vs 3) and you may need to explicitly specify the interpreter path in `inventory/hosts` during installation (e.g. `ansible_python_interpreter=/usr/bin/python3`) - A `cron`-like tool installed on the server such as `cron` or `anacron` to automatically schedule the Let's Encrypt SSL certificates's renewal. *This can be ignored if you use your own SSL certificates.* @@ -22,6 +22,17 @@ This playbook doesn't support running on ARM (see [this issue](https://github.co - Properly configured DNS records for `` (details in [Configuring DNS](configuring-dns.md)). -- Some TCP/UDP ports open. This playbook configures the server's internal firewall for you. In most cases, you don't need to do anything special. But **if your server is running behind another firewall**, you'd need to open these ports: `80/tcp` (HTTP webserver), `443/tcp` (HTTPS webserver), `3478/tcp` (TURN over TCP), `3478/udp` (TURN over UDP), `5349/tcp` (TURN over TCP), `5349/udp` (TURN over UDP), `8448/tcp` (Matrix Federation API HTTPS webserver), the range `49152-49172/udp` (TURN over UDP), `4443/tcp` (Jitsi Harvester fallback), `10000/udp` (Jitsi video RTP). Depending on your firewall/NAT setup, incoming RTP packets on port 10000 may have the external IP of your firewall as destination address, due to the usage of STUN in JVB (see [`matrix_jitsi_jvb_stun_servers`](../roles/matrix-jitsi/defaults/main.yml)). +- Some TCP/UDP ports open. This playbook configures the server's internal firewall for you. In most cases, you don't need to do anything special. But **if your server is running behind another firewall**, you'd need to open these ports: + + - `80/tcp`: HTTP webserver + - `443/tcp`: HTTPS webserver + - `3478/tcp`: TURN over TCP (used by Coturn) + - `3478/udp`: TURN over UDP (used by Coturn) + - `5349/tcp`: TURN over TCP (used by Coturn) + - `5349/udp`: TURN over UDP (used by Coturn) + - `8448/tcp`: Matrix Federation API HTTPS webserver. In some cases, this **may necessary even with federation disabled**. Integration Servers (like Dimension) and Identity Servers (like ma1sd) may need to access `openid` APIs on the federation port. + - the range `49152-49172/udp`: TURN over UDP + - `4443/tcp`: Jitsi Harvester fallback + - `10000/udp`: Jitsi video RTP. Depending on your firewall/NAT setup, incoming RTP packets on port `10000` may have the external IP of your firewall as destination address, due to the usage of STUN in JVB (see [`matrix_jitsi_jvb_stun_servers`](../roles/matrix-jitsi/defaults/main.yml)). When ready to proceed, continue with [Configuring DNS](configuring-dns.md). diff --git a/docs/self-building.md b/docs/self-building.md index 169e4aa50..fa4db2220 100644 --- a/docs/self-building.md +++ b/docs/self-building.md @@ -15,6 +15,7 @@ List of roles where self-building the Docker image is currently possible: - `matrix-client-element` - `matrix-registration` - `matrix-coturn` +- `matrix-corporal` - `matrix-ma1sd` - `matrix-mailer` - `matrix-bridge-mautrix-facebook` diff --git a/docs/updating-users-passwords.md b/docs/updating-users-passwords.md index 785bc17cb..7d2f28329 100644 --- a/docs/updating-users-passwords.md +++ b/docs/updating-users-passwords.md @@ -26,7 +26,7 @@ and then connecting to the postgres server and executing: ``` UPDATE users SET password_hash = '' WHERE name = '@someone:server.com' ``` -` + where `` is the hash returned by the docker command above. diff --git a/examples/hosts b/examples/hosts index 3b14f09e4..daf2cfc5b 100644 --- a/examples/hosts +++ b/examples/hosts @@ -10,6 +10,9 @@ # # If you're running this Ansible playbook on the same server as the one you're installing to, # consider adding an additional `ansible_connection=local` argument below. +# +# Ansible may fail to discover which Python interpreter to use on the host for some distros (like Ubuntu 20.04). +# You may sometimes need to explicitly add `ansible_python_interpreter=/usr/bin/python3` to lines below. [matrix_servers] matrix. ansible_host= ansible_ssh_user=root diff --git a/group_vars/matrix_servers b/group_vars/matrix_servers index 07f9a88c7..b8fc9eb4e 100755 --- a/group_vars/matrix_servers +++ b/group_vars/matrix_servers @@ -24,20 +24,6 @@ matrix_identity_server_url: "{{ ('https://' + matrix_server_fqn_matrix) if matri # ###################################################################### -###################################################################### -# -# matrix-architecture -# -###################################################################### - -matrix_architecture: "amd64" - -###################################################################### -# -# /matrix-architecture -# -###################################################################### - ###################################################################### # @@ -61,12 +47,19 @@ matrix_appservice_discord_systemd_required_services_list: | ['docker.service'] + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) }} matrix_appservice_discord_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'discord.as.token') | to_uuid }}" matrix_appservice_discord_homeserver_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'discord.hs.token') | to_uuid }}" +# We only make this use Postgres if our own Postgres server is enabled. +# It's only then (for now) that we can automatically create the necessary database and user for this service. +matrix_appservice_discord_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_appservice_discord_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'as.discord.db') | to_uuid }}" + ###################################################################### # # /matrix-bridge-appservice-discord @@ -135,6 +128,10 @@ matrix_appservice_slack_systemd_required_services_list: | (['matrix-synapse.service'] if matrix_synapse_enabled else []) }} +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_appservice_slack_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_appservice_slack_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'as.slack.db') | to_uuid }}" + ###################################################################### # # /matrix-bridge-appservice-slack @@ -170,6 +167,10 @@ matrix_appservice_irc_appservice_token: "{{ matrix_synapse_macaroon_secret_key | matrix_appservice_irc_homeserver_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'irc.hs.token') | to_uuid }}" +matrix_appservice_irc_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'nedb' }}" +matrix_appservice_irc_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'as.irc.db') | to_uuid }}" + + ###################################################################### # # /matrix-bridge-appservice-irc @@ -193,6 +194,8 @@ matrix_mautrix_facebook_systemd_required_services_list: | ['docker.service'] + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) }} matrix_mautrix_facebook_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'fb.as.token') | to_uuid }}" @@ -201,6 +204,12 @@ matrix_mautrix_facebook_homeserver_token: "{{ matrix_synapse_macaroon_secret_key matrix_mautrix_facebook_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +matrix_mautrix_facebook_bridge_presence: "{{ matrix_synapse_use_presence if matrix_synapse_enabled else true }}" + +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_mautrix_facebook_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_mautrix_facebook_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mau.fb.db') | to_uuid }}" + ###################################################################### # # /matrix-bridge-mautrix-facebook @@ -224,6 +233,8 @@ matrix_mautrix_hangouts_systemd_required_services_list: | ['docker.service'] + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) }} matrix_mautrix_hangouts_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'ho.as.token') | to_uuid }}" @@ -234,6 +245,10 @@ matrix_mautrix_hangouts_container_http_host_bind_port: "{{ '' if matrix_nginx_pr matrix_mautrix_hangouts_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_mautrix_hangouts_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_mautrix_hangouts_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mau.hangouts.db') | to_uuid }}" + ###################################################################### # # /matrix-bridge-mautrix-hangouts @@ -258,6 +273,8 @@ matrix_mautrix_telegram_systemd_required_services_list: | ['docker.service'] + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) }} matrix_mautrix_telegram_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'telegr.as.token') | to_uuid }}" @@ -270,6 +287,10 @@ matrix_mautrix_telegram_container_http_host_bind_port: "{{ '' if matrix_nginx_pr matrix_mautrix_telegram_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_mautrix_telegram_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_mautrix_telegram_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mau.telegram.db') | to_uuid }}" + ###################################################################### # # /matrix-bridge-mautrix-telegram @@ -290,6 +311,8 @@ matrix_mautrix_whatsapp_systemd_required_services_list: | ['docker.service'] + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) }} matrix_mautrix_whatsapp_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'whats.as.token') | to_uuid }}" @@ -298,6 +321,10 @@ matrix_mautrix_whatsapp_homeserver_token: "{{ matrix_synapse_macaroon_secret_key matrix_mautrix_whatsapp_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_mautrix_whatsapp_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_mautrix_whatsapp_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mauwhatsapp.db') | to_uuid }}" + ###################################################################### # # /matrix-bridge-mautrix-whatsapp @@ -346,6 +373,8 @@ matrix_mx_puppet_skype_systemd_required_services_list: | ['docker.service'] + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) }} matrix_mx_puppet_skype_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'skype.as.tok') | to_uuid }}" @@ -354,6 +383,10 @@ matrix_mx_puppet_skype_homeserver_token: "{{ matrix_synapse_macaroon_secret_key matrix_mx_puppet_skype_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_mx_puppet_skype_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_mx_puppet_skype_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxpup.skype.db') | to_uuid }}" + ###################################################################### # # /matrix-bridge-mx-puppet-skype @@ -377,6 +410,8 @@ matrix_mx_puppet_slack_systemd_required_services_list: | ['docker.service'] + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) }} matrix_mx_puppet_slack_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxslk.as.tok') | to_uuid }}" @@ -385,6 +420,10 @@ matrix_mx_puppet_slack_homeserver_token: "{{ matrix_synapse_macaroon_secret_key matrix_mx_puppet_slack_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_mx_puppet_slack_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_mx_puppet_slack_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxpup.slack.db') | to_uuid }}" + ###################################################################### # # /matrix-bridge-mx-puppet-slack @@ -407,6 +446,8 @@ matrix_mx_puppet_twitter_systemd_required_services_list: | ['docker.service'] + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) }} matrix_mx_puppet_twitter_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxtwt.as.tok') | to_uuid }}" @@ -417,6 +458,10 @@ matrix_mx_puppet_twitter_login_shared_secret: "{{ matrix_synapse_ext_password_pr matrix_mx_puppet_twitter_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_enabled else ('127.0.0.1:' ~ matrix_mx_puppet_twitter_appservice_port) }}" +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_mx_puppet_twitter_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_mx_puppet_twitter_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxpup.twitter.db') | to_uuid }}" + ###################################################################### # # /matrix-bridge-mx-puppet-twitter @@ -440,6 +485,8 @@ matrix_mx_puppet_instagram_systemd_required_services_list: | ['docker.service'] + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) }} matrix_mx_puppet_instagram_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxig.as.tok') | to_uuid }}" @@ -448,6 +495,10 @@ matrix_mx_puppet_instagram_homeserver_token: "{{ matrix_synapse_macaroon_secret_ matrix_mx_puppet_instagram_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_mx_puppet_instagram_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_mx_puppet_instagram_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxpup.ig.db') | to_uuid }}" + ###################################################################### # # /matrix-bridge-mx-puppet-instagram @@ -470,6 +521,8 @@ matrix_mx_puppet_discord_systemd_required_services_list: | ['docker.service'] + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) }} matrix_mx_puppet_discord_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxdsc.as.tok') | to_uuid }}" @@ -478,6 +531,10 @@ matrix_mx_puppet_discord_homeserver_token: "{{ matrix_synapse_macaroon_secret_ke matrix_mx_puppet_discord_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_mx_puppet_discord_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_mx_puppet_discord_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxpup.dsc.db') | to_uuid }}" + ###################################################################### # # /matrix-bridge-mx-puppet-discord @@ -500,6 +557,8 @@ matrix_mx_puppet_steam_systemd_required_services_list: | ['docker.service'] + (['matrix-synapse.service'] if matrix_synapse_enabled else []) + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) }} matrix_mx_puppet_steam_appservice_token: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxste.as.tok') | to_uuid }}" @@ -508,6 +567,10 @@ matrix_mx_puppet_steam_homeserver_token: "{{ matrix_synapse_macaroon_secret_key matrix_mx_puppet_steam_login_shared_secret: "{{ matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret if matrix_synapse_ext_password_provider_shared_secret_auth_enabled else '' }}" +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_mx_puppet_steam_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_mx_puppet_steam_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mxpup.steam.db') | to_uuid }}" + ###################################################################### # # /matrix-bridge-mx-puppet-steam @@ -524,6 +587,17 @@ matrix_mx_puppet_steam_login_shared_secret: "{{ matrix_synapse_ext_password_prov # We don't enable bots by default. matrix_bot_matrix_reminder_bot_enabled: false +matrix_bot_matrix_reminder_bot_systemd_required_services_list: | + {{ + ['docker.service'] + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) + }} + +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_bot_matrix_reminder_bot_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_bot_matrix_reminder_bot_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'reminder.bot.db') | to_uuid }}" + ###################################################################### # # /matrix-bot-matrix-reminder-bot @@ -539,6 +613,8 @@ matrix_bot_matrix_reminder_bot_enabled: false matrix_corporal_enabled: false +matrix_corporal_container_image_self_build: "{{ matrix_architecture != 'amd64' }}" + # Normally, matrix-nginx-proxy is enabled and nginx can reach matrix-corporal over the container network. # If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose # matrix-corporal's web-server ports to the local host. @@ -616,7 +692,16 @@ matrix_dimension_container_http_host_bind_port: "{{ '' if matrix_nginx_proxy_ena matrix_integration_manager_rest_url: "{{ matrix_dimension_integrations_rest_url if matrix_dimension_enabled else None }}" matrix_integration_manager_ui_url: "{{ matrix_dimension_integrations_ui_url if matrix_dimension_enabled else None }}" -matrix_dimension_homeserver_federationUrl: "http://matrix-synapse:{{ 8048 if matrix_synapse_federation_enabled|bool else 8008 }}" +matrix_dimension_systemd_required_services_list: | + {{ + ['docker.service'] + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) + }} + +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_dimension_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_dimension_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'dimension.db') | to_uuid }}" ###################################################################### # @@ -762,6 +847,11 @@ matrix_ma1sd_threepid_medium_email_connectors_smtp_tls: 0 matrix_ma1sd_self_check_validate_certificates: "{{ false if matrix_ssl_retrieval_method == 'self-signed' else true }}" +matrix_ma1sd_systemd_required_services_list: | + {{ + (['matrix-postgres.service'] if matrix_postgres_enabled else []) + }} + matrix_ma1sd_systemd_wanted_services_list: | {{ (['matrix-corporal.service'] if matrix_corporal_enabled else ['matrix-synapse.service']) @@ -771,6 +861,10 @@ matrix_ma1sd_systemd_wanted_services_list: | (['matrix-mailer.service'] if matrix_mailer_enabled else []) }} +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_ma1sd_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_ma1sd_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'ma1sd.db') | to_uuid }}" + ###################################################################### # # /matrix-ma1sd @@ -812,7 +906,8 @@ matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container: "matrix-ma1sd: matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container: "127.0.0.1:8090" # By default, we do TLS termination for the Matrix Federation API (port 8448) at matrix-nginx-proxy. -matrix_nginx_proxy_proxy_matrix_federation_api_enabled: true +# Unless this is handled there OR Synapse's federation listener port is disabled, we'll reverse-proxy. +matrix_nginx_proxy_proxy_matrix_federation_api_enabled: "{{ matrix_synapse_federation_port_enabled and not matrix_synapse_tls_federation_listener_enabled }}" matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container: "matrix-synapse:8048" matrix_nginx_proxy_proxy_matrix_federation_api_addr_sans_container: "127.0.0.1:8048" @@ -886,6 +981,133 @@ matrix_postgres_connection_username: "synapse" matrix_postgres_connection_password: "synapse-password" matrix_postgres_db_name: "homeserver" +matrix_postgres_pgloader_container_image_self_build: "{{ matrix_architecture != 'amd64' }}" + +matrix_postgres_additional_databases: | + {{ + ([{ + 'name': matrix_ma1sd_database_name, + 'username': matrix_ma1sd_database_username, + 'password': matrix_ma1sd_database_password, + }] if (matrix_ma1sd_enabled and matrix_ma1sd_database_engine == 'postgres' and matrix_ma1sd_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_bot_matrix_reminder_bot_database_name, + 'username': matrix_bot_matrix_reminder_bot_database_username, + 'password': matrix_bot_matrix_reminder_bot_database_password, + }] if (matrix_bot_matrix_reminder_bot_enabled and matrix_bot_matrix_reminder_bot_database_engine == 'postgres' and matrix_bot_matrix_reminder_bot_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_registration_database_name, + 'username': matrix_registration_database_username, + 'password': matrix_registration_database_password, + }] if (matrix_registration_enabled and matrix_registration_database_engine == 'postgres' and matrix_registration_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_appservice_discord_database_name, + 'username': matrix_appservice_discord_database_username, + 'password': matrix_appservice_discord_database_password, + }] if (matrix_appservice_discord_enabled and matrix_appservice_discord_database_engine == 'postgres' and matrix_appservice_discord_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_appservice_slack_database_name, + 'username': matrix_appservice_slack_database_username, + 'password': matrix_appservice_slack_database_password, + }] if (matrix_appservice_slack_enabled and matrix_appservice_slack_database_engine == 'postgres' and matrix_appservice_slack_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_appservice_irc_database_name, + 'username': matrix_appservice_irc_database_username, + 'password': matrix_appservice_irc_database_password, + }] if (matrix_appservice_irc_enabled and matrix_appservice_irc_database_engine == 'postgres' and matrix_appservice_irc_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_mautrix_facebook_database_name, + 'username': matrix_mautrix_facebook_database_username, + 'password': matrix_mautrix_facebook_database_password, + }] if (matrix_mautrix_facebook_enabled and matrix_mautrix_facebook_database_engine == 'postgres' and matrix_mautrix_facebook_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_mautrix_hangouts_database_name, + 'username': matrix_mautrix_hangouts_database_username, + 'password': matrix_mautrix_hangouts_database_password, + }] if (matrix_mautrix_hangouts_enabled and matrix_mautrix_hangouts_database_engine == 'postgres' and matrix_mautrix_hangouts_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_mautrix_telegram_database_name, + 'username': matrix_mautrix_telegram_database_username, + 'password': matrix_mautrix_telegram_database_password, + }] if (matrix_mautrix_telegram_enabled and matrix_mautrix_telegram_database_engine == 'postgres' and matrix_mautrix_telegram_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_mautrix_whatsapp_database_name, + 'username': matrix_mautrix_whatsapp_database_username, + 'password': matrix_mautrix_whatsapp_database_password, + }] if (matrix_mautrix_whatsapp_enabled and matrix_mautrix_whatsapp_database_engine == 'postgres' and matrix_mautrix_whatsapp_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': 'matrix_bridge_sms', + 'username': 'matrix_bridge_sms', + 'password': matrix_synapse_macaroon_secret_key | password_hash('sha512', 'bridge.sms.db') | to_uuid, + }] if matrix_sms_bridge_enabled else []) + + + ([{ + 'name': matrix_mx_puppet_skype_database_name, + 'username': matrix_mx_puppet_skype_database_username, + 'password': matrix_mx_puppet_skype_database_password, + }] if (matrix_mx_puppet_skype_enabled and matrix_mx_puppet_skype_database_engine == 'postgres' and matrix_mx_puppet_skype_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_mx_puppet_slack_database_name, + 'username': matrix_mx_puppet_slack_database_username, + 'password': matrix_mx_puppet_slack_database_password, + }] if (matrix_mx_puppet_slack_enabled and matrix_mx_puppet_slack_database_engine == 'postgres' and matrix_mx_puppet_slack_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_mx_puppet_twitter_database_name, + 'username': matrix_mx_puppet_twitter_database_username, + 'password': matrix_mx_puppet_twitter_database_password, + }] if (matrix_mx_puppet_twitter_enabled and matrix_mx_puppet_twitter_database_engine == 'postgres' and matrix_mx_puppet_twitter_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_mx_puppet_instagram_database_name, + 'username': matrix_mx_puppet_instagram_database_username, + 'password': matrix_mx_puppet_instagram_database_password, + }] if (matrix_mx_puppet_instagram_enabled and matrix_mx_puppet_instagram_database_engine == 'postgres' and matrix_mx_puppet_instagram_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_mx_puppet_discord_database_name, + 'username': matrix_mx_puppet_discord_database_username, + 'password': matrix_mx_puppet_discord_database_password, + }] if (matrix_mx_puppet_discord_enabled and matrix_mx_puppet_discord_database_engine == 'postgres' and matrix_mx_puppet_discord_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_mx_puppet_steam_database_name, + 'username': matrix_mx_puppet_steam_database_username, + 'password': matrix_mx_puppet_steam_database_password, + }] if (matrix_mx_puppet_steam_enabled and matrix_mx_puppet_steam_database_engine == 'postgres' and matrix_mx_puppet_steam_database_hostname == 'matrix-postgres') else []) + + + ([{ + 'name': matrix_dimension_database_name, + 'username': matrix_dimension_database_username, + 'password': matrix_dimension_database_password, + }] if (matrix_dimension_enabled and matrix_dimension_database_engine == 'postgres' and matrix_dimension_database_hostname == 'matrix-postgres') else []) + }} + +matrix_postgres_import_roles_to_ignore: | + {{ + [matrix_postgres_connection_username] + + + matrix_postgres_additional_databases|map(attribute='username')|list + }} + +matrix_postgres_import_databases_to_ignore: | + {{ + [matrix_postgres_db_name] + + + matrix_postgres_additional_databases|map(attribute='name')|list + }} + ###################################################################### # # /matrix-postgres @@ -986,12 +1208,7 @@ matrix_synapse_tls_federation_listener_enabled: false matrix_synapse_tls_certificate_path: ~ matrix_synapse_tls_private_key_path: ~ -matrix_synapse_http_listener_resource_names: | - {{ - ["client"] - + - ( ["openid"] if matrix_dimension_enabled and not matrix_synapse_federation_enabled else [] ) - }} +matrix_synapse_federation_port_openid_resource_required: "{{ not matrix_synapse_federation_enabled and (matrix_dimension_enabled or matrix_ma1sd_enabled) }}" matrix_synapse_email_enabled: "{{ matrix_mailer_enabled }}" matrix_synapse_email_smtp_host: "matrix-mailer" @@ -1087,6 +1304,17 @@ matrix_registration_api_validate_certs: "{{ false if matrix_ssl_retrieval_method matrix_registration_container_image_self_build: "{{ matrix_architecture != 'amd64' }}" +matrix_registration_systemd_required_services_list: | + {{ + ['docker.service'] + + + (['matrix-postgres.service'] if matrix_postgres_enabled else []) + }} + +# Postgres is the default, except if not using `matrix_postgres` (internal postgres) +matrix_registration_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}" +matrix_registration_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mx.registr.db') | to_uuid }}" + ###################################################################### # # /matrix-registration diff --git a/roles/matrix-base/defaults/main.yml b/roles/matrix-base/defaults/main.yml index 164509b75..2cc4b42f8 100644 --- a/roles/matrix-base/defaults/main.yml +++ b/roles/matrix-base/defaults/main.yml @@ -23,6 +23,17 @@ matrix_server_fqn_jitsi: "jitsi.{{ matrix_domain }}" matrix_federation_public_port: 8448 +# The architecture that your server runs. +# Recognized values by us are 'amd64', 'arm32' and 'arm64'. +# Not all architectures support all services, so your experience (on non-amd64) may vary. +# See docs/alternative-architectures.md +matrix_architecture: amd64 + +# The architecture for Debian packages. +# See: https://wiki.debian.org/SupportedArchitectures +# We just remap from our `matrix_architecture` values to what Debian and possibly other distros call things. +matrix_debian_arch: "{{ 'armhf' if matrix_architecture == 'arm32' else matrix_architecture }}" + matrix_user_username: "matrix" matrix_user_groupname: "matrix" @@ -106,7 +117,6 @@ matrix_docker_package_name: docker-ce run_postgres_import: true run_postgres_upgrade: true run_postgres_import_sqlite_db: true -run_postgres_synapse_janitor: true run_postgres_vacuum: true run_synapse_register_user: true run_synapse_update_user_password: true diff --git a/roles/matrix-base/files/yum.repos.d/docker-ce.repo b/roles/matrix-base/files/yum.repos.d/docker-ce.repo index 56242d984..1abdbe367 100644 --- a/roles/matrix-base/files/yum.repos.d/docker-ce.repo +++ b/roles/matrix-base/files/yum.repos.d/docker-ce.repo @@ -1,62 +1,62 @@ [docker-ce-stable] name=Docker CE Stable - $basearch -baseurl=https://download.docker.com/linux/centos/7/$basearch/stable +baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/stable enabled=1 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg [docker-ce-stable-debuginfo] name=Docker CE Stable - Debuginfo $basearch -baseurl=https://download.docker.com/linux/centos/7/debug-$basearch/stable +baseurl=https://download.docker.com/linux/centos/$releasever/debug-$basearch/stable enabled=0 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg [docker-ce-stable-source] name=Docker CE Stable - Sources -baseurl=https://download.docker.com/linux/centos/7/source/stable -enabled=0 -gpgcheck=1 -gpgkey=https://download.docker.com/linux/centos/gpg - -[docker-ce-edge] -name=Docker CE Edge - $basearch -baseurl=https://download.docker.com/linux/centos/7/$basearch/edge -enabled=0 -gpgcheck=1 -gpgkey=https://download.docker.com/linux/centos/gpg - -[docker-ce-edge-debuginfo] -name=Docker CE Edge - Debuginfo $basearch -baseurl=https://download.docker.com/linux/centos/7/debug-$basearch/edge -enabled=0 -gpgcheck=1 -gpgkey=https://download.docker.com/linux/centos/gpg - -[docker-ce-edge-source] -name=Docker CE Edge - Sources -baseurl=https://download.docker.com/linux/centos/7/source/edge +baseurl=https://download.docker.com/linux/centos/$releasever/source/stable enabled=0 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg [docker-ce-test] name=Docker CE Test - $basearch -baseurl=https://download.docker.com/linux/centos/7/$basearch/test +baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/test enabled=0 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg [docker-ce-test-debuginfo] name=Docker CE Test - Debuginfo $basearch -baseurl=https://download.docker.com/linux/centos/7/debug-$basearch/test +baseurl=https://download.docker.com/linux/centos/$releasever/debug-$basearch/test enabled=0 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg [docker-ce-test-source] name=Docker CE Test - Sources -baseurl=https://download.docker.com/linux/centos/7/source/test +baseurl=https://download.docker.com/linux/centos/$releasever/source/test +enabled=0 +gpgcheck=1 +gpgkey=https://download.docker.com/linux/centos/gpg + +[docker-ce-nightly] +name=Docker CE Nightly - $basearch +baseurl=https://download.docker.com/linux/centos/$releasever/$basearch/nightly +enabled=0 +gpgcheck=1 +gpgkey=https://download.docker.com/linux/centos/gpg + +[docker-ce-nightly-debuginfo] +name=Docker CE Nightly - Debuginfo $basearch +baseurl=https://download.docker.com/linux/centos/$releasever/debug-$basearch/nightly +enabled=0 +gpgcheck=1 +gpgkey=https://download.docker.com/linux/centos/gpg + +[docker-ce-nightly-source] +name=Docker CE Nightly - Sources +baseurl=https://download.docker.com/linux/centos/$releasever/source/nightly enabled=0 gpgcheck=1 gpgkey=https://download.docker.com/linux/centos/gpg diff --git a/roles/matrix-base/tasks/server_base/setup_debian.yml b/roles/matrix-base/tasks/server_base/setup_debian.yml index 6d8d18fa3..37706d1f9 100644 --- a/roles/matrix-base/tasks/server_base/setup_debian.yml +++ b/roles/matrix-base/tasks/server_base/setup_debian.yml @@ -11,7 +11,7 @@ - name: Ensure Docker's APT key is trusted apt_key: - url: https://download.docker.com/linux/ubuntu/gpg + url: "https://download.docker.com/linux/{{ ansible_distribution|lower }}/gpg" id: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88 state: present register: add_repository_key @@ -20,7 +20,7 @@ - name: Ensure Docker repository is enabled apt_repository: - repo: "deb [arch=amd64] https://download.docker.com/linux/{{ ansible_distribution|lower }} {{ ansible_distribution_release }} stable" + repo: "deb [arch={{ matrix_debian_arch }}] https://download.docker.com/linux/{{ ansible_distribution|lower }} {{ ansible_distribution_release }} stable" state: present update_cache: yes when: matrix_docker_installation_enabled|bool and matrix_docker_package_name == 'docker-ce' diff --git a/roles/matrix-base/tasks/server_base/setup_raspbian.yml b/roles/matrix-base/tasks/server_base/setup_raspbian.yml index a9a5d20af..421905a0d 100644 --- a/roles/matrix-base/tasks/server_base/setup_raspbian.yml +++ b/roles/matrix-base/tasks/server_base/setup_raspbian.yml @@ -5,6 +5,7 @@ name: - apt-transport-https - ca-certificates + - gnupg state: present update_cache: yes @@ -19,7 +20,7 @@ - name: Ensure Docker repository is enabled apt_repository: - repo: "deb [arch=armhf] https://download.docker.com/linux/raspbian {{ ansible_distribution_release }} stable" + repo: "deb [arch={{ matrix_debian_arch }}] https://download.docker.com/linux/raspbian {{ ansible_distribution_release }} stable" state: present update_cache: yes when: matrix_docker_installation_enabled|bool and matrix_docker_package_name == 'docker-ce' @@ -27,7 +28,6 @@ - name: Ensure APT packages are installed apt: name: - - python-docker - "{{ matrix_ntpd_package }}" - fuse state: latest @@ -37,5 +37,6 @@ apt: name: - "{{ matrix_docker_package_name }}" + - "python{{'3' if ansible_python.version.major == 3 else ''}}-docker" state: latest when: matrix_docker_installation_enabled|bool diff --git a/roles/matrix-bot-matrix-reminder-bot/defaults/main.yml b/roles/matrix-bot-matrix-reminder-bot/defaults/main.yml index 330281473..29bc83073 100644 --- a/roles/matrix-bot-matrix-reminder-bot/defaults/main.yml +++ b/roles/matrix-bot-matrix-reminder-bot/defaults/main.yml @@ -21,6 +21,34 @@ matrix_bot_matrix_reminder_bot_systemd_required_services_list: ['docker.service' matrix_bot_matrix_reminder_bot_systemd_wanted_services_list: [] +# Database-related configuration fields. +# +# To use SQLite, stick to these defaults. +# +# To use Postgres: +# - change the engine (`matrix_bot_matrix_reminder_bot_database_engine: 'postgres'`) +# - adjust your database credentials via the `matrix_bot_matrix_reminder_bot_database_*` variables +matrix_bot_matrix_reminder_bot_database_engine: 'sqlite' + +matrix_bot_matrix_reminder_bot_sqlite_database_path_local: "{{ matrix_bot_matrix_reminder_bot_data_path }}/bot.db" +matrix_bot_matrix_reminder_bot_sqlite_database_path_in_container: "/data/bot.db" + +matrix_bot_matrix_reminder_bot_database_username: 'matrix_reminder_bot' +matrix_bot_matrix_reminder_bot_database_password: 'some-password' +matrix_bot_matrix_reminder_bot_database_hostname: 'matrix-postgres' +matrix_bot_matrix_reminder_bot_database_port: 5432 +matrix_bot_matrix_reminder_bot_database_name: 'matrix_reminder_bot' + +matrix_bot_matrix_reminder_bot_database_connection_string: 'postgres://{{ matrix_bot_matrix_reminder_bot_database_username }}:{{ matrix_bot_matrix_reminder_bot_database_password }}@{{ matrix_bot_matrix_reminder_bot_database_hostname }}:{{ matrix_bot_matrix_reminder_bot_database_port }}/{{ matrix_bot_matrix_reminder_bot_database_name }}' + +matrix_bot_matrix_reminder_bot_storage_database: "{{ + { + 'sqlite': ('sqlite://' + matrix_bot_matrix_reminder_bot_sqlite_database_path_in_container), + 'postgres': matrix_bot_matrix_reminder_bot_database_connection_string, + }[matrix_bot_matrix_reminder_bot_database_engine] +}}" + + # The bot's username. This user needs to be created manually beforehand. # Also see `matrix_bot_matrix_reminder_bot_user_password`. matrix_bot_matrix_reminder_bot_matrix_user_id_localpart: "bot.matrix-reminder-bot" diff --git a/roles/matrix-bot-matrix-reminder-bot/tasks/main.yml b/roles/matrix-bot-matrix-reminder-bot/tasks/main.yml index ade3d1911..fc2afddb2 100644 --- a/roles/matrix-bot-matrix-reminder-bot/tasks/main.yml +++ b/roles/matrix-bot-matrix-reminder-bot/tasks/main.yml @@ -8,7 +8,14 @@ - setup-all - setup-bot-matrix-reminder-bot -- import_tasks: "{{ role_path }}/tasks/setup.yml" +- import_tasks: "{{ role_path }}/tasks/setup_install.yml" + when: "run_setup|bool and matrix_bot_matrix_reminder_bot_enabled|bool" + tags: + - setup-all + - setup-bot-matrix-reminder-bot + +- import_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + when: "run_setup|bool and not matrix_bot_matrix_reminder_bot_enabled|bool" tags: - setup-all - setup-bot-matrix-reminder-bot diff --git a/roles/matrix-bot-matrix-reminder-bot/tasks/setup.yml b/roles/matrix-bot-matrix-reminder-bot/tasks/setup.yml deleted file mode 100644 index bc211e310..000000000 --- a/roles/matrix-bot-matrix-reminder-bot/tasks/setup.yml +++ /dev/null @@ -1,88 +0,0 @@ ---- - -# -# Tasks related to setting up matrix-reminder-bot -# - -- name: Ensure matrix-reminder-bot paths exist - file: - path: "{{ item.path }}" - state: directory - mode: 0750 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - with_items: - - { path: "{{ matrix_bot_matrix_reminder_bot_config_path }}", when: true } - - { path: "{{ matrix_bot_matrix_reminder_bot_data_path }}", when: true } - - { path: "{{ matrix_bot_matrix_reminder_bot_data_store_path }}", when: true } - when: matrix_bot_matrix_reminder_bot_enabled|bool and item.when - -- name: Ensure matrix-reminder-bot image is pulled - docker_image: - name: "{{ matrix_bot_matrix_reminder_bot_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_bot_matrix_reminder_bot_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_bot_matrix_reminder_bot_docker_image_force_pull }}" - when: matrix_bot_matrix_reminder_bot_enabled|bool - -- name: Ensure matrix-reminder-bot config installed - copy: - content: "{{ matrix_bot_matrix_reminder_bot_configuration|to_nice_yaml }}" - dest: "{{ matrix_bot_matrix_reminder_bot_config_path }}/config.yaml" - mode: 0644 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - when: matrix_bot_matrix_reminder_bot_enabled|bool - -- name: Ensure matrix-matrix-reminder-bot.service installed - template: - src: "{{ role_path }}/templates/systemd/matrix-bot-matrix-reminder-bot.service.j2" - dest: "{{ matrix_systemd_path }}/matrix-bot-matrix-reminder-bot.service" - mode: 0644 - register: matrix_bot_matrix_reminder_bot_systemd_service_result - when: matrix_bot_matrix_reminder_bot_enabled|bool - -- name: Ensure systemd reloaded after matrix-matrix-reminder-bot.service installation - service: - daemon_reload: yes - when: "matrix_bot_matrix_reminder_bot_enabled|bool and matrix_bot_matrix_reminder_bot_systemd_service_result.changed" - -# -# Tasks related to getting rid of matrix-reminder-bot (if it was previously enabled) -# - -- name: Check existence of matrix-matrix-reminder-bot service - stat: - path: "{{ matrix_systemd_path }}/matrix-matrix-reminder-bot.service" - register: matrix_bot_matrix_reminder_bot_service_stat - -- name: Ensure matrix-matrix-reminder-bot is stopped - service: - name: matrix-matrix-reminder-bot - state: stopped - daemon_reload: yes - register: stopping_result - when: "not matrix_bot_matrix_reminder_bot_enabled|bool and matrix_bot_matrix_reminder_bot_service_stat.stat.exists" - -- name: Ensure matrix-matrix-reminder-bot.service doesn't exist - file: - path: "{{ matrix_systemd_path }}/matrix-matrix-reminder-bot.service" - state: absent - when: "not matrix_bot_matrix_reminder_bot_enabled|bool and matrix_bot_matrix_reminder_bot_service_stat.stat.exists" - -- name: Ensure systemd reloaded after matrix-matrix-reminder-bot.service removal - service: - daemon_reload: yes - when: "not matrix_bot_matrix_reminder_bot_enabled|bool and matrix_bot_matrix_reminder_bot_service_stat.stat.exists" - -- name: Ensure Matrix matrix-reminder-bot paths don't exist - file: - path: "{{ matrix_bot_matrix_reminder_bot_base_path }}" - state: absent - when: "not matrix_bot_matrix_reminder_bot_enabled|bool" - -- name: Ensure matrix-reminder-bot Docker image doesn't exist - docker_image: - name: "{{ matrix_bot_matrix_reminder_bot_docker_image }}" - state: absent - when: "not matrix_bot_matrix_reminder_bot_enabled|bool" diff --git a/roles/matrix-bot-matrix-reminder-bot/tasks/setup_install.yml b/roles/matrix-bot-matrix-reminder-bot/tasks/setup_install.yml new file mode 100644 index 000000000..195485e43 --- /dev/null +++ b/roles/matrix-bot-matrix-reminder-bot/tasks/setup_install.yml @@ -0,0 +1,73 @@ +--- + +- set_fact: + matrix_bot_matrix_reminder_bot_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_bot_matrix_reminder_bot_sqlite_database_path_local }}" + register: matrix_bot_matrix_reminder_bot_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_bot_matrix_reminder_bot_sqlite_database_path_local }}" + dst: "{{ matrix_bot_matrix_reminder_bot_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_bot_matrix_reminder_bot_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-bot-matrix-reminder-bot.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_bot_matrix_reminder_bot_requires_restart: true + when: "matrix_bot_matrix_reminder_bot_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_bot_matrix_reminder_bot_database_engine == 'postgres'" + +- name: Ensure matrix-reminder-bot paths exist + file: + path: "{{ item.path }}" + state: directory + mode: 0750 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + with_items: + - { path: "{{ matrix_bot_matrix_reminder_bot_config_path }}", when: true } + - { path: "{{ matrix_bot_matrix_reminder_bot_data_path }}", when: true } + - { path: "{{ matrix_bot_matrix_reminder_bot_data_store_path }}", when: true } + when: "item.when|bool" + +- name: Ensure matrix-reminder-bot image is pulled + docker_image: + name: "{{ matrix_bot_matrix_reminder_bot_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_bot_matrix_reminder_bot_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_bot_matrix_reminder_bot_docker_image_force_pull }}" + +- name: Ensure matrix-reminder-bot config installed + copy: + content: "{{ matrix_bot_matrix_reminder_bot_configuration|to_nice_yaml }}" + dest: "{{ matrix_bot_matrix_reminder_bot_config_path }}/config.yaml" + mode: 0644 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_user_groupname }}" + +- name: Ensure matrix-bot-matrix-reminder-bot.service installed + template: + src: "{{ role_path }}/templates/systemd/matrix-bot-matrix-reminder-bot.service.j2" + dest: "{{ matrix_systemd_path }}/matrix-bot-matrix-reminder-bot.service" + mode: 0644 + register: matrix_bot_matrix_reminder_bot_systemd_service_result + +- name: Ensure systemd reloaded after matrix-bot-matrix-reminder-bot.service installation + service: + daemon_reload: yes + when: "matrix_bot_matrix_reminder_bot_systemd_service_result.changed|bool" + +- name: Ensure matrix-bot-matrix-reminder-bot.service restarted, if necessary + service: + name: "matrix-bot-matrix-reminder-bot.service" + state: restarted + when: "matrix_bot_matrix_reminder_bot_requires_restart|bool" diff --git a/roles/matrix-bot-matrix-reminder-bot/tasks/setup_uninstall.yml b/roles/matrix-bot-matrix-reminder-bot/tasks/setup_uninstall.yml new file mode 100644 index 000000000..744f474d1 --- /dev/null +++ b/roles/matrix-bot-matrix-reminder-bot/tasks/setup_uninstall.yml @@ -0,0 +1,35 @@ +--- + +- name: Check existence of matrix-matrix-reminder-bot service + stat: + path: "{{ matrix_systemd_path }}/matrix-bot-matrix-reminder-bot.service" + register: matrix_bot_matrix_reminder_bot_service_stat + +- name: Ensure matrix-matrix-reminder-bot is stopped + service: + name: matrix-matrix-reminder-bot + state: stopped + daemon_reload: yes + register: stopping_result + when: "matrix_bot_matrix_reminder_bot_service_stat.stat.exists|bool" + +- name: Ensure matrix-bot-matrix-reminder-bot.service doesn't exist + file: + path: "{{ matrix_systemd_path }}/matrix-bot-matrix-reminder-bot.service" + state: absent + when: "matrix_bot_matrix_reminder_bot_service_stat.stat.exists|bool" + +- name: Ensure systemd reloaded after matrix-bot-matrix-reminder-bot.service removal + service: + daemon_reload: yes + when: "matrix_bot_matrix_reminder_bot_service_stat.stat.exists|bool" + +- name: Ensure Matrix matrix-reminder-bot paths don't exist + file: + path: "{{ matrix_bot_matrix_reminder_bot_base_path }}" + state: absent + +- name: Ensure matrix-reminder-bot Docker image doesn't exist + docker_image: + name: "{{ matrix_bot_matrix_reminder_bot_docker_image }}" + state: absent diff --git a/roles/matrix-bot-matrix-reminder-bot/templates/config.yaml.j2 b/roles/matrix-bot-matrix-reminder-bot/templates/config.yaml.j2 index e5ba56517..596439584 100644 --- a/roles/matrix-bot-matrix-reminder-bot/templates/config.yaml.j2 +++ b/roles/matrix-bot-matrix-reminder-bot/templates/config.yaml.j2 @@ -23,7 +23,7 @@ storage: # For Postgres, this would look like: # database: "postgres://username:password@localhost/dbname?sslmode=disable" #database: "postgres://matrix-reminder-bot:remindme@localhost/matrix-reminder-bot?sslmode=disable" - database: "sqlite:///data/bot.db" + database: {{ matrix_bot_matrix_reminder_bot_storage_database|to_json }} # The path to a directory for internal bot storage # containing encryption keys, sync tokens, etc. store_path: "/data/store" diff --git a/roles/matrix-bot-matrix-reminder-bot/templates/systemd/matrix-bot-matrix-reminder-bot.service.j2 b/roles/matrix-bot-matrix-reminder-bot/templates/systemd/matrix-bot-matrix-reminder-bot.service.j2 index 23493c543..950242e57 100644 --- a/roles/matrix-bot-matrix-reminder-bot/templates/systemd/matrix-bot-matrix-reminder-bot.service.j2 +++ b/roles/matrix-bot-matrix-reminder-bot/templates/systemd/matrix-bot-matrix-reminder-bot.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_bot_matrix_reminder_bot_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-appservice-discord/defaults/main.yml b/roles/matrix-bridge-appservice-discord/defaults/main.yml index d03ebb92d..c7cdddb6e 100644 --- a/roles/matrix-bridge-appservice-discord/defaults/main.yml +++ b/roles/matrix-bridge-appservice-discord/defaults/main.yml @@ -3,7 +3,7 @@ matrix_appservice_discord_enabled: true -matrix_appservice_discord_docker_image: "docker.io/halfshot/matrix-appservice-discord:latest" +matrix_appservice_discord_docker_image: "docker.io/halfshot/matrix-appservice-discord:v1.0.0" matrix_appservice_discord_docker_image_force_pull: "{{ matrix_appservice_discord_docker_image.endswith(':latest') }}" matrix_appservice_discord_base_path: "{{ matrix_base_data_path }}/appservice-discord" @@ -17,6 +17,8 @@ matrix_appservice_discord_bot_token: '' matrix_appservice_discord_appservice_token: '' matrix_appservice_discord_homeserver_token: '' +matrix_appservice_discord_homeserver_domain: "{{ matrix_domain }}" + # Controls whether the matrix-appservice-discord container exposes its HTTP port (tcp/9005 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:9005"), or empty string to not expose. @@ -39,6 +41,38 @@ matrix_appservice_discord_bridge_homeserverUrl: "{{ matrix_homeserver_url }}" matrix_appservice_discord_bridge_disablePresence: false matrix_appservice_discord_bridge_enableSelfServiceBridging: false +# Database-related configuration fields. +# +# To use SQLite, stick to these defaults. +# +# To use Postgres: +# - change the engine (`matrix_appservice_discord_database_engine: 'postgres'`) +# - adjust your database credentials via the `matrix_appservice_discord_postgres_*` variables +matrix_appservice_discord_database_engine: 'sqlite' + +matrix_appservice_discord_sqlite_database_path_local: "{{ matrix_appservice_discord_data_path }}/discord.db" +matrix_appservice_discord_sqlite_database_path_in_container: "/data/discord.db" + +matrix_appservice_discord_database_username: 'matrix_appservice_discord' +matrix_appservice_discord_database_password: 'some-password' +matrix_appservice_discord_database_hostname: 'matrix-postgres' +matrix_appservice_discord_database_port: 5432 +matrix_appservice_discord_database_name: 'matrix_appservice_discord' + +# These 2 variables are what actually ends up in the bridge configuration. +# It's best if you don't change them directly, but rather redefine the sub-variables that constitute them. +matrix_appservice_discord_database_filename: "{{ matrix_appservice_discord_sqlite_database_path_in_container }}" +matrix_appservice_discord_database_connString: 'postgresql://{{ matrix_appservice_discord_database_username }}:{{ matrix_appservice_discord_database_password }}@{{ matrix_appservice_discord_database_hostname }}:{{ matrix_appservice_discord_database_port }}/{{ matrix_appservice_discord_database_name }}' + + +# Tells whether the bot should make use of "Privileged Gateway Intents". +# +# Enabling this means that you need to enable it for the bot (Discord application) as well, +# by triggering all Intent checkboxes on a page like this: `https://discord.com/developers/applications/694448564151123988/bot` +# +# Learn more: https://gist.github.com/advaith1/e69bcc1cdd6d0087322734451f15aa2f +matrix_appservice_discord_auth_usePrivilegedIntents: false + matrix_appservice_discord_configuration_yaml: "{{ lookup('template', 'templates/config.yaml.j2') }}" matrix_appservice_discord_configuration_extension_yaml: | @@ -62,10 +96,10 @@ matrix_appservice_discord_registration_yaml: | namespaces: users: - exclusive: true - regex: '^@_discord_.*' + regex: '@_discord_.*:{{ matrix_appservice_discord_homeserver_domain|regex_escape }}' aliases: - exclusive: true - regex: '^#_discord_.*' + regex: '#_discord_.*:{{ matrix_appservice_discord_homeserver_domain|regex_escape }}' url: {{ matrix_appservice_discord_appservice_url }} sender_localpart: _discord_bot rate_limited: false diff --git a/roles/matrix-bridge-appservice-discord/tasks/setup_install.yml b/roles/matrix-bridge-appservice-discord/tasks/setup_install.yml index 935d07ebf..6d3fdd0ff 100644 --- a/roles/matrix-bridge-appservice-discord/tasks/setup_install.yml +++ b/roles/matrix-bridge-appservice-discord/tasks/setup_install.yml @@ -1,5 +1,31 @@ --- +- set_fact: + matrix_appservice_discord_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_appservice_discord_sqlite_database_path_local }}" + register: matrix_appservice_discord_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_appservice_discord_sqlite_database_path_local }}" + dst: "{{ matrix_appservice_discord_database_connString }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_appservice_discord_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-appservice-discord.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_appservice_discord_requires_restart: true + when: "matrix_appservice_discord_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_appservice_discord_database_engine == 'postgres'" + - name: Ensure Appservice Discord image is pulled docker_image: name: "{{ matrix_appservice_discord_docker_image }}" @@ -80,3 +106,9 @@ service: daemon_reload: yes when: "matrix_appservice_discord_systemd_service_result.changed" + +- name: Ensure matrix-appservice-discord.service restarted, if necessary + service: + name: "matrix-appservice-discord.service" + state: restarted + when: "matrix_appservice_discord_requires_restart|bool" diff --git a/roles/matrix-bridge-appservice-discord/tasks/validate_config.yml b/roles/matrix-bridge-appservice-discord/tasks/validate_config.yml index 46612ba53..73253ba03 100644 --- a/roles/matrix-bridge-appservice-discord/tasks/validate_config.yml +++ b/roles/matrix-bridge-appservice-discord/tasks/validate_config.yml @@ -10,6 +10,7 @@ - "matrix_appservice_discord_bot_token" - "matrix_appservice_discord_appservice_token" - "matrix_appservice_discord_homeserver_token" + - "matrix_appservice_discord_homeserver_domain" - name: (Deprecation) Catch and report renamed appservice-discord variables fail: @@ -19,3 +20,7 @@ when: "item.old in vars" with_items: - {'old': 'matrix_appservice_discord_container_expose_client_server_api_port', 'new': ''} + +- name: Require a valid database engine + fail: msg="`matrix_appservice_discord_database_engine` needs to be either 'sqlite' or 'postgres'" + when: "matrix_appservice_discord_database_engine not in ['sqlite', 'postgres']" diff --git a/roles/matrix-bridge-appservice-discord/templates/config.yaml.j2 b/roles/matrix-bridge-appservice-discord/templates/config.yaml.j2 index e91c60ded..b2ecd1981 100644 --- a/roles/matrix-bridge-appservice-discord/templates/config.yaml.j2 +++ b/roles/matrix-bridge-appservice-discord/templates/config.yaml.j2 @@ -26,10 +26,17 @@ bridge: disableReadReceipts: false # Disable Join Leave echos from matrix disableJoinLeaveNotifications: false + # Disable Invite echos from matrix + disableInviteNotifications: false + # Auto-determine the language of code blocks (this can be CPU-intensive) + determineCodeLanguage: false # Authentication configuration for the discord bot. auth: clientID: {{ matrix_appservice_discord_client_id|string|to_json }} botToken: {{ matrix_appservice_discord_bot_token }} + # You must enable "Privileged Gateway Intents" in your bot settings on discord.com (e.g. https://discord.com/developers/applications/12345/bot) + # for this to work + usePrivilegedIntents: {{ matrix_appservice_discord_auth_usePrivilegedIntents|to_json }} logging: # What level should the logger output to the console at. console: "warn" #silly, verbose, info, http, warn, error, silent @@ -45,16 +52,17 @@ logging: # enable: # - "DiscordBot" database: - userStorePath: "/data/user-store.db" - roomStorePath: "/data/room-store.db" # You may either use SQLite or Postgresql for the bridge database, which contains # important mappings for events and user puppeting configurations. # Use the filename option for SQLite, or connString for Postgresql. # If you are migrating, see https://github.com/Half-Shot/matrix-appservice-discord/blob/master/docs/howto.md#migrate-to-postgres-from-sqlite # WARNING: You will almost certainly be fine with sqlite unless your bridge # is in heavy demand and you suffer from IO slowness. - filename: "/data/discord.db" - # connString: "postgresql://user:password@localhost/database_name" + {% if matrix_appservice_discord_database_engine == 'sqlite' %} + filename: {{ matrix_appservice_discord_database_filename|to_json }} + {% else %} + connString: {{ matrix_appservice_discord_database_connString|to_json }} + {% endif %} room: # Set the default visibility of alias rooms, defaults to "public". # One of: "public", "private" @@ -82,10 +90,12 @@ channel: limits: # Delay in milliseconds between discord users joining a room. roomGhostJoinDelay: 6000 - # Delay in milliseconds before sending messages to discord to avoid echos. - # (Copies of a sent message may arrive from discord before we've + # Lock timeout in milliseconds before sending messages to discord to avoid + # echos. Default is rather high as the lock will most likely time out + # before anyways. + # echos = (Copies of a sent message may arrive from discord before we've # fininished handling it, causing us to echo it back to the room) - discordSendDelay: 750 + discordSendDelay: 1500 ghosts: # Pattern for the ghosts nick, available is :nick, :username, :tag and :id nickPattern: ":nick" diff --git a/roles/matrix-bridge-appservice-discord/templates/systemd/matrix-appservice-discord.service.j2 b/roles/matrix-bridge-appservice-discord/templates/systemd/matrix-appservice-discord.service.j2 index 6a44a3e6d..7ec15752b 100644 --- a/roles/matrix-bridge-appservice-discord/templates/systemd/matrix-appservice-discord.service.j2 +++ b/roles/matrix-bridge-appservice-discord/templates/systemd/matrix-appservice-discord.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_appservice_discord_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-appservice-irc/defaults/main.yml b/roles/matrix-bridge-appservice-irc/defaults/main.yml index 7ff2d46a3..0b671e762 100644 --- a/roles/matrix-bridge-appservice-irc/defaults/main.yml +++ b/roles/matrix-bridge-appservice-irc/defaults/main.yml @@ -16,6 +16,25 @@ matrix_appservice_irc_homeserver_domain: '{{ matrix_domain }}' matrix_appservice_irc_homeserver_enablePresence: true matrix_appservice_irc_appservice_address: 'http://matrix-appservice-irc:9999' +matrix_appservice_irc_database_engine: nedb +matrix_appservice_irc_database_username: matrix_appservice_irc +matrix_appservice_irc_database_password: ~ +matrix_appservice_irc_database_hostname: 'matrix-postgres' +matrix_appservice_irc_database_port: 5432 +matrix_appservice_irc_database_name: matrix_appservice_irc + +# This is just the Postgres connection string, if Postgres is used. +# Naming clashes with `matrix_appservice_irc_database_connectionString` somewhat. +matrix_appservice_irc_database_connection_string: 'postgresql://{{ matrix_appservice_irc_database_username }}:{{ matrix_appservice_irc_database_password }}@{{ matrix_appservice_irc_database_hostname }}:{{ matrix_appservice_irc_database_port }}/{{ matrix_appservice_irc_database_name }}?sslmode=disable' + +# This is what actually goes into `database.connectionString` for the bridge. +matrix_appservice_irc_database_connectionString: "{{ + { + 'nedb': 'nedb:///data', + 'postgres': matrix_appservice_irc_database_connection_string, + }[matrix_appservice_irc_database_engine] +}}" + matrix_appservice_irc_ircService_servers: [] # Example of `matrix_appservice_irc_ircService_servers` with one server (and all its options): diff --git a/roles/matrix-bridge-appservice-irc/tasks/migrate_nedb_to_postgres.yml b/roles/matrix-bridge-appservice-irc/tasks/migrate_nedb_to_postgres.yml new file mode 100644 index 000000000..3fab195ab --- /dev/null +++ b/roles/matrix-bridge-appservice-irc/tasks/migrate_nedb_to_postgres.yml @@ -0,0 +1,64 @@ +- name: Fail if Postgres not enabled + fail: + msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot migrate." + when: "not matrix_postgres_enabled|bool" + +# Defaults + +- name: Set postgres_start_wait_time, if not provided + set_fact: + postgres_start_wait_time: 15 + when: "postgres_start_wait_time|default('') == ''" + +# Actual import work + +- name: Ensure matrix-postgres is started + service: + name: matrix-postgres + state: started + daemon_reload: yes + register: matrix_postgres_service_start_result + +- name: Wait a bit, so that Postgres can start + wait_for: + timeout: "{{ postgres_start_wait_time }}" + delegate_to: 127.0.0.1 + become: false + when: "matrix_postgres_service_start_result.changed|bool" + +- name: Ensure matrix-appservice-irc is stopped + service: + name: matrix-appservice-irc + state: stopped + +- name: Import appservice-irc NeDB database into Postgres + command: + cmd: >- + {{ matrix_host_command_docker }} run + --rm + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} + --cap-drop=ALL + --network={{ matrix_docker_network }} + --mount type=bind,src={{ matrix_appservice_irc_data_path }},dst=/data + --entrypoint=/bin/sh + {{ matrix_appservice_irc_docker_image }} + -c + '/usr/local/bin/node /app/lib/scripts/migrate-db-to-pgres.js --dbdir /data --privateKey /data/passkey.pem --connectionString {{ matrix_appservice_irc_database_connection_string }}' + +- name: Archive NeDB database files + command: + cmd: "mv {{ matrix_appservice_irc_data_path }}/{{ item }} {{ matrix_appservice_irc_data_path }}/{{ item }}.backup" + with_items: + - rooms.db + - users.db + +- name: Inject result + set_fact: + matrix_playbook_runtime_results: | + {{ + matrix_playbook_runtime_results|default([]) + + + [ + "NOTE: Your appservice-irc database files have been imported into Postgres. The original database files have been moved from `{{ matrix_appservice_irc_data_path }}/*.db` to `{{ matrix_appservice_irc_data_path }}/*.db.backup`. When you've confirmed that the import went well and everything works, you should be able to safely delete these files." + ] + }} diff --git a/roles/matrix-bridge-appservice-irc/tasks/setup_install.yml b/roles/matrix-bridge-appservice-irc/tasks/setup_install.yml index 5e313347f..00568c0df 100644 --- a/roles/matrix-bridge-appservice-irc/tasks/setup_install.yml +++ b/roles/matrix-bridge-appservice-irc/tasks/setup_install.yml @@ -1,12 +1,5 @@ --- -- name: Ensure Appservice IRC image is pulled - docker_image: - name: "{{ matrix_appservice_irc_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_appservice_irc_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_appservice_irc_docker_image_force_pull }}" - - name: Ensure Appservice IRC paths exist file: path: "{{ item }}" @@ -24,25 +17,48 @@ path: "{{ matrix_appservice_irc_base_path }}/passkey.pem" register: matrix_appservice_irc_stat_passkey -- name: (Data relocation) Ensure matrix-appservice-irc.service is stopped - service: - name: matrix-appservice-irc - state: stopped - daemon_reload: yes - failed_when: false +- block: + - name: (Data relocation) Ensure matrix-appservice-irc.service is stopped + service: + name: matrix-appservice-irc + state: stopped + daemon_reload: yes + failed_when: false + + - name: (Data relocation) Move AppService IRC passkey.pem file to ./data directory + command: "mv {{ matrix_appservice_irc_base_path }}/passkey.pem {{ matrix_appservice_irc_data_path }}/passkey.pem" + + - name: (Data relocation) Move AppService IRC database files to ./data directory + command: "mv {{ matrix_appservice_irc_base_path }}/{{ item }} {{ matrix_appservice_irc_data_path }}/{{ item }}" + with_items: + - rooms.db + - users.db + failed_when: false when: "matrix_appservice_irc_stat_passkey.stat.exists" -- name: (Data relocation) Move AppService IRC passkey.pem file to ./data directory - command: "mv {{ matrix_appservice_irc_base_path }}/passkey.pem {{ matrix_appservice_irc_data_path }}/passkey.pem" - when: "matrix_appservice_irc_stat_passkey.stat.exists" +- set_fact: + matrix_appservice_irc_requires_restart: false -- name: (Data relocation) Move AppService IRC database files to ./data directory - command: "mv {{ matrix_appservice_irc_base_path }}/{{ item }} {{ matrix_appservice_irc_data_path }}/{{ item }}" - with_items: - - rooms.db - - users.db - failed_when: false - when: "matrix_appservice_irc_stat_passkey.stat.exists" +- block: + - name: Check if a nedb database already exists + stat: + path: "{{ matrix_appservice_irc_data_path }}/users.db" + register: matrix_appservice_irc_nedb_database_path_local_stat_result + + - block: + - import_tasks: "{{ role_path }}/tasks/migrate_nedb_to_postgres.yml" + + - set_fact: + matrix_appservice_irc_requires_restart: true + when: "matrix_appservice_irc_nedb_database_path_local_stat_result.stat.exists|bool" + when: "matrix_appservice_irc_database_engine == 'postgres'" + +- name: Ensure Appservice IRC image is pulled + docker_image: + name: "{{ matrix_appservice_irc_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_appservice_irc_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_appservice_irc_docker_image_force_pull }}" - name: Ensure Matrix Appservice IRC config installed copy: @@ -147,3 +163,9 @@ service: daemon_reload: yes when: "matrix_appservice_irc_systemd_service_result.changed" + +- name: Ensure matrix-appservice-irc.service restarted, if necessary + service: + name: "matrix-appservice-irc.service" + state: restarted + when: "matrix_appservice_irc_requires_restart|bool" diff --git a/roles/matrix-bridge-appservice-irc/templates/config.yaml.j2 b/roles/matrix-bridge-appservice-irc/templates/config.yaml.j2 index 3daa18987..94bbda7b8 100644 --- a/roles/matrix-bridge-appservice-irc/templates/config.yaml.j2 +++ b/roles/matrix-bridge-appservice-irc/templates/config.yaml.j2 @@ -127,8 +127,8 @@ advanced: # Use an external database to store bridge state. database: # database engine (must be 'postgres' or 'nedb'). Default: nedb - engine: "nedb" + engine: {{ matrix_appservice_irc_database_engine|to_json }} # Either a PostgreSQL connection string, or a path to the NeDB storage directory. # For postgres, it must start with postgres:// # For NeDB, it must start with nedb://. The path is relative to the project directory. - connectionString: "nedb:///data" + connectionString: {{ matrix_appservice_irc_database_connectionString|to_json }} diff --git a/roles/matrix-bridge-appservice-irc/templates/systemd/matrix-appservice-irc.service.j2 b/roles/matrix-bridge-appservice-irc/templates/systemd/matrix-appservice-irc.service.j2 index 95638612b..c86eb8352 100644 --- a/roles/matrix-bridge-appservice-irc/templates/systemd/matrix-appservice-irc.service.j2 +++ b/roles/matrix-bridge-appservice-irc/templates/systemd/matrix-appservice-irc.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_appservice_irc_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-appservice-slack/defaults/main.yml b/roles/matrix-bridge-appservice-slack/defaults/main.yml index f2cfd8f93..28646a3c4 100644 --- a/roles/matrix-bridge-appservice-slack/defaults/main.yml +++ b/roles/matrix-bridge-appservice-slack/defaults/main.yml @@ -45,6 +45,26 @@ matrix_appservice_slack_appservice_token: '' matrix_appservice_slack_homeserver_token: '' matrix_appservice_slack_id_token: '' +matrix_appservice_slack_database_engine: nedb +matrix_appservice_slack_database_username: matrix_appservice_slack +matrix_appservice_slack_database_password: ~ +matrix_appservice_slack_database_hostname: 'matrix-postgres' +matrix_appservice_slack_database_port: 5432 +matrix_appservice_slack_database_name: matrix_appservice_slack + +# This is just the Postgres connection string, if Postgres is used. +# Naming clashes with `matrix_appservice_slack_database_connectionString` somewhat. +matrix_appservice_slack_database_connection_string: 'postgresql://{{ matrix_appservice_slack_database_username }}:{{ matrix_appservice_slack_database_password }}@{{ matrix_appservice_slack_database_hostname }}:{{ matrix_appservice_slack_database_port }}/{{ matrix_appservice_slack_database_name }}?sslmode=disable' + +# This is what actually goes into `database.connectionString` for the bridge. +matrix_appservice_slack_database_connectionString: "{{ + { + 'nedb': 'nedb:///data', + 'postgres': matrix_appservice_slack_database_connection_string, + }[matrix_appservice_slack_database_engine] +}}" + + matrix_appservice_slack_configuration_yaml: "{{ lookup('template', 'templates/config.yaml.j2') }}" matrix_appservice_slack_configuration_extension_yaml: | diff --git a/roles/matrix-bridge-appservice-slack/tasks/migrate_nedb_to_postgres.yml b/roles/matrix-bridge-appservice-slack/tasks/migrate_nedb_to_postgres.yml new file mode 100644 index 000000000..fedad9775 --- /dev/null +++ b/roles/matrix-bridge-appservice-slack/tasks/migrate_nedb_to_postgres.yml @@ -0,0 +1,66 @@ +- name: Fail if Postgres not enabled + fail: + msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot migrate." + when: "not matrix_postgres_enabled|bool" + +# Defaults + +- name: Set postgres_start_wait_time, if not provided + set_fact: + postgres_start_wait_time: 15 + when: "postgres_start_wait_time|default('') == ''" + +# Actual import work + +- name: Ensure matrix-postgres is started + service: + name: matrix-postgres + state: started + daemon_reload: yes + register: matrix_postgres_service_start_result + +- name: Wait a bit, so that Postgres can start + wait_for: + timeout: "{{ postgres_start_wait_time }}" + delegate_to: 127.0.0.1 + become: false + when: "matrix_postgres_service_start_result.changed|bool" + +- name: Ensure matrix-appservice-slack is stopped + service: + name: matrix-appservice-slack + state: stopped + +- name: Import appservice-slack NeDB database into Postgres + command: + cmd: >- + {{ matrix_host_command_docker }} run + --rm + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} + --cap-drop=ALL + --network={{ matrix_docker_network }} + --mount type=bind,src={{ matrix_appservice_slack_data_path }},dst=/data + --entrypoint=/bin/sh + {{ matrix_appservice_slack_docker_image }} + -c + '/usr/local/bin/node /usr/src/app/lib/scripts/migrateToPostgres.js --dbdir /data --connectionString {{ matrix_appservice_slack_database_connection_string }}' + +- name: Archive NeDB database files + command: + cmd: "mv {{ matrix_appservice_slack_data_path }}/{{ item }} {{ matrix_appservice_slack_data_path }}/{{ item }}.backup" + with_items: + - teams.db + - room-store.db + - user-store.db + - event-store.db + +- name: Inject result + set_fact: + matrix_playbook_runtime_results: | + {{ + matrix_playbook_runtime_results|default([]) + + + [ + "NOTE: Your appservice-slack database files have been imported into Postgres. The original database files have been moved from `{{ matrix_appservice_slack_data_path }}/*.db` to `{{ matrix_appservice_slack_data_path }}/*.db.backup`. When you've confirmed that the import went well and everything works, you should be able to safely delete these files." + ] + }} diff --git a/roles/matrix-bridge-appservice-slack/tasks/setup_install.yml b/roles/matrix-bridge-appservice-slack/tasks/setup_install.yml index 94e0fedf9..29b0f39e9 100644 --- a/roles/matrix-bridge-appservice-slack/tasks/setup_install.yml +++ b/roles/matrix-bridge-appservice-slack/tasks/setup_install.yml @@ -1,12 +1,5 @@ --- -- name: Ensure Appservice Slack image is pulled - docker_image: - name: "{{ matrix_appservice_slack_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_appservice_slack_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_appservice_slack_docker_image_force_pull }}" - - name: Ensure AppService Slack paths exist file: path: "{{ item }}" @@ -19,6 +12,30 @@ - "{{ matrix_appservice_slack_config_path }}" - "{{ matrix_appservice_slack_data_path }}" +- set_fact: + matrix_appservice_slack_requires_restart: false + +- block: + - name: Check if a nedb database already exists + stat: + path: "{{ matrix_appservice_slack_data_path }}/teams.db" + register: matrix_appservice_slack_nedb_database_path_local_stat_result + + - block: + - import_tasks: "{{ role_path }}/tasks/migrate_nedb_to_postgres.yml" + + - set_fact: + matrix_appservice_slack_requires_restart: true + when: "matrix_appservice_slack_nedb_database_path_local_stat_result.stat.exists|bool" + when: "matrix_appservice_slack_database_engine == 'postgres'" + +- name: Ensure Appservice Slack image is pulled + docker_image: + name: "{{ matrix_appservice_slack_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_appservice_slack_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_appservice_slack_docker_image_force_pull }}" + - name: Ensure Matrix Appservice Slack config installed copy: content: "{{ matrix_appservice_slack_configuration|to_nice_yaml }}" @@ -46,3 +63,9 @@ service: daemon_reload: yes when: "matrix_appservice_slack_systemd_service_result.changed" + +- name: Ensure matrix-appservice-slack.service restarted, if necessary + service: + name: "matrix-appservice-slack.service" + state: restarted + when: "matrix_appservice_slack_requires_restart|bool" diff --git a/roles/matrix-bridge-appservice-slack/tasks/validate_config.yml b/roles/matrix-bridge-appservice-slack/tasks/validate_config.yml index 5da5f947e..8af10f2f5 100644 --- a/roles/matrix-bridge-appservice-slack/tasks/validate_config.yml +++ b/roles/matrix-bridge-appservice-slack/tasks/validate_config.yml @@ -9,4 +9,4 @@ - "matrix_appservice_slack_control_room_id" - "matrix_appservice_slack_appservice_token" - "matrix_appservice_slack_homeserver_token" - - "matrix_appservice_slack_id_token" \ No newline at end of file + - "matrix_appservice_slack_id_token" diff --git a/roles/matrix-bridge-appservice-slack/templates/config.yaml.j2 b/roles/matrix-bridge-appservice-slack/templates/config.yaml.j2 index 8f48d3178..bf8072c15 100644 --- a/roles/matrix-bridge-appservice-slack/templates/config.yaml.j2 +++ b/roles/matrix-bridge-appservice-slack/templates/config.yaml.j2 @@ -9,6 +9,12 @@ homeserver: url: "{{ matrix_appservice_slack_homeserver_url }}" server_name: "{{ matrix_domain }}" +{% if matrix_appservice_slack_database_engine == 'nedb' %} dbdir: "/data" +{% else %} +db: + engine: {{ matrix_appservice_slack_database_engine|to_json }} + connectionString: {{ matrix_appservice_slack_database_connectionString|to_json }} +{% endif %} matrix_admin_room: "{{ matrix_appservice_slack_control_room_id }}" diff --git a/roles/matrix-bridge-appservice-slack/templates/systemd/matrix-appservice-slack.service.j2 b/roles/matrix-bridge-appservice-slack/templates/systemd/matrix-appservice-slack.service.j2 index 1c68294fb..b16b2fed0 100644 --- a/roles/matrix-bridge-appservice-slack/templates/systemd/matrix-appservice-slack.service.j2 +++ b/roles/matrix-bridge-appservice-slack/templates/systemd/matrix-appservice-slack.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_appservice_slack_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-appservice-webhooks/templates/systemd/matrix-appservice-webhooks.service.j2 b/roles/matrix-bridge-appservice-webhooks/templates/systemd/matrix-appservice-webhooks.service.j2 index 08f5813f1..2049ee65d 100644 --- a/roles/matrix-bridge-appservice-webhooks/templates/systemd/matrix-appservice-webhooks.service.j2 +++ b/roles/matrix-bridge-appservice-webhooks/templates/systemd/matrix-appservice-webhooks.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_appservice_webhooks_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-mautrix-facebook/defaults/main.yml b/roles/matrix-bridge-mautrix-facebook/defaults/main.yml index c14a3acea..da9b1889f 100644 --- a/roles/matrix-bridge-mautrix-facebook/defaults/main.yml +++ b/roles/matrix-bridge-mautrix-facebook/defaults/main.yml @@ -32,9 +32,42 @@ matrix_mautrix_facebook_systemd_wanted_services_list: [] matrix_mautrix_facebook_appservice_token: '' matrix_mautrix_facebook_homeserver_token: '' + +# Database-related configuration fields. +# +# To use SQLite, stick to these defaults. +# +# To use Postgres: +# - change the engine (`matrix_mautrix_facebook_database_engine: 'postgres'`) +# - adjust your database credentials via the `matrix_mautrix_facebook_postgres_*` variables +matrix_mautrix_facebook_database_engine: 'sqlite' + +matrix_mautrix_facebook_sqlite_database_path_local: "{{ matrix_mautrix_facebook_data_path }}/mautrix-facebook.db" +matrix_mautrix_facebook_sqlite_database_path_in_container: "/data/mautrix-facebook.db" + +matrix_mautrix_facebook_database_username: 'matrix_mautrix_facebook' +matrix_mautrix_facebook_database_password: 'some-password' +matrix_mautrix_facebook_database_hostname: 'matrix-postgres' +matrix_mautrix_facebook_database_port: 5432 +matrix_mautrix_facebook_database_name: 'matrix_mautrix_facebook' + +matrix_mautrix_facebook_database_connection_string: 'postgres://{{ matrix_mautrix_facebook_database_username }}:{{ matrix_mautrix_facebook_database_password }}@{{ matrix_mautrix_facebook_database_hostname }}:{{ matrix_mautrix_facebook_database_port }}/{{ matrix_mautrix_facebook_database_name }}' + +matrix_mautrix_facebook_appservice_database: "{{ + { + 'sqlite': ('sqlite:///' + matrix_mautrix_facebook_sqlite_database_path_in_container), + 'postgres': matrix_mautrix_facebook_database_connection_string, + }[matrix_mautrix_facebook_database_engine] +}}" + + # Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). matrix_mautrix_facebook_login_shared_secret: '' +matrix_mautrix_facebook_bridge_login_shared_secret_map: "{{ {matrix_mautrix_facebook_homeserver_domain: matrix_mautrix_facebook_login_shared_secret} if matrix_mautrix_facebook_login_shared_secret else {} }}" + +matrix_mautrix_facebook_bridge_presence: true + # Default configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/matrix-bridge-mautrix-facebook/tasks/setup_install.yml b/roles/matrix-bridge-mautrix-facebook/tasks/setup_install.yml index 638854e2c..599984634 100644 --- a/roles/matrix-bridge-mautrix-facebook/tasks/setup_install.yml +++ b/roles/matrix-bridge-mautrix-facebook/tasks/setup_install.yml @@ -8,6 +8,32 @@ The matrix-bridge-mautrix-facebook role needs to execute before the matrix-synapse role. when: "matrix_synapse_role_executed|default(False)" +- set_fact: + matrix_mautrix_facebook_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_mautrix_facebook_sqlite_database_path_local }}" + register: matrix_mautrix_facebook_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_mautrix_facebook_sqlite_database_path_local }}" + dst: "{{ matrix_mautrix_facebook_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_mautrix_facebook_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-mautrix-facebook.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_mautrix_facebook_requires_restart: true + when: "matrix_mautrix_facebook_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_mautrix_facebook_database_engine == 'postgres'" + - name: Ensure Mautrix Facebook image is pulled docker_image: name: "{{ matrix_mautrix_facebook_docker_image }}" @@ -94,3 +120,9 @@ service: daemon_reload: yes when: "matrix_mautrix_facebook_systemd_service_result.changed" + +- name: Ensure matrix-mautrix-facebook.service restarted, if necessary + service: + name: "matrix-mautrix-facebook.service" + state: restarted + when: "matrix_mautrix_facebook_requires_restart|bool" diff --git a/roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 index 7da7b5880..09287362f 100644 --- a/roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2 @@ -27,7 +27,7 @@ appservice: # Format examples: # SQLite: sqlite:///filename.db # Postgres: postgres://username:password@hostname/dbname - database: sqlite:////data/mautrix-facebook.db + database: {{ matrix_mautrix_facebook_appservice_database|to_json }} # Public part of web server for out-of-Matrix interaction with the bridge. public: @@ -81,23 +81,32 @@ bridge: command_prefix: "!fb" # Number of chats to sync (and create portals for) on startup/login. - # Maximum 20, set 0 to disable automatic syncing. + # Set 0 to disable automatic syncing. initial_chat_sync: 10 # Whether or not the Facebook users of logged in Matrix users should be # invited to private chats when the user sends a message from another client. invite_own_puppet_to_pm: false - # Whether or not to use /sync to get presence, read receipts and typing notifications when using - # your own Matrix account as the Matrix puppet for your Facebook account. + # Whether or not to use /sync to get presence, read receipts and typing notifications + # when double puppeting is enabled sync_with_custom_puppets: true - # Shared secret for https://github.com/devture/matrix-synapse-shared-secret-auth + # Whether or not to update the m.direct account data event when double puppeting is enabled. + # Note that updating the m.direct event is not atomic (except with mautrix-asmux) + # and is therefore prone to race conditions. + sync_direct_chat_list: false + # Servers to always allow double puppeting from + double_puppet_server_map: {} + # example.com: https://example.com + # Allow using double puppeting from any server with a valid client .well-known file. + double_puppet_allow_discovery: false + # Shared secrets for https://github.com/devture/matrix-synapse-shared-secret-auth # # If set, custom puppets will be enabled automatically for local users # instead of users having to find an access token and run `login-matrix` # manually. - login_shared_secret: {{ matrix_mautrix_facebook_login_shared_secret|to_json }} - # Whether or not to bridge presence in both directions. Facebook allows users not to broadcast - # presence, but then it won't send other users' presence to the client. - presence: true + # If using this for other servers than the bridge's server, + # you must also set the URL in the double_puppet_server_map. + login_shared_secret_map: {{ matrix_mautrix_facebook_bridge_login_shared_secret_map|to_json }} + presence: {{ matrix_mautrix_facebook_bridge_presence|to_json }} # Whether or not to update avatars when syncing all contacts at startup. update_avatar_initial_sync: true # End-to-bridge encryption support options. These require matrix-nio to be installed with pip diff --git a/roles/matrix-bridge-mautrix-facebook/templates/systemd/matrix-mautrix-facebook.service.j2 b/roles/matrix-bridge-mautrix-facebook/templates/systemd/matrix-mautrix-facebook.service.j2 index b593a2e35..caa52eb57 100644 --- a/roles/matrix-bridge-mautrix-facebook/templates/systemd/matrix-mautrix-facebook.service.j2 +++ b/roles/matrix-bridge-mautrix-facebook/templates/systemd/matrix-mautrix-facebook.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_mautrix_facebook_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple @@ -17,6 +18,7 @@ ExecStartPre={{ matrix_host_command_docker }} run --rm --name matrix-mautrix-fac --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ + --network={{ matrix_docker_network }} \ -v {{ matrix_mautrix_facebook_config_path }}:/config:z \ -v {{ matrix_mautrix_facebook_data_path }}:/data:z \ {{ matrix_mautrix_facebook_docker_image }} \ diff --git a/roles/matrix-bridge-mautrix-hangouts/defaults/main.yml b/roles/matrix-bridge-mautrix-hangouts/defaults/main.yml index ab6700528..8dfee030b 100644 --- a/roles/matrix-bridge-mautrix-hangouts/defaults/main.yml +++ b/roles/matrix-bridge-mautrix-hangouts/defaults/main.yml @@ -39,6 +39,35 @@ matrix_mautrix_hangouts_systemd_wanted_services_list: [] matrix_mautrix_hangouts_appservice_token: '' matrix_mautrix_hangouts_homeserver_token: '' + +# Database-related configuration fields. +# +# To use SQLite, stick to these defaults. +# +# To use Postgres: +# - change the engine (`matrix_mautrix_hangouts_database_engine: 'postgres'`) +# - adjust your database credentials via the `matrix_mautrix_hangouts_postgres_*` variables +matrix_mautrix_hangouts_database_engine: 'sqlite' + +matrix_mautrix_hangouts_sqlite_database_path_local: "{{ matrix_mautrix_hangouts_data_path }}/mautrix-hangouts.db" +matrix_mautrix_hangouts_sqlite_database_path_in_container: "/data/mautrix-hangouts.db" + +matrix_mautrix_hangouts_database_username: 'matrix_mautrix_hangouts' +matrix_mautrix_hangouts_database_password: 'some-password' +matrix_mautrix_hangouts_database_hostname: 'matrix-postgres' +matrix_mautrix_hangouts_database_port: 5432 +matrix_mautrix_hangouts_database_name: 'matrix_mautrix_hangouts' + +matrix_mautrix_hangouts_database_connection_string: 'postgres://{{ matrix_mautrix_hangouts_database_username }}:{{ matrix_mautrix_hangouts_database_password }}@{{ matrix_mautrix_hangouts_database_hostname }}:{{ matrix_mautrix_hangouts_database_port }}/{{ matrix_mautrix_hangouts_database_name }}' + +matrix_mautrix_hangouts_appservice_database: "{{ + { + 'sqlite': ('sqlite:///' + matrix_mautrix_hangouts_sqlite_database_path_in_container), + 'postgres': matrix_mautrix_hangouts_database_connection_string, + }[matrix_mautrix_hangouts_database_engine] +}}" + + # Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). matrix_mautrix_hangouts_login_shared_secret: '' diff --git a/roles/matrix-bridge-mautrix-hangouts/tasks/setup_install.yml b/roles/matrix-bridge-mautrix-hangouts/tasks/setup_install.yml index eec5e0067..2ce8a4416 100644 --- a/roles/matrix-bridge-mautrix-hangouts/tasks/setup_install.yml +++ b/roles/matrix-bridge-mautrix-hangouts/tasks/setup_install.yml @@ -8,6 +8,32 @@ The matrix-bridge-mautrix-hangouts role needs to execute before the matrix-synapse role. when: "matrix_synapse_role_executed|default(False)" +- set_fact: + matrix_mautrix_hangouts_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_mautrix_hangouts_sqlite_database_path_local }}" + register: matrix_mautrix_hangouts_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_mautrix_hangouts_sqlite_database_path_local }}" + dst: "{{ matrix_mautrix_hangouts_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_mautrix_hangouts_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-mautrix-hangouts.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_mautrix_hangouts_requires_restart: true + when: "matrix_mautrix_hangouts_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_mautrix_hangouts_database_engine == 'postgres'" + - name: Ensure Mautrix Hangouts image is pulled docker_image: name: "{{ matrix_mautrix_hangouts_docker_image }}" @@ -93,3 +119,9 @@ service: daemon_reload: yes when: "matrix_mautrix_hangouts_systemd_service_result.changed" + +- name: Ensure matrix-mautrix-hangouts.service restarted, if necessary + service: + name: "matrix-mautrix-hangouts.service" + state: restarted + when: "matrix_mautrix_hangouts_requires_restart|bool" diff --git a/roles/matrix-bridge-mautrix-hangouts/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-hangouts/templates/config.yaml.j2 index f274b2034..cc2ca90bb 100644 --- a/roles/matrix-bridge-mautrix-hangouts/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-hangouts/templates/config.yaml.j2 @@ -27,7 +27,7 @@ appservice: # Format examples: # SQLite: sqlite:///filename.db # Postgres: postgres://username:password@hostname/dbname - database: sqlite:////data/mautrix-hangouts.db + database: {{ matrix_mautrix_hangouts_appservice_database|to_json }} # The unique ID of this appservice. id: hangouts diff --git a/roles/matrix-bridge-mautrix-hangouts/templates/systemd/matrix-mautrix-hangouts.service.j2 b/roles/matrix-bridge-mautrix-hangouts/templates/systemd/matrix-mautrix-hangouts.service.j2 index 584339820..395591905 100644 --- a/roles/matrix-bridge-mautrix-hangouts/templates/systemd/matrix-mautrix-hangouts.service.j2 +++ b/roles/matrix-bridge-mautrix-hangouts/templates/systemd/matrix-mautrix-hangouts.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_mautrix_hangouts_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple @@ -17,6 +18,7 @@ ExecStartPre={{ matrix_host_command_docker }} run --rm --name matrix-mautrix-han --log-driver=none \ --user={{ matrix_user_uid }}:{{ matrix_user_gid }} \ --cap-drop=ALL \ + --network={{ matrix_docker_network }} \ -v {{ matrix_mautrix_hangouts_config_path }}:/config:z \ -v {{ matrix_mautrix_hangouts_data_path }}:/data:z \ {{ matrix_mautrix_hangouts_docker_image }} \ diff --git a/roles/matrix-bridge-mautrix-telegram/defaults/main.yml b/roles/matrix-bridge-mautrix-telegram/defaults/main.yml index b6f4ef5bb..3f81617a9 100644 --- a/roles/matrix-bridge-mautrix-telegram/defaults/main.yml +++ b/roles/matrix-bridge-mautrix-telegram/defaults/main.yml @@ -43,6 +43,35 @@ matrix_mautrix_telegram_systemd_wanted_services_list: [] matrix_mautrix_telegram_appservice_token: '' matrix_mautrix_telegram_homeserver_token: '' + +# Database-related configuration fields. +# +# To use SQLite, stick to these defaults. +# +# To use Postgres: +# - change the engine (`matrix_mautrix_telegram_database_engine: 'postgres'`) +# - adjust your database credentials via the `matrix_mautrix_telegram_postgres_*` variables +matrix_mautrix_telegram_database_engine: 'sqlite' + +matrix_mautrix_telegram_sqlite_database_path_local: "{{ matrix_mautrix_telegram_data_path }}/mautrix-telegram.db" +matrix_mautrix_telegram_sqlite_database_path_in_container: "/data/mautrix-telegram.db" + +matrix_mautrix_telegram_database_username: 'matrix_mautrix_telegram' +matrix_mautrix_telegram_database_password: 'some-password' +matrix_mautrix_telegram_database_hostname: 'matrix-postgres' +matrix_mautrix_telegram_database_port: 5432 +matrix_mautrix_telegram_database_name: 'matrix_mautrix_telegram' + +matrix_mautrix_telegram_database_connection_string: 'postgres://{{ matrix_mautrix_telegram_database_username }}:{{ matrix_mautrix_telegram_database_password }}@{{ matrix_mautrix_telegram_database_hostname }}:{{ matrix_mautrix_telegram_database_port }}/{{ matrix_mautrix_telegram_database_name }}' + +matrix_mautrix_telegram_appservice_database: "{{ + { + 'sqlite': ('sqlite:///' + matrix_mautrix_telegram_sqlite_database_path_in_container), + 'postgres': matrix_mautrix_telegram_database_connection_string, + }[matrix_mautrix_telegram_database_engine] +}}" + + # Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). matrix_mautrix_telegram_login_shared_secret: '' diff --git a/roles/matrix-bridge-mautrix-telegram/tasks/setup_install.yml b/roles/matrix-bridge-mautrix-telegram/tasks/setup_install.yml index 9dc38ec2c..e9a93c72a 100644 --- a/roles/matrix-bridge-mautrix-telegram/tasks/setup_install.yml +++ b/roles/matrix-bridge-mautrix-telegram/tasks/setup_install.yml @@ -8,6 +8,32 @@ The matrix-bridge-mautrix-telegram role needs to execute before the matrix-synapse role. when: "matrix_synapse_role_executed|default(False)" +- set_fact: + matrix_mautrix_telegram_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_mautrix_telegram_sqlite_database_path_local }}" + register: matrix_mautrix_telegram_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_mautrix_telegram_sqlite_database_path_local }}" + dst: "{{ matrix_mautrix_telegram_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_mautrix_telegram_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-mautrix-telegram.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_mautrix_telegram_requires_restart: true + when: "matrix_mautrix_telegram_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_mautrix_telegram_database_engine == 'postgres'" + - name: Ensure Mautrix Telegram image is pulled docker_image: name: "{{ matrix_mautrix_telegram_docker_image }}" @@ -71,3 +97,9 @@ service: daemon_reload: yes when: "matrix_mautrix_telegram_systemd_service_result.changed" + +- name: Ensure matrix-mautrix-telegram.service restarted, if necessary + service: + name: "matrix-mautrix-telegram.service" + state: restarted + when: "matrix_mautrix_telegram_requires_restart|bool" diff --git a/roles/matrix-bridge-mautrix-telegram/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-telegram/templates/config.yaml.j2 index 1a4ac43e9..52efba02d 100644 --- a/roles/matrix-bridge-mautrix-telegram/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-telegram/templates/config.yaml.j2 @@ -27,7 +27,7 @@ appservice: # Format examples: # SQLite: sqlite:///filename.db # Postgres: postgres://username:password@hostname/dbname - database: sqlite:////data/mautrix-telegram.db + database: {{ matrix_mautrix_telegram_appservice_database|to_json }} # Public part of web server for out-of-Matrix interaction with the bridge. # Used for things like login if the user wants to make sure the 2FA password isn't stored in diff --git a/roles/matrix-bridge-mautrix-telegram/templates/systemd/matrix-mautrix-telegram.service.j2 b/roles/matrix-bridge-mautrix-telegram/templates/systemd/matrix-mautrix-telegram.service.j2 index c0fb89381..d61cdaa41 100644 --- a/roles/matrix-bridge-mautrix-telegram/templates/systemd/matrix-mautrix-telegram.service.j2 +++ b/roles/matrix-bridge-mautrix-telegram/templates/systemd/matrix-mautrix-telegram.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_mautrix_telegram_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-mautrix-whatsapp/defaults/main.yml b/roles/matrix-bridge-mautrix-whatsapp/defaults/main.yml index e9929f96e..beda6d7da 100644 --- a/roles/matrix-bridge-mautrix-whatsapp/defaults/main.yml +++ b/roles/matrix-bridge-mautrix-whatsapp/defaults/main.yml @@ -27,6 +27,42 @@ matrix_mautrix_whatsapp_systemd_wanted_services_list: [] matrix_mautrix_whatsapp_appservice_token: '' matrix_mautrix_whatsapp_homeserver_token: '' + +# Database-related configuration fields. +# +# To use SQLite, stick to these defaults. +# +# To use Postgres: +# - change the engine (`matrix_mautrix_whatsapp_database_engine: 'postgres'`) +# - adjust your database credentials via the `matrix_mautrix_whatsapp_postgres_*` variables +matrix_mautrix_whatsapp_database_engine: 'sqlite' + +matrix_mautrix_whatsapp_sqlite_database_path_local: "{{ matrix_mautrix_whatsapp_data_path }}/mautrix-whatsapp.db" +matrix_mautrix_whatsapp_sqlite_database_path_in_container: "/data/mautrix-whatsapp.db" + +matrix_mautrix_whatsapp_database_username: 'matrix_mautrix_whatsapp' +matrix_mautrix_whatsapp_database_password: 'some-password' +matrix_mautrix_whatsapp_database_hostname: 'matrix-postgres' +matrix_mautrix_whatsapp_database_port: 5432 +matrix_mautrix_whatsapp_database_name: 'matrix_mautrix_whatsapp' + +matrix_mautrix_whatsapp_database_connection_string: 'postgresql://{{ matrix_mautrix_whatsapp_database_username }}:{{ matrix_mautrix_whatsapp_database_password }}@{{ matrix_mautrix_whatsapp_database_hostname }}:{{ matrix_mautrix_whatsapp_database_port }}/{{ matrix_mautrix_whatsapp_database_name }}?sslmode=disable' + +matrix_mautrix_whatsapp_appservice_database_type: "{{ + { + 'sqlite': 'sqlite3', + 'postgres':'postgres', + }[matrix_mautrix_whatsapp_database_engine] +}}" + +matrix_mautrix_whatsapp_appservice_database_uri: "{{ + { + 'sqlite': matrix_mautrix_whatsapp_sqlite_database_path_in_container, + 'postgres': matrix_mautrix_whatsapp_database_connection_string, + }[matrix_mautrix_whatsapp_database_engine] +}}" + + # Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). matrix_mautrix_whatsapp_login_shared_secret: '' diff --git a/roles/matrix-bridge-mautrix-whatsapp/tasks/setup_install.yml b/roles/matrix-bridge-mautrix-whatsapp/tasks/setup_install.yml index 6bd87bbe2..1cfa60f8b 100644 --- a/roles/matrix-bridge-mautrix-whatsapp/tasks/setup_install.yml +++ b/roles/matrix-bridge-mautrix-whatsapp/tasks/setup_install.yml @@ -8,6 +8,32 @@ The matrix-bridge-mautrix-whatsapp role needs to execute before the matrix-synapse role. when: "matrix_synapse_role_executed|default(False)" +- set_fact: + matrix_mautrix_whatsapp_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_mautrix_whatsapp_sqlite_database_path_local }}" + register: matrix_mautrix_whatsapp_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_mautrix_whatsapp_sqlite_database_path_local }}" + dst: "{{ matrix_mautrix_whatsapp_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_mautrix_whatsapp_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-mautrix-whatsapp.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_mautrix_whatsapp_requires_restart: true + when: "matrix_mautrix_whatsapp_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_mautrix_whatsapp_database_engine == 'postgres'" + - name: Ensure Mautrix Whatsapp image is pulled docker_image: name: "{{ matrix_mautrix_whatsapp_docker_image }}" @@ -26,12 +52,12 @@ - "{{ matrix_mautrix_whatsapp_base_path }}" - "{{ matrix_mautrix_whatsapp_config_path }}" - "{{ matrix_mautrix_whatsapp_data_path }}" - + - name: Check if an old database file exists stat: path: "{{ matrix_mautrix_whatsapp_base_path }}/mautrix-whatsapp.db" register: matrix_mautrix_whatsapp_stat_database - + - name: Check if an old matrix state file exists stat: path: "{{ matrix_mautrix_whatsapp_base_path }}/mx-state.json" @@ -48,7 +74,7 @@ - name: (Data relocation) Move mautrix-whatsapp database file to ./data directory command: "mv {{ matrix_mautrix_whatsapp_base_path }}/mautrix-whatsapp.db {{ matrix_mautrix_whatsapp_data_path }}/mautrix-whatsapp.db" when: "matrix_mautrix_whatsapp_stat_database.stat.exists" - + - name: (Data relocation) Move mautrix-whatsapp mx-state file to ./data directory command: "mv {{ matrix_mautrix_whatsapp_base_path }}/mx-state.json {{ matrix_mautrix_whatsapp_data_path }}/mx-state.json" when: "matrix_mautrix_whatsapp_stat_mx_state.stat.exists" @@ -80,3 +106,9 @@ service: daemon_reload: yes when: "matrix_mautrix_whatsapp_systemd_service_result.changed" + +- name: Ensure matrix-mautrix-whatsapp.service restarted, if necessary + service: + name: "matrix-mautrix-whatsapp.service" + state: restarted + when: "matrix_mautrix_whatsapp_requires_restart|bool" diff --git a/roles/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 b/roles/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 index a527a1884..892166956 100644 --- a/roles/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mautrix-whatsapp/templates/config.yaml.j2 @@ -19,11 +19,11 @@ appservice: # Database config. database: # The database type. "sqlite3" and "postgres" are supported. - type: sqlite3 + type: {{ matrix_mautrix_whatsapp_appservice_database_type|to_json }} # The database URI. # SQLite: File name is enough. https://github.com/mattn/go-sqlite3#connection-string # Postgres: Connection string. For example, postgres://user:password@host/database - uri: mautrix-whatsapp.db + uri: {{ matrix_mautrix_whatsapp_appservice_database_uri|to_json }} # Maximum number of connections. Mostly relevant for Postgres. max_open_conns: 20 max_idle_conns: 2 diff --git a/roles/matrix-bridge-mautrix-whatsapp/templates/systemd/matrix-mautrix-whatsapp.service.j2 b/roles/matrix-bridge-mautrix-whatsapp/templates/systemd/matrix-mautrix-whatsapp.service.j2 index ac2b961e5..972d4e013 100644 --- a/roles/matrix-bridge-mautrix-whatsapp/templates/systemd/matrix-mautrix-whatsapp.service.j2 +++ b/roles/matrix-bridge-mautrix-whatsapp/templates/systemd/matrix-mautrix-whatsapp.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_mautrix_whatsapp_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-mx-puppet-discord/defaults/main.yml b/roles/matrix-bridge-mx-puppet-discord/defaults/main.yml index 268bbf8f8..97b203137 100644 --- a/roles/matrix-bridge-mx-puppet-discord/defaults/main.yml +++ b/roles/matrix-bridge-mx-puppet-discord/defaults/main.yml @@ -55,6 +55,20 @@ matrix_mx_puppet_discord_homeserver_token: '' # Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). matrix_mx_puppet_discord_login_shared_secret: '' +# Database configuration +matrix_mx_puppet_discord_database_engine: 'sqlite' + +matrix_mx_puppet_discord_sqlite_database_path_local: "{{ matrix_mx_puppet_discord_data_path }}/database.db" +matrix_mx_puppet_discord_sqlite_database_path_in_container: "/data/database.db" + +matrix_mx_puppet_discord_database_username: matrix_mx_puppet_discord +matrix_mx_puppet_discord_database_password: ~ +matrix_mx_puppet_discord_database_hostname: 'matrix-postgres' +matrix_mx_puppet_discord_database_port: 5432 +matrix_mx_puppet_discord_database_name: matrix_mx_puppet_discord + +matrix_mx_puppet_discord_database_connection_string: 'postgresql://{{ matrix_mx_puppet_discord_database_username }}:{{ matrix_mx_puppet_discord_database_password }}@{{ matrix_mx_puppet_discord_database_hostname }}:{{ matrix_mx_puppet_discord_database_port }}/{{ matrix_mx_puppet_discord_database_name }}?sslmode=disable' + # Default configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/matrix-bridge-mx-puppet-discord/tasks/setup_install.yml b/roles/matrix-bridge-mx-puppet-discord/tasks/setup_install.yml index c3f2b01fb..c7865e98a 100644 --- a/roles/matrix-bridge-mx-puppet-discord/tasks/setup_install.yml +++ b/roles/matrix-bridge-mx-puppet-discord/tasks/setup_install.yml @@ -8,14 +8,6 @@ The matrix-bridge-mx-puppet-discord role needs to execute before the matrix-synapse role. when: "matrix_synapse_role_executed|default(False)" -- name: Ensure MX Puppet Discord image is pulled - docker_image: - name: "{{ matrix_mx_puppet_discord_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_mx_puppet_discord_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mx_puppet_discord_docker_image_force_pull }}" - when: matrix_mx_puppet_discord_enabled|bool and not matrix_mx_puppet_discord_container_image_self_build - - name: Ensure MX Puppet Discord paths exist file: path: "{{ item.path }}" @@ -30,6 +22,57 @@ - { path: "{{ matrix_mx_puppet_discord_docker_src_files_path }}", when: "{{ matrix_mx_puppet_discord_container_image_self_build }}" } when: matrix_mx_puppet_discord_enabled|bool and item.when|bool +- name: Check if an old database file already exists + stat: + path: "{{ matrix_mx_puppet_discord_base_path }}/database.db" + register: matrix_mx_puppet_discord_stat_database + +- block: + - name: (Data relocation) Ensure matrix-mx-puppet-discord.service is stopped + service: + name: matrix-mx-puppet-discord + state: stopped + daemon_reload: yes + failed_when: False + + - name: (Data relocation) Move mx-puppet-discord database file to ./data directory + command: "mv {{ matrix_mx_puppet_discord_base_path }}/database.db {{ matrix_mx_puppet_discord_data_path }}/database.db" + when: "matrix_mx_puppet_discord_stat_database.stat.exists" + +- set_fact: + matrix_mx_puppet_discord_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_mx_puppet_discord_sqlite_database_path_local }}" + register: matrix_mx_puppet_discord_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_mx_puppet_discord_sqlite_database_path_local }}" + dst: "{{ matrix_mx_puppet_discord_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_mx_puppet_discord_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-mx-puppet-discord.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_mx_puppet_discord_requires_restart: true + when: "matrix_mx_puppet_discord_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_mx_puppet_discord_database_engine == 'postgres'" + +- name: Ensure MX Puppet Discord image is pulled + docker_image: + name: "{{ matrix_mx_puppet_discord_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_mx_puppet_discord_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mx_puppet_discord_docker_image_force_pull }}" + when: matrix_mx_puppet_discord_enabled|bool and not matrix_mx_puppet_discord_container_image_self_build + - name: Ensure MX Puppet Discord repository is present on self build git: repo: "{{ matrix_mx_puppet_discord_container_image_self_build_repo }}" @@ -49,23 +92,6 @@ pull: yes when: "matrix_mx_puppet_discord_enabled|bool and matrix_mx_puppet_discord_container_image_self_build|bool" -- name: Check if an old database file already exists - stat: - path: "{{ matrix_mx_puppet_discord_base_path }}/database.db" - register: matrix_mx_puppet_discord_stat_database - -- name: (Data relocation) Ensure matrix-mx-puppet-discord.service is stopped - service: - name: matrix-mx-puppet-discord - state: stopped - daemon_reload: yes - failed_when: false - when: "matrix_mx_puppet_discord_stat_database.stat.exists" - -- name: (Data relocation) Move mx-puppet-discord database file to ./data directory - command: "mv {{ matrix_mx_puppet_discord_base_path }}/database.db {{ matrix_mx_puppet_discord_data_path }}/database.db" - when: "matrix_mx_puppet_discord_stat_database.stat.exists" - - name: Ensure mx-puppet-discord config.yaml installed copy: content: "{{ matrix_mx_puppet_discord_configuration|to_nice_yaml }}" @@ -93,3 +119,9 @@ service: daemon_reload: yes when: "matrix_mx_puppet_discord_systemd_service_result.changed" + +- name: Ensure matrix-mx-puppet-discord.service restarted, if necessary + service: + name: "matrix-mx-puppet-discord.service" + state: restarted + when: "matrix_mx_puppet_discord_requires_restart|bool" diff --git a/roles/matrix-bridge-mx-puppet-discord/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-discord/templates/config.yaml.j2 index 2c7037961..1f4548d83 100644 --- a/roles/matrix-bridge-mx-puppet-discord/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-discord/templates/config.yaml.j2 @@ -100,15 +100,18 @@ namePatterns: group: :name database: +{% if matrix_mx_puppet_discord_database_engine == 'sqlite' %} + # Use SQLite3 as a database backend + # The name of the database file + filename: {{ matrix_mx_puppet_discord_sqlite_database_path_in_container|to_json }} +{% else %} # Use Postgres as a database backend # If set, will be used instead of SQLite3 # Connection string to connect to the Postgres instance # with username "user", password "pass", host "localhost" and database name "dbname". # Modify each value as necessary - #connString: "postgres://user:pass@localhost/dbname?sslmode=disable" - # Use SQLite3 as a database backend - # The name of the database file - filename: /data/database.db + connString: {{ matrix_mx_puppet_discord_database_connection_string|to_json }} +{% endif %} logging: # Log level of console output diff --git a/roles/matrix-bridge-mx-puppet-discord/templates/systemd/matrix-mx-puppet-discord.service.j2 b/roles/matrix-bridge-mx-puppet-discord/templates/systemd/matrix-mx-puppet-discord.service.j2 index 78737a17b..36a4da8bf 100644 --- a/roles/matrix-bridge-mx-puppet-discord/templates/systemd/matrix-mx-puppet-discord.service.j2 +++ b/roles/matrix-bridge-mx-puppet-discord/templates/systemd/matrix-mx-puppet-discord.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_mx_puppet_discord_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-mx-puppet-instagram/defaults/main.yml b/roles/matrix-bridge-mx-puppet-instagram/defaults/main.yml index 21cde53fa..cd08c010a 100644 --- a/roles/matrix-bridge-mx-puppet-instagram/defaults/main.yml +++ b/roles/matrix-bridge-mx-puppet-instagram/defaults/main.yml @@ -46,6 +46,19 @@ matrix_mx_puppet_instagram_homeserver_token: '' # Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). matrix_mx_puppet_instagram_login_shared_secret: '' +matrix_mx_puppet_instagram_database_engine: sqlite + +matrix_mx_puppet_instagram_sqlite_database_path_local: "{{ matrix_mx_puppet_instagram_data_path }}/database.db" +matrix_mx_puppet_instagram_sqlite_database_path_in_container: "/data/database.db" + +matrix_mx_puppet_instagram_database_username: matrix_mx_puppet_instagram +matrix_mx_puppet_instagram_database_password: ~ +matrix_mx_puppet_instagram_database_hostname: 'matrix-postgres' +matrix_mx_puppet_instagram_database_port: 5432 +matrix_mx_puppet_instagram_database_name: matrix_mx_puppet_instagram + +matrix_mx_puppet_instagram_database_connection_string: 'postgresql://{{ matrix_mx_puppet_instagram_database_username }}:{{ matrix_mx_puppet_instagram_database_password }}@{{ matrix_mx_puppet_instagram_database_hostname }}:{{ matrix_mx_puppet_instagram_database_port }}/{{ matrix_mx_puppet_instagram_database_name }}?sslmode=disable' + # Default configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/matrix-bridge-mx-puppet-instagram/tasks/setup_install.yml b/roles/matrix-bridge-mx-puppet-instagram/tasks/setup_install.yml index 78bd78c16..5701a9165 100644 --- a/roles/matrix-bridge-mx-puppet-instagram/tasks/setup_install.yml +++ b/roles/matrix-bridge-mx-puppet-instagram/tasks/setup_install.yml @@ -8,6 +8,33 @@ The matrix-bridge-mx-puppet-instagram role needs to execute before the matrix-synapse role. when: "matrix_synapse_role_executed|default(False)" + +- set_fact: + matrix_mx_puppet_instagram_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_mx_puppet_instagram_sqlite_database_path_local }}" + register: matrix_mx_puppet_instagram_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_mx_puppet_instagram_sqlite_database_path_local }}" + dst: "{{ matrix_mx_puppet_instagram_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_mx_puppet_instagram_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-mx-puppet-instagram.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_mx_puppet_instagram_requires_restart: true + when: "matrix_mx_puppet_instagram_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_mx_puppet_instagram_database_engine == 'postgres'" + - name: Ensure mx-puppet-instagram image is pulled docker_image: name: "{{ matrix_mx_puppet_instagram_docker_image }}" @@ -77,4 +104,8 @@ daemon_reload: yes when: "matrix_mx_puppet_instagram_systemd_service_result.changed" - +- name: Ensure matrix-mx-puppet-instagram.service restarted, if necessary + service: + name: "matrix-mx-puppet-instagram.service" + state: restarted + when: "matrix_mx_puppet_instagram_requires_restart|bool" diff --git a/roles/matrix-bridge-mx-puppet-instagram/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-instagram/templates/config.yaml.j2 index 634fbaecd..b830da2b6 100644 --- a/roles/matrix-bridge-mx-puppet-instagram/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-instagram/templates/config.yaml.j2 @@ -44,15 +44,18 @@ provisioning: apiPrefix: /_matrix/provision database: +{% if matrix_mx_puppet_instagram_database_engine == 'postgres' %} # Use Postgres as a database backend # If set, will be used instead of SQLite3 # Connection string to connect to the Postgres instance # with username "user", password "pass", host "localhost" and database name "dbname". # Modify each value as necessary - #connString: "postgres://user:pass@localhost/dbname?sslmode=disable" + connString: {{ matrix_mx_puppet_instagram_database_connection_string|to_json }} +{% else %} # Use SQLite3 as a database backend # The name of the database file - filename: /data/database.db + filename: {{ matrix_mx_puppet_instagram_sqlite_database_path_in_container|to_json }} +{% endif %} logging: # Log level of console output diff --git a/roles/matrix-bridge-mx-puppet-instagram/templates/systemd/matrix-mx-puppet-instagram.service.j2 b/roles/matrix-bridge-mx-puppet-instagram/templates/systemd/matrix-mx-puppet-instagram.service.j2 index 81e3e0817..4c94c84a3 100644 --- a/roles/matrix-bridge-mx-puppet-instagram/templates/systemd/matrix-mx-puppet-instagram.service.j2 +++ b/roles/matrix-bridge-mx-puppet-instagram/templates/systemd/matrix-mx-puppet-instagram.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_mx_puppet_instagram_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-mx-puppet-skype/defaults/main.yml b/roles/matrix-bridge-mx-puppet-skype/defaults/main.yml index 0c16a8bab..83cd3dc55 100644 --- a/roles/matrix-bridge-mx-puppet-skype/defaults/main.yml +++ b/roles/matrix-bridge-mx-puppet-skype/defaults/main.yml @@ -53,6 +53,20 @@ matrix_mx_puppet_skype_homeserver_token: '' # Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). matrix_mx_puppet_skype_login_shared_secret: '' +# Database configuration, role default is `sqlite` but playbook default is `postgres` +matrix_mx_puppet_skype_database_engine: sqlite + +matrix_mx_puppet_skype_sqlite_database_path_local: "{{ matrix_mx_puppet_skype_data_path }}/database.db" +matrix_mx_puppet_skype_sqlite_database_path_in_container: "/data/database.db" + +matrix_mx_puppet_skype_database_username: matrix_mx_puppet_skype +matrix_mx_puppet_skype_database_password: ~ +matrix_mx_puppet_skype_database_hostname: 'matrix-postgres' +matrix_mx_puppet_skype_database_port: 5432 +matrix_mx_puppet_skype_database_name: matrix_mx_puppet_skype + +matrix_mx_puppet_skype_database_connection_string: 'postgresql://{{ matrix_mx_puppet_skype_database_username }}:{{ matrix_mx_puppet_skype_database_password }}@{{ matrix_mx_puppet_skype_database_hostname }}:{{ matrix_mx_puppet_skype_database_port }}/{{ matrix_mx_puppet_skype_database_name }}?sslmode=disable' + # Default configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/matrix-bridge-mx-puppet-skype/tasks/setup_install.yml b/roles/matrix-bridge-mx-puppet-skype/tasks/setup_install.yml index fb5185eda..68a1d7f42 100644 --- a/roles/matrix-bridge-mx-puppet-skype/tasks/setup_install.yml +++ b/roles/matrix-bridge-mx-puppet-skype/tasks/setup_install.yml @@ -8,14 +8,6 @@ The matrix-bridge-mx-puppet-skype role needs to execute before the matrix-synapse role. when: "matrix_synapse_role_executed|default(False)" -- name: Ensure MX Puppet Skype image is pulled - docker_image: - name: "{{ matrix_mx_puppet_skype_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_mx_puppet_skype_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mx_puppet_skype_docker_image_force_pull }}" - when: matrix_mx_puppet_skype_enabled|bool and not matrix_mx_puppet_skype_container_image_self_build - - name: Ensure MX Puppet Skype paths exist file: path: "{{ item.path }}" @@ -30,6 +22,57 @@ - { path: "{{ matrix_mx_puppet_skype_docker_src_files_path }}", when: "{{ matrix_mx_puppet_skype_container_image_self_build }}" } when: matrix_mx_puppet_skype_enabled|bool and item.when|bool +- name: Check if an old database file already exists + stat: + path: "{{ matrix_mx_puppet_skype_base_path }}/database.db" + register: matrix_mx_puppet_skype_stat_database + +- name: (Data relocation) Ensure matrix-mx-puppet-skype.service is stopped + service: + name: matrix-mx-puppet-skype + state: stopped + daemon_reload: yes + failed_when: false + when: "matrix_mx_puppet_skype_stat_database.stat.exists" + +- name: (Data relocation) Move mx-puppet-skype database file to ./data directory + command: "mv {{ matrix_mx_puppet_skype_base_path }}/database.db {{ matrix_mx_puppet_skype_data_path }}/database.db" + when: "matrix_mx_puppet_skype_stat_database.stat.exists" + +- set_fact: + matrix_mx_puppet_skype_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_mx_puppet_skype_sqlite_database_path_local }}" + register: matrix_mx_puppet_skype_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_mx_puppet_skype_sqlite_database_path_local }}" + dst: "{{ matrix_mx_puppet_skype_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_mx_puppet_skype_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-mx-puppet-skype.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_mx_puppet_skype_requires_restart: true + when: "matrix_mx_puppet_skype_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_mx_puppet_skype_database_engine == 'postgres'" + +- name: Ensure MX Puppet Skype image is pulled + docker_image: + name: "{{ matrix_mx_puppet_skype_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_mx_puppet_skype_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mx_puppet_skype_docker_image_force_pull }}" + when: matrix_mx_puppet_skype_enabled|bool and not matrix_mx_puppet_skype_container_image_self_build + - name: Ensure MX Puppet Skype repository is present on self build git: repo: "{{ matrix_mx_puppet_skype_container_image_self_build_repo }}" @@ -49,23 +92,6 @@ pull: yes when: "matrix_mx_puppet_skype_enabled|bool and matrix_mx_puppet_skype_container_image_self_build|bool" -- name: Check if an old database file already exists - stat: - path: "{{ matrix_mx_puppet_skype_base_path }}/database.db" - register: matrix_mx_puppet_skype_stat_database - -- name: (Data relocation) Ensure matrix-mx-puppet-skype.service is stopped - service: - name: matrix-mx-puppet-skype - state: stopped - daemon_reload: yes - failed_when: false - when: "matrix_mx_puppet_skype_stat_database.stat.exists" - -- name: (Data relocation) Move mx-puppet-skype database file to ./data directory - command: "mv {{ matrix_mx_puppet_skype_base_path }}/database.db {{ matrix_mx_puppet_skype_data_path }}/database.db" - when: "matrix_mx_puppet_skype_stat_database.stat.exists" - - name: Ensure mx-puppet-skype config.yaml installed copy: content: "{{ matrix_mx_puppet_skype_configuration|to_nice_yaml }}" @@ -93,3 +119,9 @@ service: daemon_reload: yes when: "matrix_mx_puppet_skype_systemd_service_result.changed" + +- name: Ensure matrix-mx-puppet-skype.service restarted, if necessary + service: + name: "matrix-mx-puppet-skype.service" + state: restarted + when: "matrix_mx_puppet_skype_requires_restart|bool" diff --git a/roles/matrix-bridge-mx-puppet-skype/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-skype/templates/config.yaml.j2 index c7b5c870e..d41d3a235 100644 --- a/roles/matrix-bridge-mx-puppet-skype/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-skype/templates/config.yaml.j2 @@ -68,15 +68,18 @@ logging: # - Store database: +{% if matrix_mx_puppet_skype_database_engine == 'postgres' %} # Use Postgres as a database backend # If set, will be used instead of SQLite3 # Connection string to connect to the Postgres instance # with username "user", password "pass", host "localhost" and database name "dbname". # Modify each value as necessary - #connString: "postgres://user:pass@localhost/dbname?sslmode=disable" + connString: {{ matrix_mx_puppet_skype_database_connection_string|to_json }} +{% else %} # Use SQLite3 as a database backend # The name of the database file - filename: /data/database.db + filename: {{ matrix_mx_puppet_skype_sqlite_database_path_in_container|to_json }} +{% endif %} provisioning: # Regex of Matrix IDs allowed to use the puppet bridge diff --git a/roles/matrix-bridge-mx-puppet-skype/templates/systemd/matrix-mx-puppet-skype.service.j2 b/roles/matrix-bridge-mx-puppet-skype/templates/systemd/matrix-mx-puppet-skype.service.j2 index 8a46cb25b..cd958662d 100644 --- a/roles/matrix-bridge-mx-puppet-skype/templates/systemd/matrix-mx-puppet-skype.service.j2 +++ b/roles/matrix-bridge-mx-puppet-skype/templates/systemd/matrix-mx-puppet-skype.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_mx_puppet_skype_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-mx-puppet-slack/defaults/main.yml b/roles/matrix-bridge-mx-puppet-slack/defaults/main.yml index 96ab78331..70b98ecee 100644 --- a/roles/matrix-bridge-mx-puppet-slack/defaults/main.yml +++ b/roles/matrix-bridge-mx-puppet-slack/defaults/main.yml @@ -57,6 +57,20 @@ matrix_mx_puppet_slack_homeserver_token: '' # Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). matrix_mx_puppet_slack_login_shared_secret: '' +# Database configuration, role uses 'sqlite' per default but playbook sets up postgres by default +matrix_mx_puppet_slack_database_engine: sqlite + +matrix_mx_puppet_slack_sqlite_database_path_local: "{{ matrix_mx_puppet_slack_data_path }}/database.db" +matrix_mx_puppet_slack_sqlite_database_path_in_container: "/data/database.db" + +matrix_mx_puppet_slack_database_username: matrix_mx_puppet_slack +matrix_mx_puppet_slack_database_password: ~ +matrix_mx_puppet_slack_database_hostname: 'matrix-postgres' +matrix_mx_puppet_slack_database_port: 5432 +matrix_mx_puppet_slack_database_name: matrix_mx_puppet_slack + +matrix_mx_puppet_slack_database_connection_string: 'postgresql://{{ matrix_mx_puppet_slack_database_username }}:{{ matrix_mx_puppet_slack_database_password }}@{{ matrix_mx_puppet_slack_database_hostname }}:{{ matrix_mx_puppet_slack_database_port }}/{{ matrix_mx_puppet_slack_database_name }}?sslmode=disable' + # Default configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/matrix-bridge-mx-puppet-slack/tasks/setup_install.yml b/roles/matrix-bridge-mx-puppet-slack/tasks/setup_install.yml index 2e45ecf60..04eab20a9 100644 --- a/roles/matrix-bridge-mx-puppet-slack/tasks/setup_install.yml +++ b/roles/matrix-bridge-mx-puppet-slack/tasks/setup_install.yml @@ -8,14 +8,6 @@ The matrix-bridge-mx-puppet-slack role needs to execute before the matrix-synapse role. when: "matrix_synapse_role_executed|default(False)" -- name: Ensure MX Puppet Slack image is pulled - docker_image: - name: "{{ matrix_mx_puppet_slack_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_mx_puppet_slack_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mx_puppet_slack_docker_image_force_pull }}" - when: matrix_mx_puppet_slack_enabled|bool and not matrix_mx_puppet_slack_container_image_self_build - - name: Ensure MX Puppet Slack paths exist file: path: "{{ item.path }}" @@ -30,6 +22,53 @@ - { path: "{{ matrix_mx_puppet_slack_docker_src_files_path }}", when: "{{ matrix_mx_puppet_slack_container_image_self_build }}" } when: matrix_mx_puppet_slack_enabled|bool and item.when|bool +- name: Check if an old database file already exists + stat: + path: "{{ matrix_mx_puppet_slack_base_path }}/database.db" + register: matrix_mx_puppet_slack_stat_database + +- name: (Data relocation) Ensure matrix-mx-puppet-slack.service is stopped + service: + name: matrix-mx-puppet-slack + state: stopped + daemon_reload: yes + failed_when: false + when: "matrix_mx_puppet_slack_stat_database.stat.exists" + +- set_fact: + matrix_mx_puppet_slack_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_mx_puppet_slack_sqlite_database_path_local }}" + register: matrix_mx_puppet_slack_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_mx_puppet_slack_sqlite_database_path_local }}" + dst: "{{ matrix_mx_puppet_slack_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_mx_puppet_slack_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-mx-puppet-slack.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_mx_puppet_slack_requires_restart: true + when: "matrix_mx_puppet_slack_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_mx_puppet_slack_database_engine == 'postgres'" + +- name: Ensure MX Puppet Slack image is pulled + docker_image: + name: "{{ matrix_mx_puppet_slack_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_mx_puppet_slack_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mx_puppet_slack_docker_image_force_pull }}" + when: matrix_mx_puppet_slack_enabled|bool and not matrix_mx_puppet_slack_container_image_self_build + - name: Ensure MX Puppet Slack repository is present on self build git: repo: "{{ matrix_mx_puppet_slack_container_image_self_build_repo }}" @@ -49,19 +88,6 @@ pull: yes when: "matrix_mx_puppet_slack_enabled|bool and matrix_mx_puppet_slack_container_image_self_build" -- name: Check if an old database file already exists - stat: - path: "{{ matrix_mx_puppet_slack_base_path }}/database.db" - register: matrix_mx_puppet_slack_stat_database - -- name: (Data relocation) Ensure matrix-mx-puppet-slack.service is stopped - service: - name: matrix-mx-puppet-slack - state: stopped - daemon_reload: yes - failed_when: false - when: "matrix_mx_puppet_slack_stat_database.stat.exists" - - name: (Data relocation) Move mx-puppet-slack database file to ./data directory command: "mv {{ matrix_mx_puppet_slack_base_path }}/database.db {{ matrix_mx_puppet_slack_data_path }}/database.db" when: "matrix_mx_puppet_slack_stat_database.stat.exists" @@ -93,3 +119,9 @@ service: daemon_reload: yes when: "matrix_mx_puppet_slack_systemd_service_result.changed" + +- name: Ensure matrix-mx-puppet-slack.service restarted, if necessary + service: + name: "matrix-mx-puppet-slack.service" + state: restarted + when: "matrix_mx_puppet_slack_requires_restart|bool" diff --git a/roles/matrix-bridge-mx-puppet-slack/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-slack/templates/config.yaml.j2 index b6e887848..af6b5cb88 100644 --- a/roles/matrix-bridge-mx-puppet-slack/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-slack/templates/config.yaml.j2 @@ -58,15 +58,18 @@ provisioning: apiPrefix: /_matrix/provision database: +{% if matrix_mx_puppet_slack_database_engine == 'postgres' %} # Use Postgres as a database backend # If set, will be used instead of SQLite3 # Connection string to connect to the Postgres instance # with username "user", password "pass", host "localhost" and database name "dbname". # Modify each value as necessary - #connString: "postgres://user:pass@localhost/dbname?sslmode=disable" + connString: {{ matrix_mx_puppet_slack_database_connection_string|to_json }} +{% else %} # Use SQLite3 as a database backend # The name of the database file - filename: /data/database.db + filename: {{ matrix_mx_puppet_slack_sqlite_database_path_in_container|to_json }} +{% endif %} logging: # Log level of console output diff --git a/roles/matrix-bridge-mx-puppet-slack/templates/systemd/matrix-mx-puppet-slack.service.j2 b/roles/matrix-bridge-mx-puppet-slack/templates/systemd/matrix-mx-puppet-slack.service.j2 index 23c2504f5..2b1456f53 100644 --- a/roles/matrix-bridge-mx-puppet-slack/templates/systemd/matrix-mx-puppet-slack.service.j2 +++ b/roles/matrix-bridge-mx-puppet-slack/templates/systemd/matrix-mx-puppet-slack.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_mx_puppet_slack_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-mx-puppet-steam/defaults/main.yml b/roles/matrix-bridge-mx-puppet-steam/defaults/main.yml index b5eb84731..15fa889f0 100644 --- a/roles/matrix-bridge-mx-puppet-steam/defaults/main.yml +++ b/roles/matrix-bridge-mx-puppet-steam/defaults/main.yml @@ -55,6 +55,19 @@ matrix_mx_puppet_steam_homeserver_token: '' # Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). matrix_mx_puppet_steam_login_shared_secret: '' +matrix_mx_puppet_steam_database_engine: sqlite + +matrix_mx_puppet_steam_sqlite_database_path_local: "{{ matrix_mx_puppet_steam_data_path }}/database.db" +matrix_mx_puppet_steam_sqlite_database_path_in_container: "/data/database.db" + +matrix_mx_puppet_steam_database_username: matrix_mx_puppet_steam +matrix_mx_puppet_steam_database_password: ~ +matrix_mx_puppet_steam_database_hostname: 'matrix-postgres' +matrix_mx_puppet_steam_database_port: 5432 +matrix_mx_puppet_steam_database_name: matrix_mx_puppet_steam + +matrix_mx_puppet_steam_database_connection_string: 'postgresql://{{ matrix_mx_puppet_steam_database_username }}:{{ matrix_mx_puppet_steam_database_password }}@{{ matrix_mx_puppet_steam_database_hostname }}:{{ matrix_mx_puppet_steam_database_port }}/{{ matrix_mx_puppet_steam_database_name }}?sslmode=disable' + # Default configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/matrix-bridge-mx-puppet-steam/tasks/setup_install.yml b/roles/matrix-bridge-mx-puppet-steam/tasks/setup_install.yml index 7b7f82115..6b5746562 100644 --- a/roles/matrix-bridge-mx-puppet-steam/tasks/setup_install.yml +++ b/roles/matrix-bridge-mx-puppet-steam/tasks/setup_install.yml @@ -8,14 +8,6 @@ The matrix-bridge-mx-puppet-steam role needs to execute before the matrix-synapse role. when: "matrix_synapse_role_executed|default(False)" -- name: Ensure MX Puppet Steam image is pulled - docker_image: - name: "{{ matrix_mx_puppet_steam_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_mx_puppet_steam_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mx_puppet_steam_docker_image_force_pull }}" - when: matrix_mx_puppet_steam_enabled|bool and not matrix_mx_puppet_steam_container_image_self_build - - name: Ensure MX Puppet Steam paths exist file: path: "{{ item.path }}" @@ -30,6 +22,57 @@ - { path: "{{ matrix_mx_puppet_steam_docker_src_files_path }}", when: "{{ matrix_mx_puppet_steam_container_image_self_build }}" } when: matrix_mx_puppet_steam_enabled|bool and item.when|bool +- name: Check if an old database file already exists + stat: + path: "{{ matrix_mx_puppet_steam_base_path }}/database.db" + register: matrix_mx_puppet_steam_stat_database + +- name: (Data relocation) Ensure matrix-mx-puppet-steam.service is stopped + service: + name: matrix-mx-puppet-steam + state: stopped + daemon_reload: yes + failed_when: false + when: "matrix_mx_puppet_steam_stat_database.stat.exists" + +- name: (Data relocation) Move mx-puppet-steam database file to ./data directory + command: "mv {{ matrix_mx_puppet_steam_base_path }}/database.db {{ matrix_mx_puppet_steam_data_path }}/database.db" + when: "matrix_mx_puppet_steam_stat_database.stat.exists" + +- set_fact: + matrix_mx_puppet_steam_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_mx_puppet_steam_sqlite_database_path_local }}" + register: matrix_mx_puppet_steam_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_mx_puppet_steam_sqlite_database_path_local }}" + dst: "{{ matrix_mx_puppet_steam_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_mx_puppet_steam_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-mx-puppet-steam.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_mx_puppet_steam_requires_restart: true + when: "matrix_mx_puppet_steam_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_mx_puppet_steam_database_engine == 'postgres'" + +- name: Ensure MX Puppet Steam image is pulled + docker_image: + name: "{{ matrix_mx_puppet_steam_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_mx_puppet_steam_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mx_puppet_steam_docker_image_force_pull }}" + when: matrix_mx_puppet_steam_enabled|bool and not matrix_mx_puppet_steam_container_image_self_build + - name: Ensure MX Puppet Steam repository is present on self build git: repo: "{{ matrix_mx_puppet_steam_container_image_self_build_repo }}" @@ -49,23 +92,6 @@ pull: yes when: "matrix_mx_puppet_steam_enabled|bool and matrix_mx_puppet_steam_container_image_self_build" -- name: Check if an old database file already exists - stat: - path: "{{ matrix_mx_puppet_steam_base_path }}/database.db" - register: matrix_mx_puppet_steam_stat_database - -- name: (Data relocation) Ensure matrix-mx-puppet-steam.service is stopped - service: - name: matrix-mx-puppet-steam - state: stopped - daemon_reload: yes - failed_when: false - when: "matrix_mx_puppet_steam_stat_database.stat.exists" - -- name: (Data relocation) Move mx-puppet-steam database file to ./data directory - command: "mv {{ matrix_mx_puppet_steam_base_path }}/database.db {{ matrix_mx_puppet_steam_data_path }}/database.db" - when: "matrix_mx_puppet_steam_stat_database.stat.exists" - - name: Ensure mx-puppet-steam config.yaml installed copy: content: "{{ matrix_mx_puppet_steam_configuration|to_nice_yaml }}" @@ -93,3 +119,9 @@ service: daemon_reload: yes when: "matrix_mx_puppet_steam_systemd_service_result.changed" + +- name: Ensure matrix-mx-puppet-steam.service restarted, if necessary + service: + name: "matrix-mx-puppet-steam.service" + state: restarted + when: "matrix_mx_puppet_steam_requires_restart|bool" diff --git a/roles/matrix-bridge-mx-puppet-steam/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-steam/templates/config.yaml.j2 index d08982ca9..149e08b69 100644 --- a/roles/matrix-bridge-mx-puppet-steam/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-steam/templates/config.yaml.j2 @@ -61,15 +61,18 @@ selfService: blacklist: {{ matrix_mx_puppet_steam_provisioning_blacklist|to_json }} database: +{% if matrix_mx_puppet_steam_database_engine == 'postgres' %} # Use Postgres as a database backend # If set, will be used instead of SQLite3 # Connection string to connect to the Postgres instance # with username "user", password "pass", host "localhost" and database name "dbname". # Modify each value as necessary - #connString: "postgres://user:pass@localhost/dbname?sslmode=disable" + connString: {{ matrix_mx_puppet_steam_database_connection_string|to_json }} +{% else %} # Use SQLite3 as a database backend # The name of the database file - filename: /data/database.db + filename: {{ matrix_mx_puppet_steam_sqlite_database_path_in_container|to_json }} +{% endif %} logging: # Log level of console output diff --git a/roles/matrix-bridge-mx-puppet-steam/templates/systemd/matrix-mx-puppet-steam.service.j2 b/roles/matrix-bridge-mx-puppet-steam/templates/systemd/matrix-mx-puppet-steam.service.j2 index 986c5d2cd..e263154b6 100644 --- a/roles/matrix-bridge-mx-puppet-steam/templates/systemd/matrix-mx-puppet-steam.service.j2 +++ b/roles/matrix-bridge-mx-puppet-steam/templates/systemd/matrix-mx-puppet-steam.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_mx_puppet_steam_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-mx-puppet-twitter/defaults/main.yml b/roles/matrix-bridge-mx-puppet-twitter/defaults/main.yml index 13438a21f..28639fdad 100644 --- a/roles/matrix-bridge-mx-puppet-twitter/defaults/main.yml +++ b/roles/matrix-bridge-mx-puppet-twitter/defaults/main.yml @@ -60,6 +60,20 @@ matrix_mx_puppet_twitter_homeserver_token: '' # Can be set to enable automatic double-puppeting via Shared Secret Auth (https://github.com/devture/matrix-synapse-shared-secret-auth). matrix_mx_puppet_twitter_login_shared_secret: '' +# Database configuration +matrix_mx_puppet_twitter_database_engine: sqlite + +matrix_mx_puppet_twitter_sqlite_database_path_local: "{{ matrix_mx_puppet_twitter_data_path }}/database.db" +matrix_mx_puppet_twitter_sqlite_database_path_in_container: "/data/database.db" + +matrix_mx_puppet_twitter_database_username: mx_puppet_twitter +matrix_mx_puppet_twitter_database_password: ~ +matrix_mx_puppet_twitter_database_hostname: 'matrix-postgres' +matrix_mx_puppet_twitter_database_port: 5432 +matrix_mx_puppet_twitter_database_name: matrix_mx_puppet_twitter + +matrix_mx_puppet_twitter_database_connection_string: 'postgresql://{{ matrix_mx_puppet_twitter_database_username }}:{{ matrix_mx_puppet_twitter_database_password }}@{{ matrix_mx_puppet_twitter_database_hostname }}:{{ matrix_mx_puppet_twitter_database_port }}/{{ matrix_mx_puppet_twitter_database_name }}?sslmode=disable' + # Default configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/matrix-bridge-mx-puppet-twitter/tasks/setup_install.yml b/roles/matrix-bridge-mx-puppet-twitter/tasks/setup_install.yml index 663f822c4..1c48c0309 100644 --- a/roles/matrix-bridge-mx-puppet-twitter/tasks/setup_install.yml +++ b/roles/matrix-bridge-mx-puppet-twitter/tasks/setup_install.yml @@ -8,14 +8,6 @@ The matrix-bridge-mx-puppet-twitter role needs to execute before the matrix-synapse role. when: "matrix_synapse_role_executed|default(False)" -- name: Ensure MX Puppet Twitter image is pulled - docker_image: - name: "{{ matrix_mx_puppet_twitter_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_mx_puppet_twitter_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mx_puppet_twitter_docker_image_force_pull }}" - when: matrix_mx_puppet_twitter_enabled|bool and not matrix_mx_puppet_twitter_container_image_self_build - - name: Ensure MX Puppet Twitter paths exist file: path: "{{ item.path }}" @@ -30,6 +22,57 @@ - { path: "{{ matrix_mx_puppet_twitter_docker_src_files_path }}", when: "{{ matrix_mx_puppet_twitter_container_image_self_build }}" } when: matrix_mx_puppet_twitter_enabled|bool and item.when|bool +- name: Check if an old database file already exists + stat: + path: "{{ matrix_mx_puppet_twitter_base_path }}/database.db" + register: matrix_mx_puppet_twitter_stat_database + +- name: (Data relocation) Ensure matrix-mx-puppet-twitter.service is stopped + service: + name: matrix-mx-puppet-twitter + state: stopped + daemon_reload: yes + failed_when: false + when: "matrix_mx_puppet_twitter_stat_database.stat.exists" + +- name: (Data relocation) Move mx-puppet-twitter database file to ./data directory + command: "mv {{ matrix_mx_puppet_twitter_base_path }}/database.db {{ matrix_mx_puppet_twitter_data_path }}/database.db" + when: "matrix_mx_puppet_twitter_stat_database.stat.exists" + +- set_fact: + matrix_mx_puppet_twitter_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_mx_puppet_twitter_sqlite_database_path_local }}" + register: matrix_mx_puppet_twitter_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_mx_puppet_twitter_sqlite_database_path_local }}" + dst: "{{ matrix_mx_puppet_twitter_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_mx_puppet_twitter_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-mx-puppet-twitter.service'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_mx_puppet_twitter_requires_restart: true + when: "matrix_mx_puppet_twitter_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_mx_puppet_twitter_database_engine == 'postgres'" + +- name: Ensure MX Puppet Twitter image is pulled + docker_image: + name: "{{ matrix_mx_puppet_twitter_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_mx_puppet_twitter_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_mx_puppet_twitter_docker_image_force_pull }}" + when: matrix_mx_puppet_twitter_enabled|bool and not matrix_mx_puppet_twitter_container_image_self_build + - name: Ensure MX Puppet Twitter repository is present on self build git: repo: "{{ matrix_mx_puppet_twitter_container_image_self_build_repo }}" @@ -49,23 +92,6 @@ pull: yes when: "matrix_mx_puppet_twitter_enabled|bool and matrix_mx_puppet_twitter_container_image_self_build" -- name: Check if an old database file already exists - stat: - path: "{{ matrix_mx_puppet_twitter_base_path }}/database.db" - register: matrix_mx_puppet_twitter_stat_database - -- name: (Data relocation) Ensure matrix-mx-puppet-twitter.service is stopped - service: - name: matrix-mx-puppet-twitter - state: stopped - daemon_reload: yes - failed_when: false - when: "matrix_mx_puppet_twitter_stat_database.stat.exists" - -- name: (Data relocation) Move mx-puppet-twitter database file to ./data directory - command: "mv {{ matrix_mx_puppet_twitter_base_path }}/database.db {{ matrix_mx_puppet_twitter_data_path }}/database.db" - when: "matrix_mx_puppet_twitter_stat_database.stat.exists" - - name: Ensure mx-puppet-twitter config.yaml installed copy: content: "{{ matrix_mx_puppet_twitter_configuration|to_nice_yaml }}" @@ -93,3 +119,9 @@ service: daemon_reload: yes when: "matrix_mx_puppet_twitter_systemd_service_result.changed" + +- name: Ensure matrix-mx-puppet-twitter.service restarted, if necessary + service: + name: "matrix-mx-puppet-twitter.service" + state: restarted + when: "matrix_mx_puppet_twitter_requires_restart|bool" diff --git a/roles/matrix-bridge-mx-puppet-twitter/templates/config.yaml.j2 b/roles/matrix-bridge-mx-puppet-twitter/templates/config.yaml.j2 index 7d3033b39..bdecf1dce 100644 --- a/roles/matrix-bridge-mx-puppet-twitter/templates/config.yaml.j2 +++ b/roles/matrix-bridge-mx-puppet-twitter/templates/config.yaml.j2 @@ -54,15 +54,18 @@ provisioning: apiPrefix: /_matrix/provision database: +{% if matrix_mx_puppet_twitter_database_engine == 'postgres' %} # Use Postgres as a database backend # If set, will be used instead of SQLite3 # Connection string to connect to the Postgres instance # with username "user", password "pass", host "localhost" and database name "dbname". # Modify each value as necessary - #connString: "postgres://user:pass@localhost/dbname?sslmode=disable" + connString: {{ matrix_mx_puppet_twitter_database_connection_string|to_json }} +{% else %} # Use SQLite3 as a database backend # The name of the database file - filename: /data/database.db + filename: {{ matrix_mx_puppet_twitter_sqlite_database_path_in_container|to_json }} +{% endif %} logging: # Log level of console output diff --git a/roles/matrix-bridge-mx-puppet-twitter/templates/systemd/matrix-mx-puppet-twitter.service.j2 b/roles/matrix-bridge-mx-puppet-twitter/templates/systemd/matrix-mx-puppet-twitter.service.j2 index 6afb6fb48..8d7898ebd 100644 --- a/roles/matrix-bridge-mx-puppet-twitter/templates/systemd/matrix-mx-puppet-twitter.service.j2 +++ b/roles/matrix-bridge-mx-puppet-twitter/templates/systemd/matrix-mx-puppet-twitter.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_mx_puppet_twitter_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-bridge-sms/defaults/main.yml b/roles/matrix-bridge-sms/defaults/main.yml index 82f9b8412..3c6b4c61c 100644 --- a/roles/matrix-bridge-sms/defaults/main.yml +++ b/roles/matrix-bridge-sms/defaults/main.yml @@ -3,7 +3,7 @@ matrix_sms_bridge_enabled: true -matrix_sms_bridge_docker_image: "docker.io/folivonet/matrix-sms-bridge:0.5.4" +matrix_sms_bridge_docker_image: "docker.io/folivonet/matrix-sms-bridge:0.5.5" matrix_sms_bridge_base_path: "{{ matrix_base_data_path }}/matrix-sms-bridge" matrix_sms_bridge_config_path: "{{ matrix_base_data_path }}/matrix-sms-bridge/config" diff --git a/roles/matrix-bridge-sms/templates/systemd/matrix-sms-bridge.service.j2 b/roles/matrix-bridge-sms/templates/systemd/matrix-sms-bridge.service.j2 index 7c96f9f42..0eb0eb587 100644 --- a/roles/matrix-bridge-sms/templates/systemd/matrix-sms-bridge.service.j2 +++ b/roles/matrix-bridge-sms/templates/systemd/matrix-sms-bridge.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_sms_bridge_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-client-element/defaults/main.yml b/roles/matrix-client-element/defaults/main.yml index cbe70503c..d0297193a 100644 --- a/roles/matrix-client-element/defaults/main.yml +++ b/roles/matrix-client-element/defaults/main.yml @@ -3,7 +3,7 @@ matrix_client_element_enabled: true matrix_client_element_container_image_self_build: false matrix_client_element_container_image_self_build_repo: "https://github.com/vector-im/riot-web.git" -matrix_client_element_docker_image: "{{ matrix_client_element_docker_image_name_prefix }}vectorim/element-web:v1.7.14" +matrix_client_element_docker_image: "{{ matrix_client_element_docker_image_name_prefix }}vectorim/element-web:v1.7.16" matrix_client_element_docker_image_name_prefix: "{{ 'localhost/' if matrix_client_element_container_image_self_build else 'docker.io/' }}" matrix_client_element_docker_image_force_pull: "{{ matrix_client_element_docker_image.endswith(':latest') }}" diff --git a/roles/matrix-client-element/templates/systemd/matrix-client-element.service.j2 b/roles/matrix-client-element/templates/systemd/matrix-client-element.service.j2 index 39b7aa52e..e0dd2e7e9 100644 --- a/roles/matrix-client-element/templates/systemd/matrix-client-element.service.j2 +++ b/roles/matrix-client-element/templates/systemd/matrix-client-element.service.j2 @@ -5,6 +5,7 @@ Description=Matrix Element server Requires={{ service }} After={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-common-after/tasks/start.yml b/roles/matrix-common-after/tasks/start.yml index 4bb59bfe5..bad84d92c 100644 --- a/roles/matrix-common-after/tasks/start.yml +++ b/roles/matrix-common-after/tasks/start.yml @@ -1,5 +1,9 @@ --- +- name: Deterimne whether we should make services autostart + set_fact: + matrix_services_autostart_enabled_bool: "{{ true if matrix_services_autostart_enabled|default('') == '' else matrix_services_autostart_enabled|bool }}" + - name: Ensure systemd is reloaded service: daemon_reload: yes @@ -14,7 +18,7 @@ - name: Ensure Matrix services are started service: name: "{{ item }}" - enabled: yes + enabled: "{{ matrix_services_autostart_enabled_bool }}" state: started with_items: "{{ matrix_systemd_services_list }}" when: not ansible_check_mode diff --git a/roles/matrix-corporal/defaults/main.yml b/roles/matrix-corporal/defaults/main.yml index def3fcc94..cccaadd0b 100644 --- a/roles/matrix-corporal/defaults/main.yml +++ b/roles/matrix-corporal/defaults/main.yml @@ -3,6 +3,9 @@ matrix_corporal_enabled: true +matrix_corporal_container_image_self_build: false +matrix_corporal_container_image_self_build_repo: "https://github.com/devture/matrix-corporal.git" + # Controls whether the matrix-corporal container exposes its gateway HTTP port (tcp/41080 in the container). # # Takes an ":" or "" value (e.g. "127.0.0.1:41080"), or empty string to not expose. @@ -19,10 +22,13 @@ matrix_corporal_container_extra_arguments: [] # List of systemd services that matrix-corporal.service depends on matrix_corporal_systemd_required_services_list: ['docker.service'] -matrix_corporal_docker_image: "docker.io/devture/matrix-corporal:1.11.0" +matrix_corporal_docker_image: "{{ matrix_corporal_docker_image_name_prefix }}devture/matrix-corporal:{{ matrix_corporal_docker_image_tag }}" +matrix_corporal_docker_image_name_prefix: "{{ 'localhost/' if matrix_corporal_container_image_self_build else 'docker.io/' }}" +matrix_corporal_docker_image_tag: "1.11.0" matrix_corporal_docker_image_force_pull: "{{ matrix_corporal_docker_image.endswith(':latest') }}" matrix_corporal_base_path: "{{ matrix_base_data_path }}/corporal" +matrix_corporal_container_src_files_path: "{{ matrix_corporal_base_path }}/container-src" matrix_corporal_config_dir_path: "{{ matrix_corporal_base_path }}/config" matrix_corporal_cache_dir_path: "{{ matrix_corporal_base_path }}/cache" matrix_corporal_var_dir_path: "{{ matrix_corporal_base_path }}/var" diff --git a/roles/matrix-corporal/tasks/setup_corporal.yml b/roles/matrix-corporal/tasks/setup_corporal.yml index 72b6a5b26..188f09bfe 100644 --- a/roles/matrix-corporal/tasks/setup_corporal.yml +++ b/roles/matrix-corporal/tasks/setup_corporal.yml @@ -17,13 +17,33 @@ - "{{ matrix_corporal_var_dir_path }}" when: matrix_corporal_enabled|bool +- name: Ensure Matrix Corporal repository is present on self-build + git: + repo: "{{ matrix_corporal_container_image_self_build_repo }}" + dest: "{{ matrix_corporal_container_src_files_path }}" + version: "{{ matrix_corporal_docker_image.split(':')[1] }}" + force: "yes" + register: matrix_corporal_git_pull_results + when: "matrix_corporal_enabled|bool and matrix_corporal_container_image_self_build|bool" + +- name: Ensure Matrix Corporal Docker image is built + docker_image: + name: "{{ matrix_corporal_docker_image }}" + source: build + force_source: "{{ matrix_corporal_git_pull_results.changed }}" + build: + dockerfile: etc/docker/Dockerfile + path: "{{ matrix_corporal_container_src_files_path }}" + pull: yes + when: "matrix_corporal_enabled|bool and matrix_corporal_container_image_self_build|bool" + - name: Ensure Matrix Corporal Docker image is pulled docker_image: name: "{{ matrix_corporal_docker_image }}" source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" force_source: "{{ matrix_corporal_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_corporal_docker_image_force_pull }}" - when: matrix_corporal_enabled|bool + when: "matrix_corporal_enabled|bool and not matrix_corporal_container_image_self_build|bool" - name: Ensure Matrix Corporal config installed copy: diff --git a/roles/matrix-corporal/templates/systemd/matrix-corporal.service.j2 b/roles/matrix-corporal/templates/systemd/matrix-corporal.service.j2 index 97ec7d89d..e8ce8c0a3 100644 --- a/roles/matrix-corporal/templates/systemd/matrix-corporal.service.j2 +++ b/roles/matrix-corporal/templates/systemd/matrix-corporal.service.j2 @@ -5,6 +5,7 @@ Description=Matrix Corporal Requires={{ service }} After={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-coturn/templates/systemd/matrix-coturn.service.j2 b/roles/matrix-coturn/templates/systemd/matrix-coturn.service.j2 index e484b59b9..16ca5d2a6 100644 --- a/roles/matrix-coturn/templates/systemd/matrix-coturn.service.j2 +++ b/roles/matrix-coturn/templates/systemd/matrix-coturn.service.j2 @@ -5,6 +5,7 @@ Description=Matrix Coturn server Requires={{ service }} After={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-dimension/defaults/main.yml b/roles/matrix-dimension/defaults/main.yml index 3b69227ab..8a691946f 100644 --- a/roles/matrix-dimension/defaults/main.yml +++ b/roles/matrix-dimension/defaults/main.yml @@ -15,6 +15,12 @@ matrix_dimension_base_path: "{{ matrix_base_data_path }}/dimension" matrix_dimension_docker_image: "docker.io/turt2live/matrix-dimension:latest" matrix_dimension_docker_image_force_pull: "{{ matrix_dimension_docker_image.endswith(':latest') }}" +# List of systemd services that matrix-dimension.service depends on. +matrix_dimension_systemd_required_services_list: ['docker.service'] + +# List of systemd services that matrix-dimension.service wants +matrix_dimension_systemd_wanted_services_list: [] + # The user and group id correspond to the node user in the `turt2live/matrix-dimension` image. matrix_dimension_user_uid: '1000' matrix_dimension_user_gid: '1000' @@ -34,6 +40,28 @@ matrix_dimension_integrations_jitsi_widget_url: "https://{{ matrix_server_fqn_di matrix_dimension_homeserver_federationUrl: "http://matrix-synapse:8048" + +# Database-related configuration fields. +# +# To use SQLite, stick to these defaults. +# +# To use Postgres: +# - change the engine (`matrix_dimension_database_engine: 'postgres'`) +# - adjust your database credentials via the `matrix_dimension_postgres_*` variables +matrix_dimension_database_engine: 'sqlite' + +matrix_dimension_sqlite_database_path_local: "{{ matrix_dimension_base_path }}/dimension.db" +matrix_dimension_sqlite_database_path_in_container: "dimension.db" + +matrix_dimension_database_username: 'matrix_dimension' +matrix_dimension_database_password: 'some-password' +matrix_dimension_database_hostname: 'matrix-postgres' +matrix_dimension_database_port: 5432 +matrix_dimension_database_name: 'matrix_dimension' + +matrix_dimension_database_connection_string: 'postgres://{{ matrix_dimension_database_username }}:{{ matrix_dimension_database_password }}@{{ matrix_dimension_database_hostname }}:{{ matrix_dimension_database_port }}/{{ matrix_dimension_database_name }}' + + # Default Dimension configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/matrix-dimension/tasks/main.yml b/roles/matrix-dimension/tasks/main.yml index 1888f9452..aad552866 100644 --- a/roles/matrix-dimension/tasks/main.yml +++ b/roles/matrix-dimension/tasks/main.yml @@ -8,8 +8,14 @@ - setup-all - setup-dimension -- import_tasks: "{{ role_path }}/tasks/setup_dimension.yml" - when: run_setup|bool +- import_tasks: "{{ role_path }}/tasks/setup_install.yml" + when: run_setup|bool and matrix_dimension_enabled|bool + tags: + - setup-all + - setup-dimension + +- import_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + when: run_setup|bool and not matrix_dimension_enabled|bool tags: - setup-all - setup-dimension diff --git a/roles/matrix-dimension/tasks/setup_dimension.yml b/roles/matrix-dimension/tasks/setup_dimension.yml deleted file mode 100644 index 2437a5472..000000000 --- a/roles/matrix-dimension/tasks/setup_dimension.yml +++ /dev/null @@ -1,85 +0,0 @@ ---- - -# -# Tasks related to setting up the dimension -# - -- name: Ensure Dimension base path exists - file: - path: "{{ matrix_dimension_base_path }}" - state: directory - mode: 0770 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_dimension_user_gid }}" - when: matrix_dimension_enabled|bool - -- name: Ensure Dimension config installed - copy: - content: "{{ matrix_dimension_configuration|to_nice_yaml }}" - dest: "{{ matrix_dimension_base_path }}/config.yaml" - mode: 0640 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_dimension_user_gid }}" - when: matrix_dimension_enabled|bool - -- name: Ensure Dimension image is pulled - docker_image: - name: "{{ matrix_dimension_docker_image }}" - source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" - force_source: "{{ matrix_dimension_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" - force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_dimension_docker_image_force_pull }}" - when: matrix_dimension_enabled|bool - -- name: Ensure matrix-dimension.service installed - template: - src: "{{ role_path }}/templates/systemd/matrix-dimension.service.j2" - dest: "{{ matrix_systemd_path }}/matrix-dimension.service" - mode: 0644 - register: matrix_dimension_systemd_service_result - when: matrix_dimension_enabled|bool - -- name: Ensure systemd reloaded after matrix-dimension.service installation - service: - daemon_reload: yes - when: "matrix_dimension_enabled|bool and matrix_dimension_systemd_service_result.changed" - -# -# Tasks related to getting rid of the dimension (if it was previously enabled) -# - -- name: Check existence of matrix-dimension service - stat: - path: "{{ matrix_systemd_path }}/matrix-dimension.service" - register: matrix_dimension_service_stat - when: "not matrix_dimension_enabled|bool" - -- name: Ensure matrix-dimension is stopped - service: - name: matrix-dimension - state: stopped - daemon_reload: yes - register: stopping_result - when: "not matrix_dimension_enabled|bool and matrix_dimension_service_stat.stat.exists" - -- name: Ensure matrix-dimension.service doesn't exist - file: - path: "{{ matrix_systemd_path }}/matrix-dimension.service" - state: absent - when: "not matrix_dimension_enabled|bool and matrix_dimension_service_stat.stat.exists" - -- name: Ensure systemd reloaded after matrix-dimension.service removal - service: - daemon_reload: yes - when: "not matrix_dimension_enabled|bool and matrix_dimension_service_stat.stat.exists" - -- name: Ensure Dimension environment variables path doesn't exist - file: - path: "{{ matrix_dimension_base_path }}" - state: absent - when: "not matrix_dimension_enabled|bool" - -- name: Ensure Dimension Docker image doesn't exist - docker_image: - name: "{{ matrix_dimension_docker_image }}" - state: absent - when: "not matrix_dimension_enabled|bool" diff --git a/roles/matrix-dimension/tasks/setup_install.yml b/roles/matrix-dimension/tasks/setup_install.yml new file mode 100644 index 000000000..26a75bcbd --- /dev/null +++ b/roles/matrix-dimension/tasks/setup_install.yml @@ -0,0 +1,109 @@ +--- + +- set_fact: + matrix_dimension_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_dimension_sqlite_database_path_local }}" + register: matrix_dimension_sqlite_database_path_local_stat_result + + - block: + # pgloader makes a few columns `smallint`, instead of `boolean`. + # We need to fix them up. + - set_fact: + matrix_dimension_pgloader_additional_psql_statements_list: [] + + - set_fact: + matrix_dimension_pgloader_additional_psql_statements_list: | + {{ + matrix_dimension_pgloader_additional_psql_statements_list + + + ([] if item.default == '' else ['ALTER TABLE ' + item.table + ' ALTER COLUMN "' + item.column + '" DROP default;']) + + + (['ALTER TABLE ' + item.table + ' ALTER COLUMN "' + item.column + '" TYPE BOOLEAN USING("' + item.column + '"::text::boolean);']) + + + ([] if item.default == '' else ['ALTER TABLE ' + item.table + ' ALTER COLUMN "' + item.column + '" SET default ' + item.default + ';']) + }} + with_items: + - {'table': 'dimension_widgets', 'column': 'isEnabled', 'default': ''} + - {'table': 'dimension_widgets', 'column': 'isPublic', 'default': ''} + - {'table': 'dimension_webhook_bridges', 'column': 'isEnabled', 'default': ''} + - {'table': 'dimension_user_sticker_packs', 'column': 'isSelected', 'default': ''} + - {'table': 'dimension_scalar_tokens', 'column': 'isDimensionToken', 'default': ''} + - {'table': 'dimension_users', 'column': 'isSelfBot', 'default': 'false'} + - {'table': 'dimension_telegram_bridges', 'column': 'allowTgPuppets', 'default': ''} + - {'table': 'dimension_telegram_bridges', 'column': 'allowMxPuppets', 'default': ''} + - {'table': 'dimension_telegram_bridges', 'column': 'isEnabled', 'default': ''} + - {'table': 'dimension_sticker_packs', 'column': 'isEnabled', 'default': ''} + - {'table': 'dimension_sticker_packs', 'column': 'isPublic', 'default': ''} + - {'table': 'dimension_slack_bridges', 'column': 'isEnabled', 'default': ''} + - {'table': 'dimension_neb_integrations', 'column': 'isPublic', 'default': ''} + - {'table': 'dimension_irc_bridges', 'column': 'isEnabled', 'default': ''} + - {'table': 'dimension_irc_bridge_networks', 'column': 'isEnabled', 'default': ''} + - {'table': 'dimension_gitter_bridges', 'column': 'isEnabled', 'default': ''} + - {'table': 'dimension_custom_simple_bots', 'column': 'isEnabled', 'default': ''} + - {'table': 'dimension_custom_simple_bots', 'column': 'isPublic', 'default': ''} + - {'table': 'dimension_bridges', 'column': 'isEnabled', 'default': ''} + - {'table': 'dimension_bridges', 'column': 'isPublic', 'default': ''} + + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_dimension_sqlite_database_path_local }}" + dst: "{{ matrix_dimension_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_dimension_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-dimension.service'] + pgloader_options: ['--with "quote identifiers"'] + additional_psql_statements_list: "{{ matrix_dimension_pgloader_additional_psql_statements_list }}" + additional_psql_statements_db_name: "{{ matrix_dimension_database_name }}" + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_dimension_requires_restart: true + when: "matrix_dimension_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_dimension_database_engine == 'postgres'" + +- name: Ensure Dimension base path exists + file: + path: "{{ matrix_dimension_base_path }}" + state: directory + mode: 0770 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_dimension_user_gid }}" + +- name: Ensure Dimension config installed + copy: + content: "{{ matrix_dimension_configuration|to_nice_yaml }}" + dest: "{{ matrix_dimension_base_path }}/config.yaml" + mode: 0640 + owner: "{{ matrix_user_username }}" + group: "{{ matrix_dimension_user_gid }}" + +- name: Ensure Dimension image is pulled + docker_image: + name: "{{ matrix_dimension_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_dimension_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_dimension_docker_image_force_pull }}" + +- name: Ensure matrix-dimension.service installed + template: + src: "{{ role_path }}/templates/systemd/matrix-dimension.service.j2" + dest: "{{ matrix_systemd_path }}/matrix-dimension.service" + mode: 0644 + register: matrix_dimension_systemd_service_result + +- name: Ensure systemd reloaded after matrix-dimension.service installation + service: + daemon_reload: yes + when: "matrix_dimension_systemd_service_result.changed|bool" + +- name: Ensure matrix-dimension.service restarted, if necessary + service: + name: "matrix-dimension.service" + state: restarted + when: "matrix_dimension_requires_restart|bool" diff --git a/roles/matrix-dimension/tasks/setup_uninstall.yml b/roles/matrix-dimension/tasks/setup_uninstall.yml new file mode 100644 index 000000000..9bc4ac8b9 --- /dev/null +++ b/roles/matrix-dimension/tasks/setup_uninstall.yml @@ -0,0 +1,35 @@ +--- + +- name: Check existence of matrix-dimension service + stat: + path: "{{ matrix_systemd_path }}/matrix-dimension.service" + register: matrix_dimension_service_stat + +- name: Ensure matrix-dimension is stopped + service: + name: matrix-dimension + state: stopped + daemon_reload: yes + register: stopping_result + when: "matrix_dimension_service_stat.stat.exists|bool" + +- name: Ensure matrix-dimension.service doesn't exist + file: + path: "{{ matrix_systemd_path }}/matrix-dimension.service" + state: absent + when: "matrix_dimension_service_stat.stat.exists|bool" + +- name: Ensure systemd reloaded after matrix-dimension.service removal + service: + daemon_reload: yes + when: "matrix_dimension_service_stat.stat.exists|bool" + +- name: Ensure Dimension base directory doesn't exist + file: + path: "{{ matrix_dimension_base_path }}" + state: absent + +- name: Ensure Dimension Docker image doesn't exist + docker_image: + name: "{{ matrix_dimension_docker_image }}" + state: absent diff --git a/roles/matrix-dimension/templates/config.yaml.j2 b/roles/matrix-dimension/templates/config.yaml.j2 index a05b6c35a..200871e7e 100644 --- a/roles/matrix-dimension/templates/config.yaml.j2 +++ b/roles/matrix-dimension/templates/config.yaml.j2 @@ -44,7 +44,11 @@ widgetBlacklist: # Where the database for Dimension is database: - file: "dimension.db" +{% if matrix_dimension_database_engine == 'sqlite' %} + file: {{ matrix_dimension_sqlite_database_path_in_container|to_json }} +{% elif matrix_dimension_database_engine == 'postgres' %} + uri: {{ matrix_dimension_database_connection_string|to_json }} +{% endif %} # Display settings that apply to self-hosted go-neb instances goneb: diff --git a/roles/matrix-dimension/templates/systemd/matrix-dimension.service.j2 b/roles/matrix-dimension/templates/systemd/matrix-dimension.service.j2 index bad94f642..ff10224af 100644 --- a/roles/matrix-dimension/templates/systemd/matrix-dimension.service.j2 +++ b/roles/matrix-dimension/templates/systemd/matrix-dimension.service.j2 @@ -1,8 +1,14 @@ #jinja2: lstrip_blocks: "True" [Unit] Description=Matrix Dimension -After=docker.service -Requires=docker.service +{% for service in matrix_dimension_systemd_required_services_list %} +Requires={{ service }} +After={{ service }} +{% endfor %} +{% for service in matrix_dimension_systemd_wanted_services_list %} +Wants={{ service }} +{% endfor %} +DefaultDependencies=no [Service] Type=simple @@ -10,7 +16,9 @@ ExecStartPre=-{{ matrix_host_command_docker }} kill matrix-dimension ExecStartPre=-{{ matrix_host_command_docker }} rm matrix-dimension # Fixup database ownership if it got changed somehow (during a server migration, etc.) -ExecStartPre=-{{ matrix_host_command_chown }} {{ matrix_dimension_user_uid }}:{{ matrix_dimension_user_gid }} {{ matrix_dimension_base_path }}/dimension.db +{% if matrix_dimension_database_engine == 'sqlite' %} +ExecStartPre=-{{ matrix_host_command_chown }} {{ matrix_dimension_user_uid }}:{{ matrix_dimension_user_gid }} {{ matrix_dimension_sqlite_database_path_local }} +{% endif %} ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-dimension \ --log-driver=none \ diff --git a/roles/matrix-dynamic-dns/templates/systemd/matrix-dynamic-dns.service.j2 b/roles/matrix-dynamic-dns/templates/systemd/matrix-dynamic-dns.service.j2 index 7e73b5879..df7d810ad 100644 --- a/roles/matrix-dynamic-dns/templates/systemd/matrix-dynamic-dns.service.j2 +++ b/roles/matrix-dynamic-dns/templates/systemd/matrix-dynamic-dns.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_dynamic_dns_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-email2matrix/templates/systemd/matrix-email2matrix.service.j2 b/roles/matrix-email2matrix/templates/systemd/matrix-email2matrix.service.j2 index 52806f351..1577877b1 100644 --- a/roles/matrix-email2matrix/templates/systemd/matrix-email2matrix.service.j2 +++ b/roles/matrix-email2matrix/templates/systemd/matrix-email2matrix.service.j2 @@ -3,6 +3,7 @@ Description=Email2Matrix After=docker.service Requires=docker.service +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-jitsi/templates/jicofo/matrix-jitsi-jicofo.service.j2 b/roles/matrix-jitsi/templates/jicofo/matrix-jitsi-jicofo.service.j2 index f19d08ff2..6b5cc9419 100644 --- a/roles/matrix-jitsi/templates/jicofo/matrix-jitsi-jicofo.service.j2 +++ b/roles/matrix-jitsi/templates/jicofo/matrix-jitsi-jicofo.service.j2 @@ -5,6 +5,7 @@ Description=Matrix jitsi-jicofo server Requires={{ service }} After={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-jitsi/templates/jvb/matrix-jitsi-jvb.service.j2 b/roles/matrix-jitsi/templates/jvb/matrix-jitsi-jvb.service.j2 index 7fcfeec65..2931133fb 100644 --- a/roles/matrix-jitsi/templates/jvb/matrix-jitsi-jvb.service.j2 +++ b/roles/matrix-jitsi/templates/jvb/matrix-jitsi-jvb.service.j2 @@ -5,6 +5,7 @@ Description=Matrix jitsi-jvb server Requires={{ service }} After={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-jitsi/templates/prosody/matrix-jitsi-prosody.service.j2 b/roles/matrix-jitsi/templates/prosody/matrix-jitsi-prosody.service.j2 index 74b13df30..4f532d894 100644 --- a/roles/matrix-jitsi/templates/prosody/matrix-jitsi-prosody.service.j2 +++ b/roles/matrix-jitsi/templates/prosody/matrix-jitsi-prosody.service.j2 @@ -5,6 +5,7 @@ Description=Matrix jitsi-prosody server Requires={{ service }} After={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-jitsi/templates/web/interface_config.js.j2 b/roles/matrix-jitsi/templates/web/interface_config.js.j2 index 60fac4d55..a12ca9730 100644 --- a/roles/matrix-jitsi/templates/web/interface_config.js.j2 +++ b/roles/matrix-jitsi/templates/web/interface_config.js.j2 @@ -205,9 +205,11 @@ var interfaceConfig = { {% if matrix_jitsi_enable_transcriptions %} 'closedcaptions', {% endif %} - + {% if matrix_jitsi_enable_recording %} + 'recording', + {% endif %} 'microphone', 'camera', 'desktop', 'embedmeeting', 'fullscreen', - 'fodeviceselection', 'hangup', 'profile', 'chat', 'recording', + 'fodeviceselection', 'hangup', 'profile', 'chat', 'livestreaming', 'etherpad', 'sharedvideo', 'settings', 'raisehand', 'videoquality', 'filmstrip', 'invite', 'feedback', 'stats', 'shortcuts', 'tileview', 'videobackgroundblur', 'download', 'help', 'mute-everyone', 'security' diff --git a/roles/matrix-jitsi/templates/web/matrix-jitsi-web.service.j2 b/roles/matrix-jitsi/templates/web/matrix-jitsi-web.service.j2 index e13f96331..1978fb0ec 100644 --- a/roles/matrix-jitsi/templates/web/matrix-jitsi-web.service.j2 +++ b/roles/matrix-jitsi/templates/web/matrix-jitsi-web.service.j2 @@ -5,6 +5,7 @@ Description=Matrix jitsi-web server Requires={{ service }} After={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-ma1sd/defaults/main.yml b/roles/matrix-ma1sd/defaults/main.yml index f7a0782d5..2932f3ed4 100644 --- a/roles/matrix-ma1sd/defaults/main.yml +++ b/roles/matrix-ma1sd/defaults/main.yml @@ -39,6 +39,28 @@ matrix_ma1sd_systemd_wanted_services_list: [] # Enabling this is discouraged. Learn more here: https://github.com/ma1uta/ma1sd/blob/master/docs/features/identity.md#lookups matrix_ma1sd_matrixorg_forwarding_enabled: false + +# Database-related configuration fields. +# +# To use SQLite, stick to these defaults. +# +# To use Postgres: +# - change the engine (`matrix_ma1sd_database_engine: 'postgres'`) +# - adjust your database credentials via the `matrix_ma1sd_postgres_*` variables +matrix_ma1sd_database_engine: 'sqlite' + +matrix_ma1sd_sqlite_database_path_local: "{{ matrix_ma1sd_data_path }}/ma1sd.db" +matrix_ma1sd_sqlite_database_path_in_container: "/var/ma1sd/ma1sd.db" + +matrix_ma1sd_database_username: 'matrix_ma1sd' +matrix_ma1sd_database_password: 'some-password' +matrix_ma1sd_database_hostname: 'matrix-postgres' +matrix_ma1sd_database_port: 5432 +matrix_ma1sd_database_name: 'matrix_ma1sd' + +matrix_ma1sd_database_connection_string: 'postgresql://{{ matrix_ma1sd_database_username }}:{{ matrix_ma1sd_database_password }}@{{ matrix_ma1sd_database_hostname }}:{{ matrix_ma1sd_database_port }}/{{ matrix_ma1sd_database_name }}' + + # ma1sd has serveral supported identity stores. # One of them is storing identities directly in Synapse's database. # Learn more here: https://github.com/ma1uta/ma1sd/blob/master/docs/stores/synapse.md diff --git a/roles/matrix-ma1sd/tasks/main.yml b/roles/matrix-ma1sd/tasks/main.yml index f5ac34d65..0b8a114e1 100644 --- a/roles/matrix-ma1sd/tasks/main.yml +++ b/roles/matrix-ma1sd/tasks/main.yml @@ -8,7 +8,14 @@ - setup-all - setup-ma1sd -- import_tasks: "{{ role_path }}/tasks/setup_ma1sd.yml" +- import_tasks: "{{ role_path }}/tasks/setup_install.yml" + when: "run_setup|bool and matrix_ma1sd_enabled|bool" + tags: + - setup-all + - setup-ma1sd + +- import_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + when: "run_setup|bool and not matrix_ma1sd_enabled|bool" tags: - setup-all - setup-ma1sd diff --git a/roles/matrix-ma1sd/tasks/setup_ma1sd.yml b/roles/matrix-ma1sd/tasks/setup_install.yml similarity index 69% rename from roles/matrix-ma1sd/tasks/setup_ma1sd.yml rename to roles/matrix-ma1sd/tasks/setup_install.yml index 46acb4286..a0a327289 100644 --- a/roles/matrix-ma1sd/tasks/setup_ma1sd.yml +++ b/roles/matrix-ma1sd/tasks/setup_install.yml @@ -1,9 +1,5 @@ --- -# -# Tasks related to setting up ma1sd -# - - name: Ensure ma1sd paths exist file: path: "{{ item.path }}" @@ -15,10 +11,39 @@ - { path: "{{ matrix_ma1sd_config_path }}", when: true } - { path: "{{ matrix_ma1sd_data_path }}", when: true } - { path: "{{ matrix_ma1sd_docker_src_files_path }}", when: "{{ matrix_ma1sd_container_image_self_build }}"} - when: matrix_ma1sd_enabled|bool and item.when + when: "item.when|bool" - import_tasks: "{{ role_path }}/tasks/migrate_mxisd.yml" - when: matrix_ma1sd_enabled|bool + + +# These (SQLite -> Postgres) migration tasks are usually at the top, +# but we'd like to run them after `migrate_mxisd.yml`, which requires the ma1sd paths to exist. +- set_fact: + matrix_ma1sd_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_ma1sd_sqlite_database_path_local }}" + register: matrix_ma1sd_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_ma1sd_sqlite_database_path_local }}" + dst: "{{ matrix_ma1sd_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_ma1sd_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-ma1sd.service'] + pgloader_options: ['--with "quote identifiers"'] + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_ma1sd_requires_restart: true + when: "matrix_ma1sd_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_ma1sd_database_engine == 'postgres'" - name: Ensure ma1sd image is pulled docker_image: @@ -26,7 +51,7 @@ source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" force_source: "{{ matrix_ma1sd_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_ma1sd_docker_image_force_pull }}" - when: matrix_ma1sd_enabled|bool and not matrix_ma1sd_container_image_self_build + when: "not matrix_ma1sd_container_image_self_build|bool" - block: - name: Ensure gradle is installed for self-building (Debian) @@ -72,7 +97,7 @@ repository: "{{ matrix_ma1sd_docker_image }}" force_tag: yes source: local - when: "matrix_ma1sd_enabled|bool and matrix_ma1sd_container_image_self_build" + when: "matrix_ma1sd_container_image_self_build|bool" - name: Ensure ma1sd config installed copy: @@ -81,7 +106,6 @@ mode: 0644 owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" - when: matrix_ma1sd_enabled|bool - name: Ensure custom templates are installed if any copy: @@ -95,7 +119,7 @@ - {value: "{{ matrix_ma1sd_threepid_medium_email_custom_session_validation_template }}", location: 'validate-template.eml'} - {value: "{{ matrix_ma1sd_threepid_medium_email_custom_unbind_fraudulent_template }}", location: 'unbind-fraudulent.eml'} - {value: "{{ matrix_ma1sd_threepid_medium_email_custom_matrixid_template }}", location: 'mxid-template.eml'} - when: "matrix_ma1sd_enabled|bool and matrix_ma1sd_threepid_medium_email_custom_templates_enabled|bool and item.value" + when: "matrix_ma1sd_threepid_medium_email_custom_templates_enabled|bool and item.value" - name: Ensure matrix-ma1sd.service installed template: @@ -103,49 +127,14 @@ dest: "{{ matrix_systemd_path }}/matrix-ma1sd.service" mode: 0644 register: matrix_ma1sd_systemd_service_result - when: matrix_ma1sd_enabled|bool - name: Ensure systemd reloaded after matrix-ma1sd.service installation service: daemon_reload: yes - when: "matrix_ma1sd_enabled|bool and matrix_ma1sd_systemd_service_result.changed" + when: "matrix_ma1sd_systemd_service_result.changed|bool" -# -# Tasks related to getting rid of ma1sd (if it was previously enabled) -# - -- name: Check existence of matrix-ma1sd service - stat: - path: "{{ matrix_systemd_path }}/matrix-ma1sd.service" - register: matrix_ma1sd_service_stat - -- name: Ensure matrix-ma1sd is stopped +- name: Ensure matrix-ma1sd.service restarted, if necessary service: - name: matrix-ma1sd - state: stopped - daemon_reload: yes - register: stopping_result - when: "not matrix_ma1sd_enabled|bool and matrix_ma1sd_service_stat.stat.exists" - -- name: Ensure matrix-ma1sd.service doesn't exist - file: - path: "{{ matrix_systemd_path }}/matrix-ma1sd.service" - state: absent - when: "not matrix_ma1sd_enabled|bool and matrix_ma1sd_service_stat.stat.exists" - -- name: Ensure systemd reloaded after matrix-ma1sd.service removal - service: - daemon_reload: yes - when: "not matrix_ma1sd_enabled|bool and matrix_ma1sd_service_stat.stat.exists" - -- name: Ensure Matrix ma1sd paths don't exist - file: - path: "{{ matrix_ma1sd_base_path }}" - state: absent - when: "not matrix_ma1sd_enabled|bool" - -- name: Ensure ma1sd Docker image doesn't exist - docker_image: - name: "{{ matrix_ma1sd_docker_image }}" - state: absent - when: "not matrix_ma1sd_enabled|bool" + name: "matrix-ma1sd.service" + state: restarted + when: "matrix_ma1sd_requires_restart|bool" diff --git a/roles/matrix-ma1sd/tasks/setup_uninstall.yml b/roles/matrix-ma1sd/tasks/setup_uninstall.yml new file mode 100644 index 000000000..b36ab508f --- /dev/null +++ b/roles/matrix-ma1sd/tasks/setup_uninstall.yml @@ -0,0 +1,35 @@ +--- + +- name: Check existence of matrix-ma1sd service + stat: + path: "{{ matrix_systemd_path }}/matrix-ma1sd.service" + register: matrix_ma1sd_service_stat + +- name: Ensure matrix-ma1sd is stopped + service: + name: matrix-ma1sd + state: stopped + daemon_reload: yes + register: stopping_result + when: "matrix_ma1sd_service_stat.stat.exists|bool" + +- name: Ensure matrix-ma1sd.service doesn't exist + file: + path: "{{ matrix_systemd_path }}/matrix-ma1sd.service" + state: absent + when: "matrix_ma1sd_service_stat.stat.exists|bool" + +- name: Ensure systemd reloaded after matrix-ma1sd.service removal + service: + daemon_reload: yes + when: "matrix_ma1sd_service_stat.stat.exists|bool" + +- name: Ensure Matrix ma1sd paths don't exist + file: + path: "{{ matrix_ma1sd_base_path }}" + state: absent + +- name: Ensure ma1sd Docker image doesn't exist + docker_image: + name: "{{ matrix_ma1sd_docker_image }}" + state: absent diff --git a/roles/matrix-ma1sd/templates/ma1sd.yaml.j2 b/roles/matrix-ma1sd/templates/ma1sd.yaml.j2 index 8f3569b18..9a426c477 100644 --- a/roles/matrix-ma1sd/templates/ma1sd.yaml.j2 +++ b/roles/matrix-ma1sd/templates/ma1sd.yaml.j2 @@ -11,9 +11,19 @@ key: path: /var/ma1sd/sign.key storage: - provider: - sqlite: - database: /var/ma1sd/ma1sd.db + {% if matrix_ma1sd_database_engine == 'sqlite' %} + backend: sqlite + provider: + sqlite: + database: {{ matrix_ma1sd_sqlite_database_path_in_container|to_json }} + {% elif matrix_ma1sd_database_engine == 'postgres' %} + backend: postgresql + provider: + postgresql: + database: //{{ matrix_ma1sd_database_hostname }}:{{ matrix_ma1sd_database_port }}/{{ matrix_ma1sd_database_name }} + username: {{ matrix_ma1sd_database_username|to_json }} + password: {{ matrix_ma1sd_database_password|to_json }} + {% endif %} {% if matrix_ma1sd_dns_overwrite_enabled %} dns: @@ -73,10 +83,10 @@ hashing: - none # the same as v1 bulk lookup - sha256 # hash the 3PID and pepper. delay: 2m # how often hashes will be updated if rotation policy = per_seconds (default is 10s) - requests: 10 + requests: 10 {% endif %} synapseSql: - enabled: {{ matrix_ma1sd_synapsesql_enabled }} - type: {{ matrix_ma1sd_synapsesql_type }} - connection: {{ matrix_ma1sd_synapsesql_connection }} + enabled: {{ matrix_ma1sd_synapsesql_enabled|to_json }} + type: {{ matrix_ma1sd_synapsesql_type|to_json }} + connection: {{ matrix_ma1sd_synapsesql_connection|to_json }} diff --git a/roles/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 b/roles/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 index 3c49ec757..95f15254b 100644 --- a/roles/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 +++ b/roles/matrix-ma1sd/templates/systemd/matrix-ma1sd.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_ma1sd_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-mailer/templates/systemd/matrix-mailer.service.j2 b/roles/matrix-mailer/templates/systemd/matrix-mailer.service.j2 index 1371a8610..9345a1d6d 100644 --- a/roles/matrix-mailer/templates/systemd/matrix-mailer.service.j2 +++ b/roles/matrix-mailer/templates/systemd/matrix-mailer.service.j2 @@ -3,6 +3,7 @@ Description=Matrix mailer After=docker.service Requires=docker.service +DefaultDependencies=no [Service] Type=simple @@ -18,7 +19,7 @@ ExecStart={{ matrix_host_command_docker }} run --rm --name matrix-mailer \ --tmpfs=/var/spool/exim:rw,noexec,nosuid,size=100m \ --network={{ matrix_docker_network }} \ --env-file={{ matrix_mailer_base_path }}/env-mailer \ - --hostname={{ matrix_server_fqn_matrix }} \ + --hostname=matrix-mailer \ {% for arg in matrix_mailer_container_extra_arguments %} {{ arg }} \ {% endfor %} diff --git a/roles/matrix-nginx-proxy/defaults/main.yml b/roles/matrix-nginx-proxy/defaults/main.yml index ceba9ab64..61653db4b 100644 --- a/roles/matrix-nginx-proxy/defaults/main.yml +++ b/roles/matrix-nginx-proxy/defaults/main.yml @@ -3,7 +3,7 @@ matrix_nginx_proxy_enabled: true # We use an official nginx image, which we fix-up to run unprivileged. # An alternative would be an `nginxinc/nginx-unprivileged` image, but # that is frequently out of date. -matrix_nginx_proxy_docker_image: "docker.io/nginx:1.19.4-alpine" +matrix_nginx_proxy_docker_image: "docker.io/nginx:1.19.6-alpine" matrix_nginx_proxy_docker_image_force_pull: "{{ matrix_nginx_proxy_docker_image.endswith(':latest') }}" matrix_nginx_proxy_base_path: "{{ matrix_base_data_path }}/nginx-proxy" @@ -92,6 +92,8 @@ matrix_nginx_proxy_base_domain_homepage_template: |- +# Option to disable the access log +matrix_nginx_proxy_access_log_enabled: true # Controls whether proxying the riot domain should be done. matrix_nginx_proxy_proxy_riot_compat_redirect_enabled: false @@ -259,7 +261,7 @@ matrix_ssl_domains_to_obtain_certificates_for: [] # Controls whether to obtain production or staging certificates from Let's Encrypt. matrix_ssl_lets_encrypt_staging: false -matrix_ssl_lets_encrypt_certbot_docker_image: "docker.io/certbot/certbot:{{ matrix_ssl_architecture }}-v1.9.0" +matrix_ssl_lets_encrypt_certbot_docker_image: "docker.io/certbot/certbot:{{ matrix_ssl_architecture }}-v1.10.1" matrix_ssl_lets_encrypt_certbot_docker_image_force_pull: "{{ matrix_ssl_lets_encrypt_certbot_docker_image.endswith(':latest') }}" matrix_ssl_lets_encrypt_certbot_standalone_http_port: 2402 matrix_ssl_lets_encrypt_support_email: ~ diff --git a/roles/matrix-nginx-proxy/templates/nginx/nginx.conf.j2 b/roles/matrix-nginx-proxy/templates/nginx/nginx.conf.j2 index 51aa8a006..975c8b4f9 100644 --- a/roles/matrix-nginx-proxy/templates/nginx/nginx.conf.j2 +++ b/roles/matrix-nginx-proxy/templates/nginx/nginx.conf.j2 @@ -33,7 +33,11 @@ http { '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; + {% if matrix_nginx_proxy_access_log_enabled %} access_log /var/log/nginx/access.log main; + {% else %} + access_log off; + {% endif %} sendfile on; #tcp_nopush on; diff --git a/roles/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 b/roles/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 index 0945c25ec..58f5c953f 100644 --- a/roles/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 +++ b/roles/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_nginx_proxy_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-postgres/defaults/main.yml b/roles/matrix-postgres/defaults/main.yml index 0d2f50e3e..8f1d0d788 100644 --- a/roles/matrix-postgres/defaults/main.yml +++ b/roles/matrix-postgres/defaults/main.yml @@ -31,4 +31,49 @@ matrix_postgres_container_extra_arguments: [] # Takes an ":" or "" value (e.g. "127.0.0.1:5432"), or empty string to not expose. matrix_postgres_container_postgres_bind_port: "" -matrix_postgres_tool_synapse_janitor: "https://raw.githubusercontent.com/xwiki-labs/synapse_scripts/a9188ff175ae581610f92d58ea6eac9a114d854b/synapse_janitor.sql" +# A list of additional (databases and their credentials) to create. +# +# Example: +# matrix_postgres_additional_databases: +# - name: matrix_appservice_discord +# username: matrix_appservice_discord +# password: some_password +# - name: matrix_appservice_slack +# username: matrix_appservice_slack +# password: some_password +matrix_postgres_additional_databases: [] + +# A list of roles/users to avoid creating when importing (or upgrading) the database. +# If a dump file contains the roles and they've also been created beforehand (see `matrix_postgres_additional_databases`), +# importing would fail. +# We either need to not create them or to ignore the `CREATE ROLE` statements in the dump. +matrix_postgres_import_roles_to_ignore: [matrix_postgres_connection_username] + +matrix_postgres_import_roles_ignore_regex: "^CREATE ROLE ({{ matrix_postgres_import_roles_to_ignore|join('|') }});" + +# A list of databases to avoid creating when importing (or upgrading) the database. +# If a dump file contains the databases and they've also been created beforehand (see `matrix_postgres_additional_databases`), +# importing would fail. +# We either need to not create them or to ignore the `CREATE DATABASE` statements in the dump. +matrix_postgres_import_databases_to_ignore: [matrix_postgres_db_name] + +matrix_postgres_import_databases_ignore_regex: "^CREATE DATABASE ({{ matrix_postgres_import_databases_to_ignore|join('|') }})\\s" + +# The number of seconds to wait after starting `matrix-postgres.service` +# and before trying to run queries for creating additional databases/users against it. +# +# For most (subsequent) runs, Postgres would already be running, so no waiting will be happening at all. +matrix_postgres_additional_databases_postgres_start_wait_timeout_seconds: 15 + + +matrix_postgres_pgloader_container_image_self_build: false +matrix_postgres_pgloader_container_image_self_build_repo: "https://github.com/illagrenan/pgloader-docker.git" +matrix_postgres_pgloader_container_image_self_build_repo_branch: "v{{ matrix_postgres_pgloader_docker_image_tag }}" +matrix_postgres_pgloader_container_image_self_build_src_path: "{{ matrix_postgres_base_path }}/pgloader-container-src" + +# We use illagrenan/pgloader, instead of the more official dimitri/pgloader image, +# because the official one only provides a `latest` tag. +matrix_postgres_pgloader_docker_image: "{{ matrix_postgres_pgloader_docker_image_name_prefix }}illagrenan/pgloader:{{ matrix_postgres_pgloader_docker_image_tag }}" +matrix_postgres_pgloader_docker_image_name_prefix: "{{ 'localhost/' if matrix_postgres_pgloader_container_image_self_build else 'docker.io/' }}" +matrix_postgres_pgloader_docker_image_tag: "3.6.2" +matrix_postgres_pgloader_docker_image_force_pull: "{{ matrix_postgres_pgloader_docker_image.endswith(':latest') }}" diff --git a/roles/matrix-postgres/tasks/import_generic_sqlite_db.yml b/roles/matrix-postgres/tasks/import_generic_sqlite_db.yml new file mode 100644 index 000000000..a42c6f552 --- /dev/null +++ b/roles/matrix-postgres/tasks/import_generic_sqlite_db.yml @@ -0,0 +1,97 @@ +--- + +# Pre-checks + +- name: Fail if Postgres not enabled + fail: + msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot import." + when: "not matrix_postgres_enabled|bool" + +- name: Fail if playbook called incorrectly + fail: + msg: "The `sqlite_database_path` variable needs to be provided to this playbook, via --extra-vars" + when: "sqlite_database_path is not defined or sqlite_database_path.startswith('<')" + +- name: Check if the provided SQLite database file exists + stat: + path: "{{ sqlite_database_path }}" + register: sqlite_database_path_stat_result + +- name: Fail if provided SQLite database file doesn't exist + fail: + msg: "File cannot be found on the server at {{ sqlite_database_path }}" + when: "not sqlite_database_path_stat_result.stat.exists" + +# We either expect `postgres_db_connection_string` specifying a full Postgres database connection string, +# or `postgres_connection_string_variable_name`, specifying a name of a variable, which contains a valid connection string. + +- block: + - name: Fail if postgres_connection_string_variable_name points to an undefined variable + fail: msg="postgres_connection_string_variable_name is defined, but there is no variable with the name `{{ postgres_connection_string_variable_name }}`" + when: "postgres_connection_string_variable_name not in vars" + + - name: Get Postgres connection string from variable + set_fact: + postgres_db_connection_string: "{{ lookup('vars', postgres_connection_string_variable_name) }}" + when: 'postgres_connection_string_variable_name is defined' + +- name: Fail if playbook called incorrectly + fail: + msg: >- + Either a `postgres_db_connection_string` variable or a `postgres_connection_string_variable_name` needs to be provided to this playbook, via `--extra-vars`. + Example: `--extra-vars="postgres_db_connection_string=postgresql://username:password@localhost:/database_name"` or `--extra-vars="postgres_connection_string_variable_name=matrix_appservice_discord_database_connString"` + when: "postgres_db_connection_string is not defined or not postgres_db_connection_string.startswith('postgresql://')" + + +# Defaults + +- name: Set postgres_start_wait_time, if not provided + set_fact: + postgres_start_wait_time: 15 + when: "postgres_start_wait_time|default('') == ''" + + +# Actual import work + +- name: Ensure matrix-postgres is started + service: + name: matrix-postgres + state: started + daemon_reload: yes + register: matrix_postgres_service_start_result + +- name: Wait a bit, so that Postgres can start + wait_for: + timeout: "{{ postgres_start_wait_time }}" + delegate_to: 127.0.0.1 + become: false + when: "matrix_postgres_service_start_result.changed|bool" + +- name: Import SQLite database from {{ sqlite_database_path }} into Postgres + command: + cmd: >- + {{ matrix_host_command_docker }} run + --rm + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} + --cap-drop=ALL + --network={{ matrix_docker_network }} + --mount type=bind,src={{ sqlite_database_path }},dst=/in.db,ro + --entrypoint=/bin/sh + {{ matrix_postgres_pgloader_docker_image }} + -c + 'pgloader /in.db {{ postgres_db_connection_string }}' + +- name: Archive SQLite database ({{ sqlite_database_path }} -> {{ sqlite_database_path }}.backup) + command: + cmd: "mv {{ sqlite_database_path }} {{ sqlite_database_path }}.backup" + +- name: Inject result + set_fact: + matrix_playbook_runtime_results: | + {{ + matrix_playbook_runtime_results|default([]) + + + [ + "NOTE: Your SQLite database file has been imported into Postgres. The original file has been moved from `{{ sqlite_database_path }}` to `{{ sqlite_database_path }}.backup`. When you've confirmed that the import went well and everything works, you should be able to safely delete this file." + ] + }} diff --git a/roles/matrix-postgres/tasks/import_postgres.yml b/roles/matrix-postgres/tasks/import_postgres.yml index 33d98691d..c26affbbb 100644 --- a/roles/matrix-postgres/tasks/import_postgres.yml +++ b/roles/matrix-postgres/tasks/import_postgres.yml @@ -74,8 +74,8 @@ {{ matrix_postgres_docker_image_latest }} -c "cat /{{ server_path_postgres_dump|basename }} | {{ 'gunzip |' if server_path_postgres_dump.endswith('.gz') else '' }} - grep -vE '^CREATE ROLE {{ matrix_postgres_connection_username }}' | - grep -vE '^CREATE DATABASE {{ matrix_postgres_db_name }}' | + grep -vE '{{ matrix_postgres_import_roles_ignore_regex }}' | + grep -vE '{{ matrix_postgres_import_databases_ignore_regex }}' | psql -v ON_ERROR_STOP=1 -h matrix-postgres" # This is a hack. diff --git a/roles/matrix-postgres/tasks/import_sqlite_db.yml b/roles/matrix-postgres/tasks/import_synapse_sqlite_db.yml similarity index 100% rename from roles/matrix-postgres/tasks/import_sqlite_db.yml rename to roles/matrix-postgres/tasks/import_synapse_sqlite_db.yml diff --git a/roles/matrix-postgres/tasks/main.yml b/roles/matrix-postgres/tasks/main.yml index 41b9c8613..b9c2ae7c9 100644 --- a/roles/matrix-postgres/tasks/main.yml +++ b/roles/matrix-postgres/tasks/main.yml @@ -19,21 +19,24 @@ tags: - import-postgres -- import_tasks: "{{ role_path }}/tasks/import_sqlite_db.yml" +# The `run_postgres_import_sqlite_db` variable had better be renamed to be consistent, +# but that's a breaking change which may cause trouble for people. +- import_tasks: "{{ role_path }}/tasks/import_synapse_sqlite_db.yml" when: run_postgres_import_sqlite_db|bool tags: - - import-sqlite-db + - import-synapse-sqlite-db + +# Perhaps we need a new variable here, instead of `run_postgres_import_sqlite_db`. +- import_tasks: "{{ role_path }}/tasks/import_generic_sqlite_db.yml" + when: run_postgres_import_sqlite_db|bool + tags: + - import-generic-sqlite-db - import_tasks: "{{ role_path }}/tasks/upgrade_postgres.yml" when: run_postgres_upgrade|bool tags: - upgrade-postgres -- import_tasks: "{{ role_path }}/tasks/run_synapse_janitor.yml" - when: run_postgres_synapse_janitor|bool - tags: - - run-postgres-synapse-janitor - - import_tasks: "{{ role_path }}/tasks/run_vacuum.yml" when: run_postgres_vacuum|bool tags: diff --git a/roles/matrix-postgres/tasks/run_synapse_janitor.yml b/roles/matrix-postgres/tasks/run_synapse_janitor.yml deleted file mode 100644 index d7f283be2..000000000 --- a/roles/matrix-postgres/tasks/run_synapse_janitor.yml +++ /dev/null @@ -1,117 +0,0 @@ ---- - -# Pre-checks - -- name: Fail if Postgres not enabled - fail: - msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot run synapse-janitor." - when: "not matrix_postgres_enabled|bool" - -- name: Fail if not aware of the risks - fail: - msg: >- - Using Synapse Janitor is considered dangerous and may break your database. - See https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/465. - If you'd like to run it anyway, add `--extra-vars='i_know_synapse_janitor_is_dangerous=1'` to your command. - when: "i_know_synapse_janitor_is_dangerous|default('') == ''" - -# Defaults - -- name: Set postgres_start_wait_time, if not provided - set_fact: - postgres_start_wait_time: 15 - when: "postgres_start_wait_time|default('') == ''" - -- name: Set postgres_synapse_janitor_wait_time, if not provided - set_fact: - postgres_synapse_janitor_wait_time: "{{ 7 * 86400 }}" - when: "postgres_synapse_janitor_wait_time|default('') == ''" - -- name: Set postgres_synapse_janitor_tool_path, if not provided - set_fact: - postgres_synapse_janitor_tool_path: "{{ matrix_postgres_base_path }}/synapse_janitor.sql" - when: "postgres_synapse_janitor_tool_path|default('') == ''" - - -# Actual janitor work - -- name: Download synapse-janitor tool - get_url: - url: "{{ matrix_postgres_tool_synapse_janitor }}" - dest: "{{ postgres_synapse_janitor_tool_path }}" - force: true - mode: 0550 - owner: "{{ matrix_user_username }}" - group: "{{ matrix_user_groupname }}" - -- name: Ensure matrix-postgres is started - service: - name: matrix-postgres - state: started - daemon_reload: yes - -- name: Wait a bit, so that Postgres can start - wait_for: - timeout: "{{ postgres_start_wait_time }}" - delegate_to: 127.0.0.1 - become: false - -- import_tasks: tasks/util/detect_existing_postgres_version.yml - -- name: Abort, if no existing Postgres version detected - fail: - msg: "Could not find existing Postgres installation" - when: "not matrix_postgres_detected_existing|bool" - -- name: Generate Postgres database synapse-janitor command - set_fact: - matrix_postgres_synapse_janitor_command: >- - {{ matrix_host_command_docker }} run --rm --name matrix-postgres-synapse-janitor - --user={{ matrix_user_uid }}:{{ matrix_user_gid }} - --cap-drop=ALL - --network={{ matrix_docker_network }} - --env-file={{ matrix_postgres_base_path }}/env-postgres-psql - --mount type=bind,src={{ postgres_synapse_janitor_tool_path }},dst=/synapse_janitor.sql,ro=true - {{ matrix_postgres_docker_image_latest }} - psql -v ON_ERROR_STOP=1 -h matrix-postgres {{ matrix_synapse_database_database }} -f /synapse_janitor.sql - -- name: Note about Postgres purging alternative - debug: - msg: >- - Running synapse-janitor with the following Postgres command: `{{ matrix_postgres_synapse_janitor_command }}`. - If this crashes, you can stop all processes (`systemctl stop matrix-*`), - start Postgres only (`systemctl start matrix-postgres`) - and manually run the above command directly on the server. - -- name: Populate service facts - service_facts: - -- set_fact: - matrix_postgres_synapse_was_running: "{{ ansible_facts.services['matrix-synapse.service']|default(none) is not none and ansible_facts.services['matrix-synapse.service'].state == 'running' }}" - -- name: Ensure matrix-synapse is stopped - service: - name: matrix-synapse - state: stopped - daemon_reload: yes - -- name: Run synapse-janitor - command: "{{ matrix_postgres_synapse_janitor_command }}" - async: "{{ postgres_synapse_janitor_wait_time }}" - poll: 10 - register: matrix_postgres_synapse_janitor_result - -# Intentionally show the results -- debug: var="matrix_postgres_synapse_janitor_result" - -- name: Ensure matrix-synapse is started, if it previously was - service: - name: matrix-synapse - state: started - daemon_reload: yes - when: "matrix_postgres_synapse_was_running|bool" - -- name: Delete synapse-janitor tool - file: - path: "{{ postgres_synapse_janitor_tool_path }}" - state: absent diff --git a/roles/matrix-postgres/tasks/setup_postgres.yml b/roles/matrix-postgres/tasks/setup_postgres.yml index f186bdca4..518d1a5f0 100644 --- a/roles/matrix-postgres/tasks/setup_postgres.yml +++ b/roles/matrix-postgres/tasks/setup_postgres.yml @@ -113,6 +113,13 @@ daemon_reload: yes when: "matrix_postgres_enabled|bool and matrix_postgres_systemd_service_result.changed" +- include_tasks: + file: "{{ role_path }}/tasks/util/create_additional_databases.yml" + apply: + tags: + - always + when: "matrix_postgres_enabled|bool and matrix_postgres_additional_databases|length > 0" + # # Tasks related to getting rid of the internal postgres server (if it was previously enabled) # diff --git a/roles/matrix-postgres/tasks/upgrade_postgres.yml b/roles/matrix-postgres/tasks/upgrade_postgres.yml index 72f327b35..564265d85 100644 --- a/roles/matrix-postgres/tasks/upgrade_postgres.yml +++ b/roles/matrix-postgres/tasks/upgrade_postgres.yml @@ -135,8 +135,8 @@ {{ matrix_postgres_docker_image_latest }} -c "cat /in/{{ postgres_dump_name }} | {{ 'gunzip |' if postgres_dump_name.endswith('.gz') else '' }} - grep -vE '^CREATE ROLE {{ matrix_postgres_connection_username }}' | - grep -vE '^CREATE DATABASE {{ matrix_postgres_db_name }}' | + grep -vE '{{ matrix_postgres_import_roles_ignore_regex }}' | + grep -vE '{{ matrix_postgres_import_databases_ignore_regex }}' | psql -v ON_ERROR_STOP=1 -h matrix-postgres" # This is a hack. diff --git a/roles/matrix-postgres/tasks/util/create_additional_database.yml b/roles/matrix-postgres/tasks/util/create_additional_database.yml new file mode 100644 index 000000000..ce064d598 --- /dev/null +++ b/roles/matrix-postgres/tasks/util/create_additional_database.yml @@ -0,0 +1,40 @@ +--- + +# It'd be better if this is belonged to `validate_config.yml`, but it would have to be some loop-within-a-loop there, +# and that's ugly. We also don't expect this to catch errors often. It's more of a defensive last-minute check. +- name: Fail if additional database data appears invalid + fail: + msg: "Additional database definition ({{ additional_db }} lacks a required key: {{ item }}" + when: "item not in additional_db" + with_items: "{{ ['name', 'username', 'password'] }}" + +# The SQL statements that we'll run against Postgres are stored in a file that others can't read. +# This file will be mounted into the container and fed to Postgres. +# This way, we avoid passing sensitive data around in CLI commands that other users on the system can see. +- name: Create additional database initialization SQL file for {{ additional_db.name }} + template: + src: "{{ role_path }}/templates/init-additional-db-user-and-role.sql.j2" + dest: "/tmp/matrix-postgres-init-additional-db-user-and-role.sql" + mode: 0600 + owner: "{{ matrix_user_uid }}" + group: "{{ matrix_user_gid }}" + +- name: Execute Postgres additional database initialization SQL file for {{ additional_db.name }} + command: + cmd: >- + {{ matrix_host_command_docker }} run + --rm + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} + --cap-drop=ALL + --env-file={{ matrix_postgres_base_path }}/env-postgres-psql + --network {{ matrix_docker_network }} + --mount type=bind,src=/tmp/matrix-postgres-init-additional-db-user-and-role.sql,dst=/matrix-postgres-init-additional-db-user-and-role.sql,ro + --entrypoint=/bin/sh + {{ matrix_postgres_docker_image_to_use }} + -c + 'psql -h {{ matrix_postgres_connection_hostname }} --file=/matrix-postgres-init-additional-db-user-and-role.sql' + +- name: Delete additional database initialization SQL file for {{ additional_db.name }} + file: + path: /tmp/matrix-postgres-init-additional-db-user-and-role.sql + state: absent diff --git a/roles/matrix-postgres/tasks/util/create_additional_databases.yml b/roles/matrix-postgres/tasks/util/create_additional_databases.yml new file mode 100644 index 000000000..0ad460ddd --- /dev/null +++ b/roles/matrix-postgres/tasks/util/create_additional_databases.yml @@ -0,0 +1,23 @@ +--- + +- name: Ensure matrix-postgres is started + service: + name: matrix-postgres + state: started + daemon_reload: yes + register: matrix_postgres_service_start_result + +- name: Wait a bit, so that Postgres can start + wait_for: + timeout: "{{ matrix_postgres_additional_databases_postgres_start_wait_timeout_seconds }}" + delegate_to: 127.0.0.1 + become: false + when: "matrix_postgres_service_start_result.changed|bool" + +- name: Create additional Postgres user and database + include_tasks: "{{ role_path }}/tasks/util/create_additional_database.yml" + with_items: "{{ matrix_postgres_additional_databases }}" + loop_control: + loop_var: additional_db + # Suppress logging to avoid dumping the credentials to the shell + no_log: true diff --git a/roles/matrix-postgres/tasks/util/migrate_db_to_postgres.yml b/roles/matrix-postgres/tasks/util/migrate_db_to_postgres.yml new file mode 100644 index 000000000..af95815f7 --- /dev/null +++ b/roles/matrix-postgres/tasks/util/migrate_db_to_postgres.yml @@ -0,0 +1,167 @@ +--- + +- name: Fail if Postgres not enabled + fail: + msg: "Postgres via the matrix-postgres role is not enabled (`matrix_postgres_enabled`). Cannot migrate." + when: "not matrix_postgres_enabled|bool" + +- name: Fail if util called incorrectly (missing matrix_postgres_db_migration_request) + fail: + msg: "The `matrix_postgres_db_migration_request` variable needs to be provided to this util." + when: "matrix_postgres_db_migration_request is not defined" + +- name: Fail if util called incorrectly (invalid matrix_postgres_db_migration_request) + fail: + msg: "The `matrix_postgres_db_migration_request` variable needs to contain `{{ item }}`." + with_items: + - src + - dst + - caller + - engine_variable_name + - systemd_services_to_stop + when: "item not in matrix_postgres_db_migration_request" + +- name: Check if the provided source database file exists + stat: + path: "{{ matrix_postgres_db_migration_request.src }}" + register: matrix_postgres_db_migration_request_src_stat_result + +- name: Fail if provided source database file doesn't exist + fail: + msg: "File cannot be found on the server at {{ matrix_postgres_db_migration_request.src }}" + when: "not matrix_postgres_db_migration_request_src_stat_result.stat.exists" + +- block: + - name: Ensure pgloader repository is present on self-build + git: + repo: "{{ matrix_postgres_pgloader_container_image_self_build_repo }}" + dest: "{{ matrix_postgres_pgloader_container_image_self_build_src_path }}" + version: "{{ matrix_postgres_pgloader_container_image_self_build_repo_branch }}" + force: "yes" + register: matrix_postgres_pgloader_git_pull_results + + # If `stable` is used, we hit an error when processing /opt/src/pgloader/build/quicklisp/dists/quicklisp/software/uax-15-20201220-git/data/CompositionExclusions.txt: + # > the octet sequence #(194) cannot be decoded + # + # The issue is described here and is not getting fixed for months: https://github.com/dimitri/pgloader/pull/1179 + # + # Although we're not using the dimitri/pgloader image, the one we're using suffers from the same problem. + - name: Switch pgloader base image from Debian stable (likely 10.x/Buster) to Bullseye + lineinfile: + path: "{{ matrix_postgres_pgloader_container_image_self_build_src_path }}/Dockerfile" + regexp: "{{ item.match }}" + line: "{{ item.replace }}" + with_items: + - match: '^FROM debian:stable-slim as builder$' + replace: 'FROM debian:bullseye-slim as builder' + - match: '^FROM debian:stable-slim$' + replace: 'FROM debian:bullseye-slim' + + - name: Ensure pgloader Docker image is built + docker_image: + name: "{{ matrix_postgres_pgloader_docker_image }}" + source: build + force_source: "{{ matrix_postgres_pgloader_git_pull_results.changed }}" + build: + dockerfile: Dockerfile + path: "{{ matrix_postgres_pgloader_container_image_self_build_src_path }}" + pull: yes + when: "matrix_postgres_pgloader_container_image_self_build|bool" + +- name: Ensure pgloader Docker image is pulled + docker_image: + name: "{{ matrix_postgres_pgloader_docker_image }}" + source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" + force_source: "{{ matrix_postgres_pgloader_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" + force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_postgres_pgloader_docker_image_force_pull }}" + when: "not matrix_postgres_pgloader_container_image_self_build" + +# Defaults + +- name: Set postgres_start_wait_time, if not provided + set_fact: + postgres_start_wait_time: 15 + when: "postgres_start_wait_time|default('') == ''" + +# Actual import work + +# matrix-postgres is most likely started already +- name: Ensure matrix-postgres is started + service: + name: matrix-postgres + state: started + daemon_reload: yes + register: matrix_postgres_service_start_result + +- name: Wait a bit, so that Postgres can start + wait_for: + timeout: "{{ postgres_start_wait_time }}" + delegate_to: 127.0.0.1 + become: false + when: "matrix_postgres_service_start_result.changed|bool" + +# We only stop services here, leaving it to the caller to start them later. +# +# We can't start them, because they probably need to be reconfigured too (changing the configuration from using SQLite to Postgres, etc.), +# before starting. +# +# Since the caller will be starting them, it might make sense to leave stopping to it as well. +# However, we don't do it, because it's simpler having it here, and it also gets to happen only if we'll be doing an import. +# If we bailed out (somewhere above), nothing would have gotten stopped. It's nice to leave this running in such cases. +- name: Ensure systemd services blocking the database import are stopped + service: + name: "{{ item }}" + state: stopped + with_items: "{{ matrix_postgres_db_migration_request.systemd_services_to_stop }}" + +- name: Import {{ matrix_postgres_db_migration_request.engine_old }} database from {{ matrix_postgres_db_migration_request.src }} into Postgres + command: + cmd: >- + {{ matrix_host_command_docker }} run + --rm + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} + --cap-drop=ALL + --network={{ matrix_docker_network }} + --mount type=bind,src={{ matrix_postgres_db_migration_request.src }},dst=/in.db,ro + --entrypoint=/bin/sh + {{ matrix_postgres_pgloader_docker_image }} + -c + 'pgloader {{ matrix_postgres_db_migration_request.pgloader_options|default([])|join(' ') }} /in.db {{ matrix_postgres_db_migration_request.dst }}' + +- block: + # We can't use `{{ role_path }}` here, neither with `import_tasks`, nor with `include_tasks`, + # because it refers to the role that included this util, and not to the role this file belongs to. + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/detect_existing_postgres_version.yml" + + - set_fact: + matrix_postgres_docker_image_to_use: "{{ matrix_postgres_docker_image_latest if matrix_postgres_detected_version_corresponding_docker_image == '' else matrix_postgres_detected_version_corresponding_docker_image }}" + + - name: Execute additional Postgres SQL migration statements + command: + cmd: >- + {{ matrix_host_command_docker }} run + --rm + --user={{ matrix_user_uid }}:{{ matrix_user_gid }} + --cap-drop=ALL + --env-file={{ matrix_postgres_base_path }}/env-postgres-psql + --network={{ matrix_docker_network }} + {{ matrix_postgres_docker_image_to_use }} + psql --host=matrix-postgres --dbname={{ matrix_postgres_db_migration_request.additional_psql_statements_db_name }} --command='{{ item }}' + with_items: "{{ matrix_postgres_db_migration_request.additional_psql_statements_list }}" + + when: "matrix_postgres_db_migration_request.additional_psql_statements_list|default([])|length > 0" + +- name: Archive {{ matrix_postgres_db_migration_request.engine_old }} database ({{ matrix_postgres_db_migration_request.src }} -> {{ matrix_postgres_db_migration_request.src }}.backup) + command: + cmd: "mv {{ matrix_postgres_db_migration_request.src }} {{ matrix_postgres_db_migration_request.src }}.backup" + +- name: Inject result + set_fact: + matrix_playbook_runtime_results: | + {{ + matrix_playbook_runtime_results|default([]) + + + [ + "NOTE: Your {{ matrix_postgres_db_migration_request.engine_old }} database file has been imported into Postgres. The original database file has been moved from `{{ matrix_postgres_db_migration_request.src }}` to `{{ matrix_postgres_db_migration_request.src }}.backup`. When you've confirmed that the import went well and everything works, you should be able to safely delete this file." + ] + }} diff --git a/roles/matrix-postgres/templates/init-additional-db-user-and-role.sql.j2 b/roles/matrix-postgres/templates/init-additional-db-user-and-role.sql.j2 new file mode 100644 index 000000000..609a13448 --- /dev/null +++ b/roles/matrix-postgres/templates/init-additional-db-user-and-role.sql.j2 @@ -0,0 +1,19 @@ +-- `CREATE USER` does not support `IF NOT EXISTS`, so we use this workaround to prevent an error and raise a notice instead. +-- Seen here: https://stackoverflow.com/a/49858797 +DO $$ +BEGIN + CREATE USER {{ additional_db.username }}; + EXCEPTION WHEN DUPLICATE_OBJECT THEN + RAISE NOTICE 'not creating user {{ additional_db.username }}, since it already exists'; +END +$$; + +-- This is useful for initial user creation (since we don't assign a password above) and for handling subsequent password changes +-- TODO - we should escape quotes in the password. +ALTER ROLE {{ additional_db.username }} PASSWORD '{{ additional_db.password }}'; + +-- This will generate an error on subsequent execution +CREATE DATABASE {{ additional_db.name }} WITH LC_CTYPE 'C' LC_COLLATE 'C' OWNER {{ additional_db.username }}; + +-- This is useful for changing the database owner subsequently +ALTER DATABASE {{ additional_db.name }} OWNER TO {{ additional_db.username }}; diff --git a/roles/matrix-postgres/templates/systemd/matrix-postgres.service.j2 b/roles/matrix-postgres/templates/systemd/matrix-postgres.service.j2 index 47f19e9b2..0a935fb06 100644 --- a/roles/matrix-postgres/templates/systemd/matrix-postgres.service.j2 +++ b/roles/matrix-postgres/templates/systemd/matrix-postgres.service.j2 @@ -3,6 +3,7 @@ Description=Matrix Postgres server After=docker.service Requires=docker.service +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-registration/defaults/main.yml b/roles/matrix-registration/defaults/main.yml index 680e8bfe4..065e0c482 100644 --- a/roles/matrix-registration/defaults/main.yml +++ b/roles/matrix-registration/defaults/main.yml @@ -5,15 +5,16 @@ matrix_registration_enabled: true matrix_registration_container_image_self_build: false matrix_registration_container_image_self_build_repo: "https://github.com/ZerataX/matrix-registration" +matrix_registration_container_image_self_build_branch: "{{ 'master' if matrix_registration_version == 'latest' else matrix_registration_version }}" matrix_registration_base_path: "{{ matrix_base_data_path }}/matrix-registration" matrix_registration_config_path: "{{ matrix_registration_base_path }}/config" matrix_registration_data_path: "{{ matrix_registration_base_path }}/data" matrix_registration_docker_src_files_path: "{{ matrix_registration_base_path }}/docker-src" -matrix_registration_version: "v0.7.0" +matrix_registration_version: "v0.7.2" -matrix_registration_docker_image: "{{ matrix_registration_docker_image_name_prefix }}devture/zeratax-matrix-registration:{{ matrix_registration_version }}" +matrix_registration_docker_image: "{{ matrix_registration_docker_image_name_prefix }}zeratax/matrix-registration:{{ matrix_registration_version }}" matrix_registration_docker_image_name_prefix: "{{ 'localhost/' if matrix_registration_container_image_self_build else 'docker.io/' }}" matrix_registration_docker_image_force_pull: "{{ matrix_registration_docker_image.endswith(':latest') }}" @@ -31,10 +32,42 @@ matrix_registration_systemd_wanted_services_list: [] # Takes an ":" or "" value (e.g. "127.0.0.1:8767"), or empty string to not expose. matrix_registration_container_http_host_bind_port: '' +# Database-related configuration fields. +# +# To use SQLite, stick to these defaults. +# +# To use Postgres: +# - change the engine (`matrix_registration_database_engine: 'postgres'`) +# - adjust your database credentials via the `matrix_registration_postgres_*` variables +matrix_registration_database_engine: 'sqlite' + +matrix_registration_sqlite_database_path_local: "{{ matrix_registration_data_path }}/db.sqlite3" +matrix_registration_sqlite_database_path_in_container: "/data/db.sqlite3" + +matrix_registration_database_username: 'matrix_registration' +matrix_registration_database_password: 'some-password' +matrix_registration_database_hostname: 'matrix-postgres' +matrix_registration_database_port: 5432 +matrix_registration_database_name: 'matrix_registration' + +matrix_registration_database_connection_string: 'postgresql://{{ matrix_registration_database_username }}:{{ matrix_registration_database_password }}@{{ matrix_registration_database_hostname }}:{{ matrix_registration_database_port }}/{{ matrix_registration_database_name }}' + +# For some reason, matrix-registraiton expects the `db` field to be like this: `sqlite:////data/db.sqlite3`. +# (seems like one too many slashes, but..) +matrix_registration_db: "{{ + { + 'sqlite': ('sqlite:///' + matrix_registration_sqlite_database_path_in_container), + 'postgres': matrix_registration_database_connection_string, + }[matrix_registration_database_engine] +}}" + + # The path at which Matrix Registration will be exposed on `matrix.DOMAIN` # (only applies when matrix-nginx-proxy is used). matrix_registration_public_endpoint: /matrix-registration +matrix_registration_base_url: "{{ matrix_registration_public_endpoint }}" + matrix_registration_api_register_endpoint: "{{ matrix_homeserver_url }}{{ matrix_registration_public_endpoint }}/register" matrix_registration_api_token_endpoint: "{{ matrix_homeserver_url }}{{ matrix_registration_public_endpoint }}/token" @@ -55,7 +88,6 @@ matrix_registration_admin_secret: "" matrix_registration_riot_instance: "https://riot.im/app/" - # Default matrix-registration configuration template which covers the generic use case. # You can customize it by controlling the various variables inside it. # diff --git a/roles/matrix-registration/tasks/init.yml b/roles/matrix-registration/tasks/init.yml index 8a7cdc75c..bdb3928eb 100644 --- a/roles/matrix-registration/tasks/init.yml +++ b/roles/matrix-registration/tasks/init.yml @@ -30,15 +30,12 @@ {% endif %} {# - Workaround matrix-registration serving static files at /static - (see https://github.com/ZerataX/matrix-registration/issues/29) - - Also fixing the form, which goes to /register. + Workaround matrix-registration serving the background image at /static + (see https://github.com/ZerataX/matrix-registration/issues/47) #} sub_filter_once off; - sub_filter_types text/html text/css; + sub_filter_types text/css; sub_filter "/static/" "{{ matrix_registration_public_endpoint }}/static/"; - sub_filter "/register" "{{ matrix_registration_public_endpoint }}/register"; } - name: Register matrix-registration proxying configuration with matrix-nginx-proxy diff --git a/roles/matrix-registration/tasks/main.yml b/roles/matrix-registration/tasks/main.yml index 4a884ccd9..4fef6abe9 100644 --- a/roles/matrix-registration/tasks/main.yml +++ b/roles/matrix-registration/tasks/main.yml @@ -8,7 +8,14 @@ - setup-all - setup-matrix-registration -- import_tasks: "{{ role_path }}/tasks/setup.yml" +- import_tasks: "{{ role_path }}/tasks/setup_install.yml" + when: "run_setup|bool and matrix_registration_enabled|bool" + tags: + - setup-all + - setup-matrix-registration + +- import_tasks: "{{ role_path }}/tasks/setup_uninstall.yml" + when: "run_setup|bool and not matrix_registration_enabled|bool" tags: - setup-all - setup-matrix-registration diff --git a/roles/matrix-registration/tasks/setup.yml b/roles/matrix-registration/tasks/setup_install.yml similarity index 54% rename from roles/matrix-registration/tasks/setup.yml rename to roles/matrix-registration/tasks/setup_install.yml index 8427d9509..2b806fe04 100644 --- a/roles/matrix-registration/tasks/setup.yml +++ b/roles/matrix-registration/tasks/setup_install.yml @@ -1,8 +1,35 @@ --- -# -# Tasks related to setting up matrix-registration -# +- set_fact: + matrix_registration_requires_restart: false + +- block: + - name: Check if an SQLite database already exists + stat: + path: "{{ matrix_registration_sqlite_database_path_local }}" + register: matrix_registration_sqlite_database_path_local_stat_result + + - block: + - set_fact: + matrix_postgres_db_migration_request: + src: "{{ matrix_registration_sqlite_database_path_local }}" + dst: "{{ matrix_registration_database_connection_string }}" + caller: "{{ role_path|basename }}" + engine_variable_name: 'matrix_registration_database_engine' + engine_old: 'sqlite' + systemd_services_to_stop: ['matrix-registration.service'] + # pgloader makes `ex_date` of type `TIMESTAMP WITH TIMEZONE`, + # which makes matrix-registration choke on it later on when comparing dates. + additional_psql_statements_list: + - ALTER TABLE tokens ALTER COLUMN ex_date TYPE TIMESTAMP WITHOUT TIME ZONE; + additional_psql_statements_db_name: "{{ matrix_registration_database_name }}" + + - import_tasks: "{{ role_path }}/../matrix-postgres/tasks/util/migrate_db_to_postgres.yml" + + - set_fact: + matrix_registration_requires_restart: true + when: "matrix_registration_sqlite_database_path_local_stat_result.stat.exists|bool" + when: "matrix_registration_database_engine == 'postgres'" - name: Ensure matrix-registration paths exist file: @@ -16,7 +43,7 @@ - { path: "{{ matrix_registration_config_path }}", when: true } - { path: "{{ matrix_registration_data_path }}", when: true } - { path: "{{ matrix_registration_docker_src_files_path }}", when: "{{ matrix_registration_container_image_self_build }}"} - when: matrix_registration_enabled|bool and item.when + when: "item.when|bool" - name: Ensure matrix-registration image is pulled docker_image: @@ -24,16 +51,16 @@ source: "{{ 'pull' if ansible_version.major > 2 or ansible_version.minor > 7 else omit }}" force_source: "{{ matrix_registration_docker_image_force_pull if ansible_version.major > 2 or ansible_version.minor >= 8 else omit }}" force: "{{ omit if ansible_version.major > 2 or ansible_version.minor >= 8 else matrix_registration_docker_image_force_pull }}" - when: "matrix_registration_enabled|bool and not matrix_registration_container_image_self_build|bool" + when: "not matrix_registration_container_image_self_build|bool" - name: Ensure matrix-registration repository is present when self-building git: repo: "{{ matrix_registration_container_image_self_build_repo }}" dest: "{{ matrix_registration_docker_src_files_path }}" - version: "{{ matrix_registration_version }}" + version: "{{ matrix_registration_container_image_self_build_branch }}" force: "yes" register: matrix_registration_git_pull_results - when: "matrix_registration_enabled|bool and matrix_registration_container_image_self_build|bool" + when: "matrix_registration_container_image_self_build|bool" - name: Ensure matrix-registration Docker image is built docker_image: @@ -44,7 +71,7 @@ dockerfile: Dockerfile path: "{{ matrix_registration_docker_src_files_path }}" pull: yes - when: "matrix_registration_enabled|bool and matrix_registration_container_image_self_build|bool" + when: "matrix_registration_container_image_self_build|bool" - name: Ensure matrix-registration config installed copy: @@ -53,7 +80,6 @@ mode: 0644 owner: "{{ matrix_user_username }}" group: "{{ matrix_user_groupname }}" - when: matrix_registration_enabled|bool - name: Ensure matrix-registration.service installed template: @@ -61,43 +87,14 @@ dest: "{{ matrix_systemd_path }}/matrix-registration.service" mode: 0644 register: matrix_registration_systemd_service_result - when: matrix_registration_enabled|bool - name: Ensure systemd reloaded after matrix-registration.service installation service: daemon_reload: yes - when: "matrix_registration_enabled|bool and matrix_registration_systemd_service_result.changed" + when: "matrix_registration_systemd_service_result.changed|bool" -# -# Tasks related to getting rid of matrix-registration (if it was previously enabled) -# - -- name: Check existence of matrix-registration service - stat: - path: "{{ matrix_systemd_path }}/matrix-registration.service" - register: matrix_registration_service_stat - -- name: Ensure matrix-registration is stopped +- name: Ensure matrix-registration.service restarted, if necessary service: - name: matrix-registration - state: stopped - daemon_reload: yes - register: stopping_result - when: "not matrix_registration_enabled|bool and matrix_registration_service_stat.stat.exists" - -- name: Ensure matrix-registration.service doesn't exist - file: - path: "{{ matrix_systemd_path }}/matrix-registration.service" - state: absent - when: "not matrix_registration_enabled|bool and matrix_registration_service_stat.stat.exists" - -- name: Ensure systemd reloaded after matrix-registration.service removal - service: - daemon_reload: yes - when: "not matrix_registration_enabled|bool and matrix_registration_service_stat.stat.exists" - -- name: Ensure matrix-registration Docker image doesn't exist - docker_image: - name: "{{ matrix_registration_docker_image }}" - state: absent - when: "not matrix_registration_enabled|bool" + name: "matrix-registration.service" + state: restarted + when: "matrix_registration_requires_restart|bool" diff --git a/roles/matrix-registration/tasks/setup_uninstall.yml b/roles/matrix-registration/tasks/setup_uninstall.yml new file mode 100644 index 000000000..573f8170b --- /dev/null +++ b/roles/matrix-registration/tasks/setup_uninstall.yml @@ -0,0 +1,30 @@ +--- + +- name: Check existence of matrix-registration service + stat: + path: "{{ matrix_systemd_path }}/matrix-registration.service" + register: matrix_registration_service_stat + +- name: Ensure matrix-registration is stopped + service: + name: matrix-registration + state: stopped + daemon_reload: yes + register: stopping_result + when: "matrix_registration_service_stat.stat.exists|bool" + +- name: Ensure matrix-registration.service doesn't exist + file: + path: "{{ matrix_systemd_path }}/matrix-registration.service" + state: absent + when: "matrix_registration_service_stat.stat.exists|bool" + +- name: Ensure systemd reloaded after matrix-registration.service removal + service: + daemon_reload: yes + when: "matrix_registration_service_stat.stat.exists|bool" + +- name: Ensure matrix-registration Docker image doesn't exist + docker_image: + name: "{{ matrix_registration_docker_image }}" + state: absent diff --git a/roles/matrix-registration/templates/config.yaml.j2 b/roles/matrix-registration/templates/config.yaml.j2 index f3b1c57b7..39211b249 100644 --- a/roles/matrix-registration/templates/config.yaml.j2 +++ b/roles/matrix-registration/templates/config.yaml.j2 @@ -3,7 +3,7 @@ server_name: {{ matrix_registration_server_name|to_json }} shared_secret: {{ matrix_registration_shared_secret|to_json }} admin_secret: {{ matrix_registration_admin_secret|to_json }} riot_instance: {{ matrix_registration_riot_instance|to_json }} -db: 'sqlite:////data/db.sqlite3' +db: {{ matrix_registration_db|to_json }} host: '0.0.0.0' port: 5000 rate_limit: ["100 per day", "10 per minute"] @@ -28,3 +28,4 @@ logging: # password requirements password: min_length: 8 +base_url: {{ matrix_registration_base_url|to_json }} diff --git a/roles/matrix-registration/templates/systemd/matrix-registration.service.j2 b/roles/matrix-registration/templates/systemd/matrix-registration.service.j2 index 052b7d95d..3744c2de0 100644 --- a/roles/matrix-registration/templates/systemd/matrix-registration.service.j2 +++ b/roles/matrix-registration/templates/systemd/matrix-registration.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_registration_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-synapse-admin/defaults/main.yml b/roles/matrix-synapse-admin/defaults/main.yml index ce149dfd4..1dbf0ad2d 100644 --- a/roles/matrix-synapse-admin/defaults/main.yml +++ b/roles/matrix-synapse-admin/defaults/main.yml @@ -8,7 +8,7 @@ matrix_synapse_admin_container_self_build_repo: "https://github.com/Awesome-Tech matrix_synapse_admin_docker_src_files_path: "{{ matrix_base_data_path }}/synapse-admin/docker-src" -matrix_synapse_admin_docker_image: "{{ matrix_synapse_admin_docker_image_name_prefix }}awesometechnologies/synapse-admin:0.5.0" +matrix_synapse_admin_docker_image: "{{ matrix_synapse_admin_docker_image_name_prefix }}awesometechnologies/synapse-admin:0.6.1" matrix_synapse_admin_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_admin_container_self_build else 'docker.io/' }}" matrix_synapse_admin_docker_image_force_pull: "{{ matrix_synapse_admin_docker_image.endswith(':latest') }}" diff --git a/roles/matrix-synapse-admin/templates/systemd/matrix-synapse-admin.service.j2 b/roles/matrix-synapse-admin/templates/systemd/matrix-synapse-admin.service.j2 index c03c627bf..7b1e40de1 100644 --- a/roles/matrix-synapse-admin/templates/systemd/matrix-synapse-admin.service.j2 +++ b/roles/matrix-synapse-admin/templates/systemd/matrix-synapse-admin.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_synapse_admin_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-synapse/defaults/main.yml b/roles/matrix-synapse/defaults/main.yml index f5c2c4333..0dc71646c 100644 --- a/roles/matrix-synapse/defaults/main.yml +++ b/roles/matrix-synapse/defaults/main.yml @@ -6,8 +6,12 @@ matrix_synapse_enabled: true matrix_synapse_container_image_self_build: false matrix_synapse_container_image_self_build_repo: "https://github.com/matrix-org/synapse.git" -matrix_synapse_docker_image: "{{ matrix_synapse_docker_image_name_prefix }}matrixdotorg/synapse:v1.23.0" +matrix_synapse_docker_image: "{{ matrix_synapse_docker_image_name_prefix }}matrixdotorg/synapse:{{ matrix_synapse_docker_image_tag }}" matrix_synapse_docker_image_name_prefix: "{{ 'localhost/' if matrix_synapse_container_image_self_build else 'docker.io/' }}" +# The if statement below may look silly at times (leading to the same version being returned), +# but ARM-compatible container images are only released 1-7 hours after a release, +# so we may often be on different versions for different architectures when new Synapse releases come out. +matrix_synapse_docker_image_tag: "{{ 'v1.24.0' if matrix_architecture == 'amd64' else 'v1.24.0' }}" matrix_synapse_docker_image_force_pull: "{{ matrix_synapse_docker_image.endswith(':latest') }}" matrix_synapse_base_path: "{{ matrix_base_data_path }}/synapse" @@ -136,6 +140,11 @@ matrix_synapse_tls_private_key_path: "/data/{{ matrix_server_fqn_matrix }}.tls.k # names. matrix_synapse_http_listener_resource_names: ["client"] +# Resources served on Synapse's federation port. +# When disabling federation, we may wish to serve the `openid` resource here, +# so that services like Dimension and ma1sd can work. +matrix_synapse_federation_listener_resource_names: "{{ ['federation'] if matrix_synapse_federation_enabled else (['openid'] if matrix_synapse_federation_port_openid_resource_required else []) }}" + # Enable this to allow Synapse to report utilization statistics about your server to matrix.org # (things like number of users, number of messages sent, uptime, load, etc.) matrix_synapse_report_stats: false @@ -211,10 +220,25 @@ matrix_synapse_caches_global_factor: 0.5 # Controls whether Synapse will federate at all. # Disable this to completely isolate your server from the rest of the Matrix network. -# Also see: `matrix_synapse_tls_federation_listener_enabled` if you wish to keep federation enabled, +# +# Disabling this still keeps the federation port exposed, because it may be used for other services (`openid`). +# +# Also see: +# - `matrix_synapse_tls_federation_listener_enabled` if you wish to keep federation enabled, # but want to stop the TLS listener (port 8448). +# - `matrix_synapse_federation_port_enabled` to avoid exposing the federation ports matrix_synapse_federation_enabled: true +# Controls whether the federation ports are used at all. +# One may wish to disable federation (`matrix_synapse_federation_enabled: true`), +# but still run other resources (like `openid`) on the federation port +# by enabling them in `matrix_synapse_federation_listener_resource_names`. +matrix_synapse_federation_port_enabled: "{{ matrix_synapse_federation_enabled or matrix_synapse_federation_port_openid_resource_required }}" + +# Controls whether an `openid` listener is to be enabled. Useful when disabling federation, +# but needing the `openid` APIs for Dimension or an identity server like ma1sd. +matrix_synapse_federation_port_openid_resource_required: false + # A list of domain names that are allowed to federate with the given Synapse server. # An empty list value (`[]`) will also effectively stop federation, but if that's the desired # result, it's better to accomplish it by changing `matrix_synapse_federation_enabled`. diff --git a/roles/matrix-synapse/tasks/main.yml b/roles/matrix-synapse/tasks/main.yml index 48e72acec..e366a13ff 100644 --- a/roles/matrix-synapse/tasks/main.yml +++ b/roles/matrix-synapse/tasks/main.yml @@ -17,7 +17,7 @@ - import_tasks: "{{ role_path }}/tasks/import_media_store.yml" when: run_synapse_import_media_store|bool tags: - - import-media-store + - import-synapse-media-store - import_tasks: "{{ role_path }}/tasks/register_user.yml" when: run_synapse_register_user|bool diff --git a/roles/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 b/roles/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 index c3ec92491..0bbfde992 100644 --- a/roles/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 +++ b/roles/matrix-synapse/templates/goofys/systemd/matrix-goofys.service.j2 @@ -3,6 +3,7 @@ Description=Matrix Goofys media store After=docker.service Requires=docker.service +DefaultDependencies=no [Service] Type=simple diff --git a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 index 8c68189bc..41b28c88e 100644 --- a/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 +++ b/roles/matrix-synapse/templates/synapse/homeserver.yaml.j2 @@ -188,7 +188,7 @@ listeners: - '0.0.0.0' {% endif %} -{% if matrix_synapse_federation_enabled and matrix_synapse_tls_federation_listener_enabled %} +{% if matrix_synapse_federation_port_enabled and matrix_synapse_tls_federation_listener_enabled %} # TLS-enabled listener: for when matrix traffic is sent directly to synapse. - port: 8448 tls: true @@ -197,7 +197,7 @@ listeners: x_forwarded: false resources: - - names: [federation] + - names: {{ matrix_synapse_federation_listener_resource_names|to_json }} compress: false {% endif %} @@ -213,7 +213,7 @@ listeners: - names: {{ matrix_synapse_http_listener_resource_names|to_json }} compress: false -{% if matrix_synapse_federation_enabled %} +{% if matrix_synapse_federation_port_enabled %} # Unsecure HTTP listener (Federation API): for when matrix traffic passes through a reverse proxy # that unwraps TLS. - port: 8048 @@ -223,7 +223,7 @@ listeners: x_forwarded: true resources: - - names: [federation] + - names: {{ matrix_synapse_federation_listener_resource_names|to_json }} compress: false {% endif %} @@ -1202,8 +1202,9 @@ allow_guest_access: {{ matrix_synapse_allow_guest_access|to_json }} # email will be globally disabled. # # Additionally, if `msisdn` is not set, registration and password resets via msisdn -# will be disabled regardless. This is due to Synapse currently not supporting any -# method of sending SMS messages on its own. +# will be disabled regardless, and users will not be able to associate an msisdn +# identifier to their account. This is due to Synapse currently not supporting +# any method of sending SMS messages on its own. # # To enable using an identity server for operations regarding a particular third-party # identifier type, set the value to the URL of that identity server as shown in the @@ -1522,6 +1523,12 @@ saml2_config: # remote: # - url: https://our_idp/metadata.xml + # Allowed clock difference in seconds between the homeserver and IdP. + # + # Uncomment the below to increase the accepted time difference from 0 to 3 seconds. + # + #accepted_time_diff: 3 + # By default, the user has to go to our login page first. If you'd like # to allow IdP-initiated login, set 'allow_unsolicited: true' in a # 'service.sp' section: @@ -1644,6 +1651,14 @@ saml2_config: # - attribute: department # value: "sales" + # If the metadata XML contains multiple IdP entities then the `idp_entityid` + # option must be set to the entity to redirect users to. + # + # Most deployments only have a single IdP entity and so should omit this + # option. + # + #idp_entityid: 'https://our_idp/entityid' + # Enable OpenID Connect (OIDC) / OAuth 2.0 for registration and login. # @@ -2222,21 +2237,35 @@ password_providers: {% endif %} +## Push ## -# Clients requesting push notifications can either have the body of -# the message sent in the notification poke along with other details -# like the sender, or just the event ID and room ID (`event_id_only`). -# If clients choose the former, this option controls whether the -# notification request includes the content of the event (other details -# like the sender are still included). For `event_id_only` push, it -# has no effect. -# -# For modern android devices the notification content will still appear -# because it is loaded by the app. iPhone, however will send a -# notification saying only that a message arrived and who it came from. -# push: - include_content: {{ matrix_synapse_push_include_content|to_json }} + # Clients requesting push notifications can either have the body of + # the message sent in the notification poke along with other details + # like the sender, or just the event ID and room ID (`event_id_only`). + # If clients choose the former, this option controls whether the + # notification request includes the content of the event (other details + # like the sender are still included). For `event_id_only` push, it + # has no effect. + # + # For modern android devices the notification content will still appear + # because it is loaded by the app. iPhone, however will send a + # notification saying only that a message arrived and who it came from. + # + # The default value is "true" to include message details. Uncomment to only + # include the event ID and room ID in push notification payloads. + # + include_content: {{ matrix_synapse_push_include_content|to_json }} + + # When a push notification is received, an unread count is also sent. + # This number can either be calculated as the number of unread messages + # for the user, or the number of *rooms* the user has unread messages in. + # + # The default value is "true", meaning push clients will see the number of + # rooms with unread messages in them. Uncomment to instead send the number + # of unread messages. + # + #group_unread_count_by_room: false # Spam checkers are third-party modules that can block specific actions diff --git a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 index 791d324e7..30c85b999 100644 --- a/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 +++ b/roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2 @@ -8,6 +8,7 @@ After={{ service }} {% for service in matrix_synapse_systemd_wanted_services_list %} Wants={{ service }} {% endfor %} +DefaultDependencies=no [Service] Type=simple