Reorganize Postgres access (breaking change)

In short, this makes Synapse a 2nd class citizen,
preparing for a future where it's just one-of-many homeserver software
options.

We also no longer have a default Postgres superuser password,
which improves security.

The changelog explains more as to why this was done
and how to proceed from here.

CHANGELOG.md

@@ -1,3 +1,85 @@
+# 2021-01-22
+
+## (Breaking Change) Postgres changes that require manual intervention
+
+We've made a lot of changes to our Postgres setup and some manual action is required (described below). Sorry about the hassle.
+
+**TLDR**: people running an [external Postgres server](docs/configuring-playbook-external-postgres.md) don't need to change anything for now. Everyone else (the common/default case) is affected and manual intervention is required.
+
+### Why?
+
+- we had a default Postgres password (`matrix_postgres_connection_password: synapse-password`), which we think is **not ideal for security anymore**. We now ask you to generate/provide a strong password yourself. Postgres is normally not exposed outside the container network, making it relatively secure, but still:
+  - by tweaking the configuration, you may end up intentionally or unintentionally exposing your Postgres server to the local network (or even publicly), while still using the default default credentials (`synapse` + `synapse-password`)
+  - we can't be sure we trust all these services (bridges, etc). Some of them may try to talk to or attack `matrix-postgres` using the default credentials (`synapse` + `synapse-password`)
+  - you may have other containers running on the same Docker network, which may try to talk to or attack `matrix-postgres` using the default credentials (`synapse` + `synapse-password`)
+- our Postgres usage **was overly-focused on Synapse** (default username of `synapse` and default/main database of `homeserver`). Additional homeserver options are likely coming in the future ([Dendrite](https://matrix.org/docs/projects/server/dendrite), [Conduit](https://matrix.org/docs/projects/server/conduit), [The Construct](https://matrix.org/docs/projects/server/construct)), so being too focused on `matrix-synapse` is not great. From now on, Synapse is just another component of this playbook, which happens to have an *additional database* (called `synapse`) on the Postgres server.
+- we try to reorganize things a bit, to make the playbook even friendlier to people running an [external Postgres server](docs/configuring-playbook-external-postgres.md). Work on this will proceed in the future.
+
+So, this is some **effort to improve security** and to **prepare for a brighter future of having more homeserver options** than just Synapse.
+
+### What has really changed?
+
+- the default superuser Postgres username is now `matrix` (used to be `synapse`)
+- the default Postgres database is now `matrix` (used to be `homeserver`)
+- Synapse's database is now `synapse` (used to be `homeserver`). This is now just another "additional database" that the playbook manages for you
+- Synapse's user called `synapse` is just a regular user that can only use the `synapse` database (not a superuser anymore)
+
+### What do I do if I'm using the integrated Postgres server (default)?
+
+By default, the playbook runs an integrated Postgres server for you in a container (`matrix-postgres`). Unless you've explicitly configured an [external Postgres server](docs/configuring-playbook-external-postgres.md), these steps are meant for you.
+
+To migrate to the new setup, expect a few minutes of downtime, while you follow these steps:
+
+1. Generate a strong password to be used for your superuser Postgres user (called `matrix`). You can use `pwgen -s 64 1` to generate it, or some other tool.
+
+2. Update your playbook's `inventory/host_vars/matrix.DOMAIN/vars.yml` file, adding a line like this:
+```yaml
+matrix_postgres_connection_password: YOUR_POSTGRES_PASSWORD_HERE
+```
+
+.. where `YOUR_POSTGRES_PASSWORD_HERE` is to be replaced with the password you generated during step #1.
+
+3. Stop all services: `ansible-playbook -i inventory/hosts setup.yml --tags=stop`
+4. Log in to the server via SSH. The next commands will be performed there.
+5. Start the Postgres database server: `systemctl start matrix-postgres`
+6. Open a Postgres shell: `/usr/local/bin/matrix-postgres-cli`
+7. Execute the following query, while making sure to **change the password inside**:
+
+```sql
+CREATE ROLE matrix LOGIN SUPERUSER PASSWORD 'YOUR_POSTGRES_PASSWORD_HERE';
+```
+
+.. where `YOUR_POSTGRES_PASSWORD_HERE` is to be replaced with the password you generated during step #1.
+
+8. Execute the following queries as you see them (no modifications necessary, so you can just paste them):
+
+```sql
+CREATE DATABASE matrix OWNER matrix;
+
+ALTER DATABASE postgres OWNER TO matrix;
+ALTER DATABASE template0 OWNER TO matrix;
+ALTER DATABASE template1 OWNER TO matrix;
+
+\c matrix;
+
+ALTER DATABASE homeserver RENAME TO synapse;
+
+ALTER ROLE synapse NOSUPERUSER NOCREATEDB NOCREATEROLE;
+
+\quit
+```
+
+You may need to press *Enter* after pasting the lines above.
+
+1. Re-run the playbook normally: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start`
+
+### What do I do if I'm using an external Postgres server?
+
+If you've explicitly configured an [external Postgres server](docs/configuring-playbook-external-postgres.md), there are changes that you need to do at this time.
+
+The fact that we've renamed Synapse's database from `homeserver` to `synapse` (in our defaults) should not affect you, as you're already explicitly defining `matrix_synapse_database_database` (if you've followed our guide, that is). If you're not explicitly defining this variable, you may wish to do so (`matrix_synapse_database_database: homeserver`), to avoid the new `synapse` default and keep things as they were.
+
+
 # 2021-01-20
 
 ## (Breaking Change) The mautrix-facebook bridge now requires a Postgres database

docs/configuring-playbook-external-postgres.md

@@ -4,6 +4,7 @@ By default, this playbook would set up a PostgreSQL database server on your mach
 If that's alright, you can skip this.
 
 If you'd like to use an external PostgreSQL server that you manage, you can edit your configuration file  (`inventory/host_vars/matrix.<your-domain>/vars.yml`).
+
 It should be something like this:
 
 ```yaml

docs/maintenance-postgres.md

@@ -19,6 +19,17 @@ You can use the `/usr/local/bin/matrix-postgres-cli` tool to get interactive ter
 
 If you are using an [external Postgres server](configuring-playbook-external-postgres.md), the above tool will not be available.
 
+By default, this tool puts you in the `matrix` database, which contains nothing.
+
+To see the available databases, run `\list` (or just `\l`).
+
+To change to another database (for example `synapse`), run `\connect synapse` (or just `\c synapse`).
+
+You can then proceed to write queries. Example: `SELECT COUNT(*) FROM users;`
+
+**Be careful**. Modifying the database directly (especially as services are running) is dangerous and may lead to irreversible database corruption.
+When in doubt, consider [making a backup](#backing-up-postgresql).
+
 
 ## Vacuuming PostgreSQL
 

examples/vars.yml

@@ -18,12 +18,18 @@ matrix_domain: YOUR_BARE_DOMAIN_NAME_HERE
 # you won't be required to define this variable (see `docs/configuring-playbook-ssl-certificates.md`).
 #
 # Example value: someone@example.com
-matrix_ssl_lets_encrypt_support_email: YOUR_EMAIL_ADDRESS_HERE
+matrix_ssl_lets_encrypt_support_email: ''
 
 # A shared secret (between Coturn and Synapse) used for authentication.
 # You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`).
-matrix_coturn_turn_static_auth_secret: ""
+matrix_coturn_turn_static_auth_secret: ''
 
 # A secret used to protect access keys issued by the server.
 # You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`).
-matrix_synapse_macaroon_secret_key: ""
+matrix_synapse_macaroon_secret_key: ''
+
+# A Postgres password to used for the superuser Postgres user (called `matrix` by default).
+#
+# The playbook creates additional Postgres users and databases (one for each enabled service)
+# using this superuser account.
+matrix_postgres_connection_password: ''

group_vars/matrix_servers

@@ -1021,16 +1021,20 @@ matrix_ssl_pre_obtaining_required_service_name: "{{ 'matrix-dynamic-dns' if matr
 
 matrix_postgres_enabled: true
 
-matrix_postgres_connection_hostname: "matrix-postgres"
+# We unset this if internal Postgres disabled, which will cascade to some other variables
-matrix_postgres_connection_username: "synapse"
+# and tell users they need to set it (either here or in those variables).
-# Please note that the max length of the password is 99 characters
+matrix_postgres_connection_hostname: "{{ 'matrix-postgres' if matrix_postgres_enabled else '' }}"
-matrix_postgres_connection_password: "synapse-password"
-matrix_postgres_db_name: "homeserver"
 
 matrix_postgres_pgloader_container_image_self_build: "{{ matrix_architecture != 'amd64' }}"
 
 matrix_postgres_additional_databases: |
   {{
+    ([{
+      'name': matrix_synapse_database_database,
+      'username': matrix_synapse_database_user,
+      'password': matrix_synapse_database_password,
+    }] if (matrix_synapse_enabled and matrix_synapse_database_database != matrix_postgres_db_name and matrix_synapse_database_host == 'matrix-postgres') else [])
+    +
     ([{
       'name': matrix_ma1sd_database_name,
       'username': matrix_ma1sd_database_username,
@@ -1243,10 +1247,7 @@ matrix_synapse_container_metrics_api_host_bind_port: "{{ '127.0.0.1:9100' if (ma
 # For exposing the Synapse Manhole port (plain HTTP) to the local host.
 matrix_synapse_container_manhole_api_host_bind_port: "{{ '127.0.0.1:9000' if matrix_synapse_manhole_enabled else '' }}"
 
-matrix_synapse_database_host: "{{ matrix_postgres_connection_hostname }}"
+matrix_synapse_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'synapse.db') | to_uuid }}"
-matrix_synapse_database_user: "{{ matrix_postgres_connection_username }}"
-matrix_synapse_database_password: "{{ matrix_postgres_connection_password }}"
-matrix_synapse_database_database: "{{ matrix_postgres_db_name }}"
 
 # We do not enable TLS in Synapse by default.
 # TLS is handled by the matrix-nginx-proxy, which proxies the requests to Synapse.

roles/matrix-postgres/defaults/main.yml

@@ -1,9 +1,10 @@
 matrix_postgres_enabled: true
 
-matrix_postgres_connection_hostname: ""
+matrix_postgres_connection_hostname: "matrix-postgres"
-matrix_postgres_connection_username: ""
+matrix_postgres_connection_port: 5432
+matrix_postgres_connection_username: "matrix"
 matrix_postgres_connection_password: ""
-matrix_postgres_db_name: ""
+matrix_postgres_db_name: "matrix"
 
 matrix_postgres_base_path: "{{ matrix_base_data_path }}/postgres"
 matrix_postgres_data_path: "{{ matrix_postgres_base_path }}/data"

roles/matrix-postgres/tasks/validate_config.yml

@@ -6,17 +6,29 @@
       The `matrix_postgres_use_external` variable defined in your configuration is not used by this playbook anymore!
       You'll need to adapt to the new way of using an external Postgres server.
       It's a combination of `matrix_postgres_enabled: false` and specifying Postgres connection
-      details in a few `matrix_synapse_database_` variables.
+      details in a few `matrix_postgres_connection_` variables.
       See the "Using an external PostgreSQL server (optional)" documentation page.
   when: "'matrix_postgres_use_external' in vars"
 
+# This is separate (from the other required variables below),
+# because we'd like to have a friendlier message for our existing users.
+- name: Fail if matrix_postgres_connection_password not defined
+  fail:
+    msg: >-
+      The playbook no longer has a default Postgres password defined in the `matrix_postgres_connection_password` variable, among lots of other Postgres changes.
+      You need to perform multiple manual steps to resolve this.
+      See our changelog for more details:
+      https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#breaking-change-postgres-changes-that-require-manual-intervention
+  when: "matrix_postgres_connection_password == ''"
+
 - name: Fail if required Postgres settings not defined
   fail:
-    msg: >
+    msg: >-
       You need to define a required configuration setting (`{{ item }}`).
   when: "vars[item] == ''"
   with_items:
     - "matrix_postgres_connection_hostname"
+    - "matrix_postgres_connection_port"
     - "matrix_postgres_connection_username"
     - "matrix_postgres_connection_password"
     - "matrix_postgres_db_name"

roles/matrix-synapse/defaults/main.yml

@@ -294,10 +294,10 @@ matrix_synapse_manhole_enabled: false
 matrix_synapse_sentry_dsn: ""
 
 # Postgres database information
-matrix_synapse_database_host: ""
+matrix_synapse_database_host: "matrix-postgres"
-matrix_synapse_database_user: ""
+matrix_synapse_database_user: "synapse"
 matrix_synapse_database_password: ""
-matrix_synapse_database_database: ""
+matrix_synapse_database_database: "synapse"
 
 matrix_synapse_turn_uris: []
 matrix_synapse_turn_shared_secret: ""

roles/matrix-synapse/tasks/validate_config.yml

@@ -7,6 +7,10 @@
   when: "vars[item] == ''"
   with_items:
     - "matrix_synapse_macaroon_secret_key"
+    - "matrix_synapse_database_host"
+    - "matrix_synapse_database_user"
+    - "matrix_synapse_database_password"
+    - "matrix_synapse_database_database"
 
 - name: (Deprecation) Catch and report renamed settings
   fail: