Upgrade mautrix-facebook to the new Postgres-only version

I had intentionally held it back in 39ea3496a4
until:
- it received more testing (there were a few bugs during the
migration, but now it seems OK)
- this migration guide was written

CHANGELOG.md

@@ -1,3 +1,63 @@
+# 2021-01-20
+
+## (Breaking Change) The mautrix-facebook bridge now requires a Postgres database
+
+A new version of the [mautrix-facebook](https://github.com/tulir/mautrix-facebook) bridge has been released. It's a full rewrite of its backend and the bridge now requires Postgres. New versions of the bridge can no longer run on SQLite.
+
+**TLDR**: if you're NOT using an [external Postgres server](docs/configuring-playbook-external-postgres.md) and have NOT forcefully kept the bridge on SQLite during [The big move to all-on-Postgres (potentially dangerous)](#the-big-move-to-all-on-postgres-potentially-dangerous), you will be automatically upgraded without manual intervention. All you need to do is send a `login` message to the Facebook bridge bot again.
+
+Whether this change requires your intervention depends mostly on:
+- whether you're using an [external Postgres server](docs/configuring-playbook-external-postgres.md). If yes, then [you need to do something](#upgrade-path-for-people-running-an-external-postgres-server).
+- or whether you've force-changed the bridge's database engine to SQLite (`matrix_mautrix_facebook_database_engine: 'sqlite'` in your `vars.yml`) some time in the past (likely during [The big move to all-on-Postgres (potentially dangerous)](#the-big-move-to-all-on-postgres-potentially-dangerous)).
+
+As already mentioned above, you most likely don't need to do anything. If you rerun the playbook and don't get an error, you've been automatically upgraded. Just send a `login` message to the Facebook bridge bot again. Otherwise, read below for a solution.
+
+### Upgrade path for people NOT running an external Postgres server (default for the playbook)
+
+If you're **not running an external Postgres server**, then this bridge either already works Postgres for you, or you've intentionally kept it back on SQLite with custom configuration (`matrix_mautrix_facebook_database_engine: 'sqlite'` in your `vars.yml`) .
+
+Simply remove that custom configuration from your `vars.yml` file (if it's there) and re-run the playbook. It should upgrade you automatically.
+You'll need to send a `login` message to the Facebook bridge bot again.
+
+Alternatively, [you can stay on SQLite for a little longer](#staying-on-sqlite-for-a-little-longer-temporary-solution).
+
+### Upgrade path for people running an external Postgres server
+
+For people using the internal Postgres server (the default for the playbook):
+- we automatically create an additional `matrix_mautrix_facebook` Postgres database and credentials to access it
+- we automatically adjust the bridge's `matrix_mautrix_facebook_database_*` variables to point the bridge to that Postgres database
+- we use [pgloader](https://pgloader.io/) to automatically import the existing SQLite data for the bridge into the `matrix_mautrix_facebook` Postgres database
+
+If you are using an [external Postgres server](docs/configuring-playbook-external-postgres.md), unfortunately we currently can't do any of that for you.
+
+You have 3 ways to proceed:
+
+- contribute to the playbook to make this possible (difficult)
+- or, do the above "steps" manually:
+  - stop the bridge (`systemctl stop matrix-mautrix-facebook`)
+  - create a new `matrix_mautrix_facebook` Postgres database for it
+  - run pgloader manually (we run it with default settings for this bridge)
+  - adjust the `matrix_mautrix_facebook_database_*` database variables (credentials, etc.)
+  - switch the bridge to use Postgres (`matrix_mautrix_facebook_database_engine: 'postgres'`)
+  - re-run the playbook (`--tags=setup-all,start`) and ensure the bridge works (`systemctl status matrix-mautrix-facebook` and `journalctl -fu matrix-mautrix-facebook`)
+  - send a `login` message to the Facebook bridge bot again
+- or, [stay on SQLite for a little longer (temporary solution)](#staying-on-sqlite-for-a-little-longer-temporary-solution)
+
+### Staying on SQLite for a little longer (temporary solution)
+
+To keep using this bridge with SQLite for a little longer (**not recommended**), use the following configuration in your `vars.yml` file:
+
+```yaml
+# Force-change the database engine to SQLite.
+matrix_mautrix_facebook_database_engine: 'sqlite'
+
+# Force-downgrade to the last bridge version which supported SQLite.
+matrix_mautrix_facebook_docker_image: "{{ matrix_mautrix_facebook_docker_image_name_prefix }}tulir/mautrix-facebook:da1b4ec596e334325a1589e70829dea46e73064b"
+```
+
+If you do this, keep in mind that **you can't run this forever**. This SQLite-supporting bridge version is not getting any updates and will break sooner or later. The playbook will also drop support for SQLite at some point in the future.
+
+
 # 2021-01-17
 
 ## matrix-corporal goes 2.0

group_vars/matrix_servers

@@ -208,7 +208,8 @@ matrix_mautrix_facebook_login_shared_secret: "{{ matrix_synapse_ext_password_pro
 
 matrix_mautrix_facebook_bridge_presence: "{{ matrix_synapse_use_presence if matrix_synapse_enabled else true }}"
 
-# Postgres is the default, except if not using `matrix_postgres` (internal postgres)
+# We'd like to force-set people with external Postgres to SQLite, so the bridge role can complain
+# and point them to a migration path.
 matrix_mautrix_facebook_database_engine: "{{ 'postgres' if matrix_postgres_enabled else 'sqlite' }}"
 matrix_mautrix_facebook_database_password: "{{ matrix_synapse_macaroon_secret_key | password_hash('sha512', 'mau.fb.db') | to_uuid }}"
 

roles/matrix-bridge-mautrix-facebook/defaults/main.yml

@@ -7,7 +7,7 @@ matrix_mautrix_facebook_container_image_self_build: false
 matrix_mautrix_facebook_container_image_self_build_repo: "https://github.com/tulir/mautrix-facebook.git"
 
 # See: https://mau.dev/tulir/mautrix-facebook/container_registry
-matrix_mautrix_facebook_docker_image: "{{ matrix_mautrix_facebook_docker_image_name_prefix }}tulir/mautrix-facebook:da1b4ec596e334325a1589e70829dea46e73064b"
+matrix_mautrix_facebook_docker_image: "{{ matrix_mautrix_facebook_docker_image_name_prefix }}tulir/mautrix-facebook:latest"
 matrix_mautrix_facebook_docker_image_name_prefix: "{{ 'localhost/' if matrix_mautrix_facebook_container_image_self_build else 'dock.mau.dev/' }}"
 matrix_mautrix_facebook_docker_image_force_pull: "{{ matrix_mautrix_facebook_docker_image.endswith(':latest') }}"
 
@@ -35,12 +35,15 @@ matrix_mautrix_facebook_homeserver_token: ''
 
 # Database-related configuration fields.
 #
-# To use SQLite, stick to these defaults.
+# To use SQLite:
+# - change the engine (`matrix_mautrix_facebook_database_engine: 'sqlite'`)
+# - change to the last bridge version that supported SQLite:
+#   `matrix_mautrix_facebook_docker_image: "{{ matrix_mautrix_facebook_docker_image_name_prefix }}tulir/mautrix-facebook:da1b4ec596e334325a1589e70829dea46e73064b"`
+# - plan your migration to Postgres, as this bridge does not support SQLite anymore (and neither will the playbook in the future).
 #
 # To use Postgres:
-# - change the engine (`matrix_mautrix_facebook_database_engine: 'postgres'`)
 # - adjust your database credentials via the `matrix_mautrix_facebook_postgres_*` variables
-matrix_mautrix_facebook_database_engine: 'sqlite'
+matrix_mautrix_facebook_database_engine: 'postgres'
 
 matrix_mautrix_facebook_sqlite_database_path_local: "{{ matrix_mautrix_facebook_data_path }}/mautrix-facebook.db"
 matrix_mautrix_facebook_sqlite_database_path_in_container: "/data/mautrix-facebook.db"

roles/matrix-bridge-mautrix-facebook/tasks/validate_config.yml

@@ -8,3 +8,24 @@
   with_items:
     - "matrix_mautrix_facebook_appservice_token"
     - "matrix_mautrix_facebook_homeserver_token"
+
+- block:
+    - name: Fail if on SQLite, unless on the last version supporting SQLite
+      fail:
+        msg: >-
+          You're trying to use the mautrix-facebook bridge with an SQLite database.
+          Going forward, this bridge only supports Postgres.
+          To learn more about this, see our changelog: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#breaking-change-the-mautrix-facebook-bridge-now-requires-a-postgres-database
+      when: "not matrix_mautrix_facebook_docker_image.endswith(':da1b4ec596e334325a1589e70829dea46e73064b')"
+
+    - name: Inject warning if still on SQLite
+      set_fact:
+        matrix_playbook_runtime_results: |
+          {{
+            matrix_playbook_runtime_results|default([])
+            +
+            [
+              "NOTE: Your mautrix-facebook bridge setup is still on SQLite. Your bridge is not getting any updates and will likely stop working at some point. To learn more about this, see our changelog: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#breaking-change-the-mautrix-facebook-bridge-now-requires-a-postgres-database"
+            ]
+          }}
+  when: "matrix_mautrix_facebook_database_engine == 'sqlite'"

roles/matrix-bridge-mautrix-facebook/templates/config.yaml.j2

@@ -8,6 +8,10 @@ homeserver:
     # Whether or not to verify the SSL certificate of the homeserver.
     # Only applies if address starts with https://
     verify_ssl: true
+    # Whether or not the homeserver supports asmux-specific endpoints,
+    # such as /_matrix/client/unstable/net.maunium.asmux/dms for atomically
+    # updating m.direct.
+    asmux: false
 
 # Application service host/registration related details
 # Changing these values requires regeneration of the registration.
@@ -22,11 +26,7 @@ appservice:
     # Usually 1 is enough, but on high-traffic bridges you might need to increase this to avoid 413s
     max_body_size: 1
 
-    # The full URI to the database. SQLite and Postgres are fully supported.
+    # The full URI to the database. Only Postgres is currently supported.
-    # Other DBMSes supported by SQLAlchemy may or may not work.
-    # Format examples:
-    #   SQLite:   sqlite:///filename.db
-    #   Postgres: postgres://username:password@hostname/dbname
     database: {{ matrix_mautrix_facebook_appservice_database|to_json }}
 
     # Public part of web server for out-of-Matrix interaction with the bridge.
@@ -38,6 +38,10 @@ appservice:
         # The base URL where the public-facing endpoints are available. The prefix is not added
         # implicitly.
         external: https://example.com/public
+        # Shared secret for integration managers such as mautrix-manager.
+        # If set to "generate", a random string will be generated on the next startup.
+        # If null, integration manager access to the API will not be possible.
+        shared_secret: generate
 
     # The unique ID of this appservice.
     id: facebook
@@ -46,12 +50,17 @@ appservice:
     # Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty
     # to leave display name/avatar as-is.
     bot_displayname: Facebook bridge bot
-    bot_avatar: mxc://maunium.net/ddtNPZSKMNqaUzqrHuWvUADv
+    bot_avatar: mxc://maunium.net/ygtkteZsXnGJLJHRchUwYWak
 
     # Authentication tokens for AS <-> HS communication.
     as_token: "{{ matrix_mautrix_facebook_appservice_token }}"
     hs_token: "{{ matrix_mautrix_facebook_homeserver_token }}"
 
+# Prometheus telemetry config. Requires prometheus-client to be installed.
+metrics:
+    enabled: false
+    listen_port: 8000
+
 # Bridge config
 bridge:
     # Localpart template of MXIDs for Facebook users.
@@ -76,6 +85,7 @@ bridge:
     # "own_nickname" (user-specific!)
     displayname_preference:
     - name
+    - first_name
 
     # The prefix for commands. Only required in non-management rooms.
     command_prefix: "!fb"
@@ -120,6 +130,18 @@ bridge:
         # Default to encryption, force-enable encryption in all portals the bridge creates
         # This will cause the bridge bot to be in private chats for the encryption to work properly.
         default: false
+        # Options for automatic key sharing.
+        key_sharing:
+            # Enable key sharing? If enabled, key requests for rooms where users are in will be fulfilled.
+            # You must use a client that supports requesting keys from other users to use this feature.
+            allow: false
+            # Require the requesting device to have a valid cross-signing signature?
+            # This doesn't require that the bridge has verified the device, only that the user has verified it.
+            # Not yet implemented.
+            require_cross_signing: false
+            # Require devices to be verified by the bridge?
+            # Verification by the bridge is not yet implemented.
+            require_verification: true
     # Whether or not the bridge should send a read receipt from the bridge bot when a message has
     # been sent to Facebook.
     delivery_receipts: false
@@ -161,6 +183,10 @@ bridge:
     # Whether or not the bridge should try to "refresh" the connection if a normal reconnection
     # attempt fails.
     refresh_on_reconnection_fail: false
+    # Set this to true to tell the bridge to re-send m.bridge events to all rooms on the next run.
+    # This field will automatically be changed back to false after it,
+    # except if the config file is not writable.
+    resend_bridge_info: false
 
     # Permissions for using the bridge.
     # Permitted values:
@@ -192,9 +218,7 @@ logging:
     loggers:
         mau:
             level: DEBUG
-        fbchat:
+        paho:
-            level: DEBUG
-        hbmqtt:
             level: INFO
         aiohttp:
             level: INFO