Commit Graph

1278 Commits

Author SHA1 Message Date
Slavi Pantaleev
2a502db239 Add (SQLite + Postgres) support and automatic migration to matrix-dimension 2020-12-14 21:01:47 +02:00
Slavi Pantaleev
0790a7b2a8 Add support for matrix_dimension_systemd_{required,wanted}_services_list
We were referencing them from `group_vars/matrix_servers` since
recently, but there were no such variables and they weren't being put to
use.
2020-12-14 20:31:07 +02:00
Slavi Pantaleev
374f43735a Separate matrix-dimension install/uninstall tasks 2020-12-14 20:05:31 +02:00
Slavi Pantaleev
8d74593878 Prepare matrix-registation for (SQLite + Postgres) support
Auto-migration and everything seems to work. It's just that
matrix-registration cannot load the Python modules required
for talking to a Postgres database.

Tracked here: https://github.com/ZerataX/matrix-registration/issues/44

Until this gets fixed, we'll continue default to 'sqlite'.
2020-12-14 18:58:37 +02:00
Slavi Pantaleev
516ccb2b2b Separate matrix-registration install/uninstall tasks 2020-12-14 18:12:14 +02:00
13d8a9b39c
hint supported automatic migration nedb->postgres 2020-12-14 16:33:40 +01:00
Slavi Pantaleev
af3ea67bba Add (SQLite + Postgres) support and automatic migration to matrix-ma1sd 2020-12-14 17:16:25 +02:00
Slavi Pantaleev
0ca48f3532 Separate matrix-ma1sd install/uninstall tasks 2020-12-14 16:57:51 +02:00
Slavi Pantaleev
7248eb3c11 Fix syntax error in roles/matrix-bridge-appservice-irc/defaults/main.yml 2020-12-14 16:25:44 +02:00
Slavi Pantaleev
cba973d6b5 Enable automatic (SQLite -> Postgres) migration for matrix-appservice-discord 2020-12-14 16:25:22 +02:00
Slavi Pantaleev
13f84e2ad5 Enable automatic (SQLite -> Postgres) migration for matrix-mautrix-whatsapp 2020-12-14 16:21:01 +02:00
Slavi Pantaleev
86a8091768 Enable automatic (SQLite -> Postgres) migration for matrix-mautrix-telegram 2020-12-14 16:19:54 +02:00
Slavi Pantaleev
3ba8520266 Enable automatic (SQLite -> Postgres) migration for matrix-mautrix-hangouts 2020-12-14 16:18:38 +02:00
Slavi Pantaleev
bbc08722c5 Enable automatic (SQLite -> Postgres) migration for matrix-mautrix-facebook 2020-12-14 16:14:23 +02:00
Slavi Pantaleev
c1431b28f0 Make use of matrix_postgres_db_migration_request.caller 2020-12-14 16:13:57 +02:00
Slavi Pantaleev
ac37091d01 Enable automatic (SQLite -> Postgres) migration for matrix-reminder-bot 2020-12-14 16:03:40 +02:00
Slavi Pantaleev
dc7850e83c Fix wording and variable names a bit 2020-12-14 16:03:40 +02:00
Slavi Pantaleev
bc376c2fb2 Add database migration utility to matrix-postgres role 2020-12-14 16:03:40 +02:00
54da61f81b
add postgres support mx-appservice-[slack|irc] with fallback to nedb in role and migration notice 2020-12-14 14:08:35 +01:00
Slavi Pantaleev
e3a0c9adda Add (Postgres + SQLite) support to matrix-reminder-bot
This has been tested and appears to work.
2020-12-14 15:02:11 +02:00
Slavi Pantaleev
dde1c9f899 Fix indentation causing YAML syntax error 2020-12-14 14:53:35 +02:00
Slavi Pantaleev
aa828ff9f6 Separate matrix-reminder-bot install/uninstall tasks 2020-12-14 14:50:04 +02:00
Slavi Pantaleev
b9a04a7f95 Rename some remaining matrix_*_postgres_* vars back to matrix_*_database_*
Looks like there are some that I missed in 087dbe4ddc
2020-12-14 14:42:18 +02:00
5d70bc1376
add postgres support for mx-puppet-* with fallback to sqlite in role and migration notice 2020-12-14 13:22:58 +01:00
Slavi Pantaleev
087dbe4ddc Rename matrix_*_postgres_* back to matrix_*_database_*
I was thinking that it makes sense to be more specific,
and using `_postgres_` also separated these variables
from the `_database_` variables that ended up in bridge configuration.

However, @jdreichmann makes a good point
(https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/740#discussion_r542281102)
that we don't need to be so specific and can allow for other engines (like MySQL) to use these variables.
2020-12-14 13:02:47 +02:00
Slavi Pantaleev
ce21ea3640 Add (Postgres + SQLite) support to matrix-mautrix-hangouts bridge
I don't use this bridge, so this is completely untested.
2020-12-14 12:34:59 +02:00
Slavi Pantaleev
43d6ff2af8 Fix sqlite usage for mautrix-facebook/mautrix-telegram
Regression since 2d99ade72f and 9bf8ce878e, respectively.

When SQLite is to be used, these bridges expect an `sqlite://`
connection string, and not a plain file name (path), like Appservice
Discord and mautrix-whatsapp do.
2020-12-14 12:30:10 +02:00
Slavi Pantaleev
6c77eae969 Add (Postgres + SQLite) support to matrix-mautrix-whatsapp bridge
I don't use this bridge, so this is completely untested.
2020-12-14 12:24:37 +02:00
Slavi Pantaleev
9bf8ce878e Add (Postgres + SQLite) support to matrix-mautrix-telegram bridge
I don't use this bridge, so this is completely untested.
2020-12-14 12:06:28 +02:00
Slavi Pantaleev
a3406a182b Move some things around 2020-12-14 12:04:47 +02:00
Slavi Pantaleev
2d99ade72f Add (Postgres + SQLite) support to matrix-mautrix-facebook bridge 2020-12-14 11:50:42 +02:00
Slavi Pantaleev
5dba0c038b Make --tags=import-generic-sqlite-db commands not pass a sensitive connection string around
Instead of passing the connection string, we can now pass a name of a
variable, which contains a connection string.

Both are supported for having extra flexibility.
2020-12-14 11:47:00 +02:00
Slavi Pantaleev
d91aa5a060 Do not introduce sub-variables exposing implementation details 2020-12-14 10:52:07 +02:00
Slavi Pantaleev
f1e85f7112 Don't mention Postgres roles, just say users 2020-12-14 10:04:37 +02:00
Slavi Pantaleev
4617984b9f Add (SQLite -> Postgres) migration instructions 2020-12-14 02:24:32 +02:00
Slavi Pantaleev
cb969c6ca2 Add --tags=import-generic-sqlite-db (pgloader import)
This can be used by various bridges, etc., to import an SQLite
(or some other supported) database into Postgres.
2020-12-14 02:23:29 +02:00
Slavi Pantaleev
c66c084027 Merge branch 'master' into postgres-per-default 2020-12-14 01:51:15 +02:00
Slavi Pantaleev
6e1dfb62f0 Rename some doc files and commands related to importing
Since we'll likely have generic SQLite database importing
via [pgloader](https://pgloader.io/) for migrating bridge
databases from SQLite to Postgres, we'd rather avoid
calling the "import Synapse SQLite database" command
as just `--tags=import-sqlite-db`.

Similarly, for the media store, we'd like to mention that it's
related to Synapse as well.

We'd like to be more explicit, so as to be less confusing,
especially in light of other homeserver implementations
coming in the future.
2020-12-14 01:51:00 +02:00
Slavi Pantaleev
b87b754372 Fail if appservice-discord wants Postgres, but has leftover SQLite data 2020-12-14 01:36:15 +02:00
Slavi Pantaleev
183d2a10db Ensure matrix-postgres.service is started before creating additional users/databases 2020-12-14 00:59:59 +02:00
Slavi Pantaleev
a374d309c8 Make appservice-discord support both SQLite and Postgres
People can toggle between them now. The playbook also defaults
to using SQLite if an external Postgres server is used.

Ideally, we'd be able to create databases/users in external Postgres
servers as well, but our initialization logic (and `docker run` command,
etc.) hardcode too many things right now.
2020-12-14 00:52:25 +02:00
Slavi Pantaleev
46a4034d3e Use "password" for additional Postgres databases, not "pass"
Being more explicit sounds better.
2020-12-14 00:43:03 +02:00
Slavi Pantaleev
3a037a5993 Ensure additional databases contain all the keys that we expect 2020-12-14 00:39:38 +02:00
Slavi Pantaleev
da4cb2f639 Do not use the postgresql_user/postgresql_db modules
While these modules are really nice and helpful, we can't use them
for at least 2 reasons:

- for us, Postgres runs in a container on a private Docker network
(`--network=matrix`) without usually being exposed to the host.
These modules execute on the host so they won't be able to reach it.

- these modules require `psycopg2`, so we need to install it before
using it. This might or might not be its own can of worms.
2020-12-14 00:31:38 +02:00
Slavi Pantaleev
bbc09d013b Do not execute additional databases creation code if not necessary
The tasks in `create_additional_databases.yml` will likely
ensure `matrix-postgres.service` is started, etc.

If no additional databases are defined, we'd rather not execute that
file and all these tasks that it may do in the future.
2020-12-13 23:46:05 +02:00
Slavi Pantaleev
c765ceb270 Prevent weird loop error
> Invalid data passed to 'loop', it requires a list, got this instead: matrix_postgres_additional_databases. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup.

Well, or working around it, as I've done in this commit (which seems
more sane than `wantlist=True` stuff).
2020-12-13 22:56:56 +02:00
Slavi Pantaleev
e2952f16f7 Determine matrix-postgres IP address without relying on jq
To avoid needing to have `jq` installed on the machine, we could:
- try to run jq in a Docker container using some small image providing
that
- better yet, avoid `jq` altogether
2020-12-13 22:45:48 +02:00
Slavi Pantaleev
f47e8a97e6 Make use of matrix_host_command_docker instead of hardcoding 2020-12-13 22:38:35 +02:00
Slavi Pantaleev
0641106370 Allow username of additional Postgres databases to be different
We'll most likely use one that matches the database name, but
it's better to have it configurable.
2020-12-13 22:37:04 +02:00
Slavi Pantaleev
527d5f57d5 Relocate Postgres additional database creation logic
Moving it above the "uninstalling" set of tasks is better.
Extracting it out to another file at the same time, for readability,
especially given that it will probably have to become more complex in
the future (potentially installing `jq`, etc.)
2020-12-13 22:37:04 +02:00