Previously `just install-service buscarron` would not fully work,
because:
- the systemd services were indeed tagged with `buscarron`
- however, the actual installation tasks are not
`install-buscarron`/`setup-buscarron`, but rather
`install-bot-buscarron`/`setup-bot-buscarron`
Services are now tagged with the `bot-` prefix to match the tags.
This allows people to try out the new Element X clients, which need to
run against the sliding-sync proxy (https://github.com/matrix-org/sliding-sync).
Supersedes https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2515
The code is based on the existing PR (#2515), but heavily reworked. Major changes:
- lots of internal refactoring and variable renaming
- fixed self-building to support non-amd64 architectures
- changed to talk to the homeserver locally, over the container network (not
publicly)
- no more matrix-nginx-proxy support due to complexity (see below)
- no more `matrix_server_fqn_sliding_sync_proxy` in favor of
`matrix_sliding_sync_hostname` and `matrix_sliding_sync_path_prefix`
- runs on `matrix.DOMAIN/sliding-sync` by default, so it can tried
easily without having to create new DNS records
The variable was necessary when multiple playbooks could have
potentially tried to manage a shared `devture-traefik.serivce` systemd service
and shared `/devture-traefik` directory.
Since adcc6d9723, we use our own `/matrix/traefik`
(`matrix-traefik.service`) installation and no conflicts can arise.
It's safe to always enable the role, just like we do with all the other roles.
The migration is automatic. Existing users should experience a bit of
downtime until the playbook runs to completion, but don't need to do
anything manually.
This change is provoked by https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2535
While my statements there ("Traefik is a shared component among
sibling/related playbooks and should retain its global
non-matrix-prefixed name and path") do make sense, there's another point
of view as well.
With the addition of docker-socket-proxy support in bf2b540807,
we potentially introduced another non-`matrix-`-prefixed systemd service
and global path (`/devture-container-socket-proxy`). It would have
started to become messy.
Traefik always being called `devture-traefik.service` and using the `/devture-traefik` path
has the following downsides:
- different playbooks may write to the same place, unintentionally,
before you disable the Traefik role in some of them.
If each playbook manages its own installation, no such conflicts
arise and you'll learn about the conflict when one of them starts its
Traefik service and fails because the ports are already in use
- the data is scattered - backing up `/matrix` is no longer enough when
some stuff lives in `/devture-traefik` or `/devture-container-socket-proxy` as well;
similarly, deleting `/matrix` is no longer enough to clean up
For this reason, the Traefik instance managed by this playbook
will now be called `matrix-traefik` and live under `/matrix/traefik`.
This also makes it obvious to users running multiple playbooks, which
Traefik instance (powered by which playbook) is the active one.
Previously, you'd look at `devture-traefik.service` and wonder which
role was managing it.
We don't need these 2 roughly-the-same settings related to the
traefik-certs-dumper role.
For Traefik, it makes sense, because it's a component used by the
various related playbooks and they could step onto each other's toes
if the role is enabled, but Traefik is disabled (in that case, uninstall
tasks will run).
As for Traefik certs dumper, the other related playbooks don't have it,
so there's no conflict. Even if they used it, each one would use its own
instance (different `devture_traefik_certs_dumper_identifier`), so there
wouldn't be a conflict and uninstall tasks can run without any danger.
This allows people wishing to change or unset the resolver,
to have a single variable which they can toggle.
Unsetting the resolver is useful for using your own certificates
(not coming from a certificate resolver).
The newly extracted role also has native Traefik support,
so we no longer need to rely on `matrix-nginx-proxy` for
reverse-proxying to Ntfy.
The new role uses port `80` inside the container (not `8080`, like
before), because that's the default assumption of the officially
published container image. Using a custom port (like `8080`), means the
default healthcheck command (which hardcodes port `80`) doesn't work.
Instead of fiddling to override the healthcheck command, we've decided
to stick to the default port instead. This only affects the
inside-the-container port, not any external ports.
The new role also supports adding the network ranges of the container's
multiple additional networks as "exempt hosts". Previously, only one
network's address range was added to "exempt hosts".
Previously, it had to go through matrix-nginx-proxy.
It's exposed to Traefik directly via container labels now
Serving at a path other than `/` doesn't work well yet.
We'd like to auto-enable traefik-certs-dumper for these setups.
`devture_traefik_certs_dumper_ssl_dir_path` will be empty though,
so the role's validation will point people in the right direction.
This gets us started on adding a Traefik role and hooking Traefik:
- directly to services which support Traefik - we only have a few of
these right now, but the list will grow
- to matrix-nginx-proxy for most services that integrate with
matrix-nginx-proxy right now
Traefik usage should be disabled by default for now and nothing should
change for people just yet.
Enabling these experiments requires additional configuration like this:
```yaml
devture_traefik_ssl_email_address: '.....'
matrix_playbook_traefik_role_enabled: true
matrix_playbook_traefik_labels_enabled: true
matrix_ssl_retrieval_method: none
matrix_nginx_proxy_https_enabled: false
matrix_nginx_proxy_container_http_host_bind_port: ''
matrix_nginx_proxy_container_federation_host_bind_port: ''
matrix_nginx_proxy_trust_forwarded_proto: true
matrix_nginx_proxy_x_forwarded_for: '$proxy_add_x_forwarded_for'
matrix_coturn_enabled: false
```
What currently works is:
reverse-proxying for all nginx-proxy based services **except** for the Matrix homeserver
(both Client-Server an Federation traffic for the homeserver don't work yet)
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2427
This just enables the endpoint, which is somewhat helpful, but not
really enough to scrape them. Ideally, we'd be injecting these targets
into the Prometheus scrape config too.
For now, registering targets with Prometheus is very manual
(`matrix_prometheus_scraper_postgres_enabled`, `matrix_prometheus_scraper_hookshot_enabled`, ..).
This should be redone - e.g. a new `matrix_prometheus_scrape_config_jobs_auto` variable,
which is dynamically built in `group_vars/matrix_servers`.
selectattr() returns a generator object, an iterator. This leads to an exception later, lists can't concated to iterators, only to other lists. So '| list' converts the iterator to a list and the script runs happily.
This extends the collection with support for seamless authentication at the Jitsi server using Matrix OpenID.
1. New role for installing the [Matrix User Verification Service](https://github.com/matrix-org/matrix-user-verification-service)
2. Changes to Jitsi role: Installing Jitsi Prosody Mods and configuring Jitsi Auth
3. Changes to Jitsi and nginx-proxy roles: Serving .well-known/element/jitsi from jitsi.DOMAIN
4. We updated the Jitsi documentation on authentication and added documentation for the user verification service.
* add prometheus-nginxlog-exporter role
* Rename matrix_prometheus_nginxlog_exporter_container_url to matrix_prometheus_nginxlog_exporter_container_hostname
* avoid referencing variables from other roles, handover info using group_vars/matrix_servers
* fix: stop service when uninstalling
fix: typo
move available arch's into a var
fix: text
* fix: prometheus enabled condition
Co-authored-by: ikkemaniac <ikkemaniac@localhost>
This role is usable on its own and it's not tied to Matrix, so
extracting it out into an independent role that we install via
ansible-galaxy makes sense.
This also fixes the confusion from the other day, where
`matrix_postgres_*` had to be renamed to `devture_postgres_*`
(unless it was about `matrix_postgres_backup_*`).
We now can safely say that ALL `matrix_postgres_*` variables need to be
renamed.
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2305
There was no need to add `synapse` to the list manually
and then add all other additional databases.
When the `synapse` database was the main database, this made sense.
Since a long time ago already, ALL databases are "additional" databases,
so the `synapse` database is part of that list.
We could additional add the main (`matrix`) database to this list,
but there's probably no point in backing that one up.
* added dendrite captcha options
* added hcaptcha doc
* proper url
* Apply suggestions from code review
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* Update main.yml
* renamed captcha vars to new naming scheme
* change vars to new format
* Rename back some incorrect renamed variables
These variables are either not just part of the `client_api` subsection,
or are not even part of that section at all. They shouldn't have been
renamed in baaef2ed616e2645550d9
* Fix up naming inconsistencies
Some of these variables had been renamed in one place,
but not in other places, so it couldn't have worked that way.
* Add validation/deprecation for renamed Dendrite variables
Related to 4097898f885cf4c73, baaef2ed616e2645550, 68f4418092fa8ad
and a0b4a0ae6b2f1f18
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
Ansible recently started showing warnings about `crypt` being
deprecated. If one installs `passlib`, the `password_hash` values that
are generated would be different by default. With this patch, we ensure
consistency regardless of which one is used.
After this patch, password hashes (and UUIDs derived from them) will
change once, but they should be stable after that.
These hashes changing is not a problem, because the playbook
changes all references to the new values. Changes are only a problem if
they're done partially and with different tools.
For example:
- `--tags=setup-COMPONENT` with `passlib`
- `--tags=setup-postgres` with `crypt` (no `passlib`)
If so, the Postgres database password's value will differ for the
configuration generated for `COMPONENT`.
The `rounds=` value is arbitrary. It doesn't matter what it is,
as long as it's different than the default for `crypt` (5000)
and the default for `passlib` for `sha512` (656000).
Source (https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_filters.html):
> To ensure idempotency, specify rounds to be neither crypt’s nor passlib’s default, which is 5000 for crypt and a variable value (535000 for sha256, 656000 for sha512) for passlib
We need this to control whether `('matrix-' + matrix_homeserver_implementation + '.service')`
would get injected into `devture_systemd_service_manager_services_list_auto`
We no longer ask users to create Matrix user accounts for these bots:
- Postmoogle
- Honoroit
- Reminder Bot
Other bots and services (matrix-registration-bot, maubot, mjolnir,
Dimension, etc.) require an Access Token to run (not a password),
so this new role doesn't help for them.
It does help for the above bots though, and for defining your own
"initial user accounts" in the `matrix_user_creator_users_additional`
variable.
On Debian 10 (buster) at least, while the Synapse systemd service unit
was referring to Goofys as "matrix-goofys" without a ".service" suffix,
systemd was ignoring the goofys dependency, starting Synapse before
Goofys. All other dependant units which work are using the ".service"
suffix.
This generally leads to the mount path goofys using having been
populated by Synapse before Goofys starts, causing it to fail due to the
mount target not being empty.
The fix seems to be to ensure that the Synapse service unit refers to
Goofys as "matrix-goofys.service".
This change causes the following two lines in
"/etc/systemd/system/matrix-synapse.service":
Requires=matrix-goofys
After=matrix-goofys
To be changed to:
Requires=matrix-goofys.service
After=matrix-goofys.service
* Make registration proxy independent of other roles, document
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Fix yml issues
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Remove undefined variable (as service HAS to be exposed
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Add registration endpint
Defines the registration endpoint that should be intercepted/forwarded to the proxy
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Add image name
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
Using `matrix_synapse_*` variables within the `matrix-grafana` role
is not a good practice.
We now have a `matrix_grafana_default_home_dashboard_path` variable
with a good universal default value and we override it via
`group_vars/matrix_servers` based on enabled components, etc.
This is a better fix for https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2133
As stream writer workers are also powered by the `generic_worker`
Synapse app, this necessitated that we provide means for distinguishing
between them and regular `generic_workers`.
I've also taken the time to optimize nginx configuration generation
(more Jinja2 macro usage, less duplication).
Worker names have also changed.
Workers are now named sequentially like this:
- `matrix-synapse-worker-0-generic`
- `matrix-synapse-worker-1-stream-writer-typing`
- `matrix-synapse-worker-2-pusher`
instead of `matrix-synapse-worker_generic_worker-18111` (indexed with a
port number).
People who modify `matrix_synapse_workers_enabled_list` directly will
need to adjust their configuration.
* Add construct for cactus comments role
* Adjust config files
* Add docker self build to defaults
* Adjust tasks
* Fix smaller syntax errors
* Fix env argument
* Add tmp path to allow container writing there
Background why I did this: https://docs.gunicorn.org/en/stable/settings.html#worker-tmp-dir
* Change port back to 5000 as not configurable in container
* Try to add appservice config file for synapse to use
* Inject appservice file
* Correct copied variable name
* Comment out unused app service file injection
would need mounting the appservice file to the synapse container i guess
* Move role before synapse to be able to inject during runtime
* Remove unused parts
* Change default user id to mirror official docs
* Add docs
* Update roles/matrix-cactus-comments/tasks/setup_install.yml
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* Update roles/matrix-cactus-comments/templates/cactus_appservice.yaml.j2
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* Generate secrets if necessary, adjust docs
* Rename cactusbot userid
* Shorten salt strings
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* Use tmpfs instead of persistent mount
* Remove proxy option as it is nonsense
* Add download and serving of cc-client files
* Add documentation on client
* Clarify docs a bit
* Add nginx proxy to required services
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Use container address
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Correct comment of user id
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Use releases or local distributed client
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Move homeserver url to defaults
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Correct truth value
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Add documentation of variables
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* Tabs vs. spaces
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* Make nginx root configurable
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Complete ake nginx root configurable
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Fix file permission
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
* Fix lint errors
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
Signed-off-by: Julian-Samuel Gebühr <julian-samuel@gebuehr.net>
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* appservice: add and use matrix_homeserver_* vars
* appservice: use the new vars
* Apply suggestions from code review
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* Fix bug that prevented user with external nginx from launch
The backslash was missing and prevented users from starting the bot
* Add necessary config for ext nginx to docs
* Add automatic config for ext nginx, adjust docs
* Remove unneeded and possibly puzzeling documentation
Adds support for: https://src.miscworks.net/fair/matrix-appservice-kakaotalk
This is pretty similar to
https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1977
which just appeared, but has mostly been done independently.
I've taken some inspiration and did some fixups based on that PR.
Thanks to https://github.com/hnarjis for taking the time to contribute!
Notable differences between this branch compared to that PR:
- better naming and documentation around the "configuration" variables
- no unnecessary (5 sec.) intentional delay when starting `matrix-appservice-kakaotalk-node.service`
- stores configuration in `config/`, not in `data/`
- passes configuration as read-only and starts the bridge with (`--no-update`) to ensure no changes are made to it
- starts containers more securely - with `matrix:matrix` user:group (not `root`) and
reduced capabilities (`--cap-drop=ALL`)
- uses `tcp` for communication between the "node" and the appservice (simpler than sharing unix sockets)
- `registration.yaml` which is closer to the one generated by `matrix-appservice-kakaotalk` (no `de.sorunome.msc2409.push_ephemeral` stuff, etc.)
- `registration.yaml` which is more customizable (customizable bot username and prefix for puppets - see `matrix_appservice_kakaotalk_appservice_bot_username` and `matrix_appservice_kakaotalk_user_prefix`)
- less fragile and more extensible bridge permissions configuration via `matrix_appservice_kakaotalk_bridge_permissions`. Doing `{% if matrix_admin %}` in the bridge configuration sometimes causes syntax problems (I hit some myself) and is not ideal. Other bridges should be redone as well.
- configurable command prefix for the bridge, instead of hardcoding `!kt` (see `matrix_appservice_kakaotalk_command_prefix`)
- logging that is more consistent with the rest of the playbook (console / journald only, no logging to files), as well as configurable log level (via `matrix_appservice_kakaotalk_logging_level`)
- somewhat more detailed documentation (`docs/configuring-playbook-bridge-appservice-kakaotalk.md`)
- removed some dead code (data relocation tasks from `tasks/setup_install.yml`, as well as likely unnecessary SQLite -> Postgres migration)
This commit adds a 'matrix-ntfy' role that runs Ntfy server in Docker with
simple configuration, and plumbing to add the role to the playbook.
TODO: documentation, self-check, database persistence.
`localhost` may resolve to `::1` on some IPv6-enabled systems, which will
not work, because we only potentially expose container ports on
`127.0.0.1` when nginx is disabled (`matrix_nginx_proxy_enabled: false`),
not on `::1`.
Fixes https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1914
* Add matrix-registration-bot
This adds an install and uninstall task plus helpers. The bot is disabled by default.
This commit does not include documentation, yet. In short, the bot can be enabled by adding
matrix_bot_matrix_registration_bot_enabled: true
matrix_bot_matrix_registration_bot_matrix_user_password: "verysecret"
matrix_bot_matrix_registration_bot_matrix_admin_token: "supersecret"
to the host_vars
* Change bot username to bot.matrix-registration-bot following convention
* Address smaller remarks, fix local docker build
* Switch to an env file
* Add environment variables extension for additional config
* Add documentation for the matrix-registration-bot
* Add screenshot on how to obtain admin access token
* Use bot as admin to only have one access token (bot and admin api)
* Use cleaner setting of matrix_synapse_registration_requires_token
* Use config file for cleaner more secure usage
* Delete unneeded env
* Rename vars to make usage clear
* Fix typos/wording and add notice about logging out
* Convert configuration to use |to_json
* Reorder role includes
Nothing should be after `matrix-common-after`.
`matrix-bot-matrix-registration-bot` can probably be anywhere, but it makes sense to put it next to the other `matrix-bot-*` roles.
* Minor group_vars/matrix_servers touchups
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
The `halfshot/matrix-hookshot` container images published to Docker Hub
(as of 2022-04-05, at least) are only available for `amd64`, not for
`arm64`. Self-building on arm64 is necessary.
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/1728
It should be noted that a `roiarthurb/matrix-hookshot` container image is available,
which is available for the arm64 platform, but that's non-official and doesn't
contain an amd64 build, so it's of limited use.
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1682
Previously, when matrix-postgres was disabled, we were setting
`matrix_mautrix_twitter_database_engine` to an invalid empty value.
Now, we always hardcode `matrix_mautrix_twitter_database_engine: postgres`,
but set/unset the database hostname and password values instead.
1.0.2 is the first container image tag that is available as a multi-arch image
with support for linux/amd64, linux/arm64/v8 (arm64) and linux/arm/v7 (arm32),
so self-building is no longer necessary on all these platforms.
4.95-r0-1 is the first container image tag that is available as a multi-arch image
with support for linux/amd64, linux/arm64/v8 (arm64) and linux/arm/v7 (arm32),
so self-building is no longer necessary on all these platforms.
2.2.3 is the first container image tag that is available as a multi-arch image
with support for linux/amd64, linux/arm64/v8 (arm64) and linux/arm/v7 (arm32),
so self-building is no longer necessary on all these platforms.
The goal is to have a single variable which tells us which homeserver
software is in use. Much simpler than having if/elif/elif checks for
variables like (`matrix_synapse_enabled` and `matrix_dendrite_enabled`, etc.)
everywhere.
This change forces ansible to decrypt the variable with ansible-vault if encrypted, to avoid the error '{"msg": "Unexpected templating type error occurred on ({{ matrix_synapse_macaroon_secret_key | password_hash('sha512') }}): secret must be unicode or bytes, not ansible.parsing.yaml.objects.AnsibleVaultEncryptedUnicode"}'
Every other variable in the playbook was found to have no problems with encryption.
The change has no negative impact on non-encrypted matrix_synapse_macaroon_secret_key.
