This is still very far from usable.
Various bridges and bots are still talking to
`matrix-nginx-proxy` instead of the new `matrix-homeserver-proxy` role.
These services need to be reworked. While reworking them,
various cleanups are being done as well as adding Traefik-labels to
those that need them.
It seems like connectivity is problematic, even though the networks
appear to be configured correctly:
> [ioredis] Unhandled error event: Error: connect ECONNREFUSED 172.22.0.2:6739
> at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)
For now, I disable pointing the queue host to Redis to avoid it.
It should be investigated.
People who enable Hookshot's new experimental encryption may encounter
this also.
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3042
Squashed based on the work done in https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/3042
commit 49932b8f3c17c4c4db7a884658c42f9a8b0550ca
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:21:31 2023 +0200
Fix syntax in matrix-bridge-hookshot/tasks/reset_encryption.yml
Also, this task always does work and side-effects, so it should always report changes
(`changed_when: true`).
commit 6bdf7a9dcb73385313a1f34d52e27ad0cf95fb3e
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:12:41 2023 +0200
Add Hookshot validation task to ensure queue settings are set when encryption is enabled
commit 8c531b7971b5dfd15ca541b5072b3eb8237cdcf9
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:10:17 2023 +0200
Add missing variables rewiring in group_vars/matrix_servers for Hookshot
commit 7d26dabc2fe692f5e1236c0e250f85996f3fd0c2
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:08:19 2023 +0200
Add defaults for matrix_hookshot_queue_host and matrix_hookshot_queue_port
commit 74f91138c92f1d1b69eb973803b882849e31a259
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:06:17 2023 +0200
Fix syntax for connecting to additional networks for Hookshot
commit ca7b41f3f2d0c2900b2805294476f70e82461304
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:05:28 2023 +0200
Fix indentation and remove unnecessary if-statements
commit ac4a918d58fc76e7332446a65609b56c5e4da00c
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:04:44 2023 +0200
Add missing --network for Hookshot
This seems to have been removed by accident.
commit 6a81fa208fca28951fc131cbf33b95ec78748a40
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:02:47 2023 +0200
Make automatic Redis enabling safer, when Hookshot encryption enabled
If we ever default encryption to enabled for Hookshot, we only wish to force-enable Redis if Hookshot is actually enabled.
commit 75a8e0f2a6cbf1562cb99c68ad1f20e4d47735ed
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sat Dec 16 09:01:10 2023 +0200
Fix typo
commit 98ad182eaccc7ab457ead4e03cc0d4f2a525a47d
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:37:40 2023 +0100
Add defaults for Hookshot's encryption
commit 29fa9fab151f513908d3e45882003da107a63c93
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:35:11 2023 +0100
Improve wording of Hookshot's encryption section
commit 4f835e0560012754d7ce0b56619a97c68a297992
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:28:52 2023 +0100
use safer mount options for the container's files
commit 8c93327e25c5e6af2442c676d5f264d4051c80e2
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:26:01 2023 +0100
fix filename
commit 03a7bb6e7798dd95f7894311e4dd34dfa09f70fa
Merge: e55d7694 06047763
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:23:44 2023 +0100
Merge branch 'HarHarLinks/hookshot-encryption' of https://github.com/real-joshua/matrix-docker-ansible-deploy into HarHarLinks/hookshot-encryption
commit 06047763bbd427dde117c6635ac7301198571158
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:15:54 2023 +0100
Update roles/custom/matrix-bridge-hookshot/templates/config.yml.j2
change the if statement to not require a variable with a length > 0 and add a filter to json for the redis host
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
commit e55d769465bd299081464e68d34851729d42d5ff
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:13:50 2023 +0100
clarify that Redis is required, standardadise on Hookshot with an upper-case first letter for consistency
commit 66706e4535704deba63e5aa2102f324f9b14dae3
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 22:08:20 2023 +0100
Update roles/custom/matrix-bridge-hookshot/templates/config.yml.j2
fix for a typo
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
commit f6aaeb9a16f84409d31188e696885afed89fbdcc
Merge: e5d34002 869dd33f
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 00:22:34 2023 +0100
Merge branch 'master' into HarHarLinks/hookshot-encryption
commit e5d34002fd6c39c3e18df04e20d0c8b25475db78
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Fri Dec 15 00:09:27 2023 +0100
Add Jinja loop to allow adding multiple networks
commit 69f947782d6e072c3edc4ee4c7d5ccf69eccf3fd
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Thu Dec 14 23:52:41 2023 +0100
split if statements for the message queue and experimental encryption support into seperate statements
commit 4c13be1c89ffb1b06475c7da546f7956e67b36a1
Author: Joshua Hoffmann <joshua.hoffmann@b1-systems.de>
Date: Thu Dec 14 23:31:19 2023 +0100
change variable name per spantaleev's suggestion (https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2979#discussion_r1379015551)
commit 9905309aa9448f91297f8c7618bf62682eee2af7
Author: HarHarLinks <kim.brose@rwth-aachen.de>
Date: Wed Nov 1 16:14:04 2023 +0100
amend docs
commit 94abf2d5bde63919c6b5597f3142eea5fed73815
Author: HarHarLinks <kim.brose@rwth-aachen.de>
Date: Wed Nov 1 16:05:22 2023 +0100
draft encryption support for hookshot
Currently v3.0.0 tested with no issues.
So remove matrix_user_verification_service_docker_image from groups_vars.
/.npm must be writable or an error will be reported.
* Inital work, copeid from mautrix-amp PR
* Some fixes leftover code copeid over from whatsapp
* Got it to run and register
* Fixed service issue with docker image
* I now realize I need 2 roles wsproxy and imessage
* Got someting working, still rough
* Closer to working but still not working
* reverting ports
* Update main.yml
* Add matrix-nginx-proxy config for mautrix-wsproxy
* Changed
* Add back file
* fix for error hopefully
* Changed the the way nginx was recieved
* basically did not add anything ugh
* Added some arguments
* just trying stuff now
* Ugh i messed up port number
* Changed docs
* Change dns config
* changed generic secret key
* Testing new nginx proxy
* test
* Fix linting errors
* Add mautrix syncproxy to wsproxy for Android SMS
* WIP
* Move wsproxy to custom
* Squashed commit of the following:
commit 943189a9aa
Merge: 4a229d68f5a09f30
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sun Nov 13 08:54:32 2022 +0200
Merge pull request #2259 from throny/patch-3
warn users about upgrading to pg15 when using borg
commit 4a229d6870
Merge: 9b326e08c68def08
Author: Slavi Pantaleev <slavi@devture.com>
Date: Sun Nov 13 08:53:13 2022 +0200
Merge pull request #2260 from etkecc/patch-117
Update ntfy 1.28.0 -> 1.29.0
commit f5a09f30b7
Author: throny <m.throne12@gmail.com>
Date: Sat Nov 12 23:48:57 2022 +0100
Update maintenance-postgres.md
commit b12cdbd99d
Author: throny <m.throne12@gmail.com>
Date: Sat Nov 12 23:40:46 2022 +0100
Update maintenance-postgres.md
commit c68def0809
Author: Aine <97398200+etkecc@users.noreply.github.com>
Date: Sat Nov 12 22:01:31 2022 +0000
Update ntfy 1.28.0 -> 1.29.0
commit adbc09f152
Author: throny <m.throne12@gmail.com>
Date: Sat Nov 12 11:20:43 2022 +0100
warn users about upgrading to pg15 when using borg
* Fix linting errors
* Cleanup after merge
* Correct outdated variable names
* Enable both Android and iMessage with wsproxy
* Restructure wsproxy service defs and nginx config
* Fix linter errors
* Apply suggestions from code review
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* Fix comments for documentation, volumes and ports
* Correct mount syntax
* Complete network and traefik support for wsproxy
* Remove wsproxy data_path
* Fix wsproxy service definitions
* Actually include syncproxy service
* Remove wsproxy PathPrefix, it needs a subdomain
There's no setting in the iMessage bridge that allows a path.
Also don't bind port by default, wsproxy has no TLS.
Syncproxy should never expose a port, it's only internal.
---------
Co-authored-by: hanthor <jreilly112@gmail.com>
Co-authored-by: Miguel Alatzar <miguel@natrx.io>
Co-authored-by: Shreyas Ajjarapu <github.tzarina@aleeas.com>
Co-authored-by: Slavi Pantaleev <slavi@devture.com>
* intial commit
* changed
* Reorderd
* merge old changes
* added changes to matrix_servers
* Remove duplicate discord
* Update main.yml
* added google message to configuring-playbook.md
* Changed docs to add new changes
* Changed bug?
* Removed problem j2 values
* Rename a service files
* change how password hash string
* Changed port number
* Change how the local part works
* Revert "Merge pull request #8 from shreyasajj/wsproxy"
This reverts commit bb1b8fc67ca39f63ca77e70077be99cb2b32c4de, reversing
changes made to cce6ba5f9d74f89172488afc8b1ef124031de8c1.
---------
Co-authored-by: Shreyas Ajjarapu <github.tzarina@aleeas.com>
Making the group match the Ansible task tags allows people to do `just install-service matrix-media-repo` and have that trigger both `--tags=matrix-media-repo` and also restart just that single group (`matrix-media-repo`).
Previously `just install-service buscarron` would not fully work,
because:
- the systemd services were indeed tagged with `buscarron`
- however, the actual installation tasks are not
`install-buscarron`/`setup-buscarron`, but rather
`install-bot-buscarron`/`setup-bot-buscarron`
Services are now tagged with the `bot-` prefix to match the tags.
This allows people to try out the new Element X clients, which need to
run against the sliding-sync proxy (https://github.com/matrix-org/sliding-sync).
Supersedes https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2515
The code is based on the existing PR (#2515), but heavily reworked. Major changes:
- lots of internal refactoring and variable renaming
- fixed self-building to support non-amd64 architectures
- changed to talk to the homeserver locally, over the container network (not
publicly)
- no more matrix-nginx-proxy support due to complexity (see below)
- no more `matrix_server_fqn_sliding_sync_proxy` in favor of
`matrix_sliding_sync_hostname` and `matrix_sliding_sync_path_prefix`
- runs on `matrix.DOMAIN/sliding-sync` by default, so it can tried
easily without having to create new DNS records
The variable was necessary when multiple playbooks could have
potentially tried to manage a shared `devture-traefik.serivce` systemd service
and shared `/devture-traefik` directory.
Since adcc6d9723, we use our own `/matrix/traefik`
(`matrix-traefik.service`) installation and no conflicts can arise.
It's safe to always enable the role, just like we do with all the other roles.
The migration is automatic. Existing users should experience a bit of
downtime until the playbook runs to completion, but don't need to do
anything manually.
This change is provoked by https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2535
While my statements there ("Traefik is a shared component among
sibling/related playbooks and should retain its global
non-matrix-prefixed name and path") do make sense, there's another point
of view as well.
With the addition of docker-socket-proxy support in bf2b540807,
we potentially introduced another non-`matrix-`-prefixed systemd service
and global path (`/devture-container-socket-proxy`). It would have
started to become messy.
Traefik always being called `devture-traefik.service` and using the `/devture-traefik` path
has the following downsides:
- different playbooks may write to the same place, unintentionally,
before you disable the Traefik role in some of them.
If each playbook manages its own installation, no such conflicts
arise and you'll learn about the conflict when one of them starts its
Traefik service and fails because the ports are already in use
- the data is scattered - backing up `/matrix` is no longer enough when
some stuff lives in `/devture-traefik` or `/devture-container-socket-proxy` as well;
similarly, deleting `/matrix` is no longer enough to clean up
For this reason, the Traefik instance managed by this playbook
will now be called `matrix-traefik` and live under `/matrix/traefik`.
This also makes it obvious to users running multiple playbooks, which
Traefik instance (powered by which playbook) is the active one.
Previously, you'd look at `devture-traefik.service` and wonder which
role was managing it.
We don't need these 2 roughly-the-same settings related to the
traefik-certs-dumper role.
For Traefik, it makes sense, because it's a component used by the
various related playbooks and they could step onto each other's toes
if the role is enabled, but Traefik is disabled (in that case, uninstall
tasks will run).
As for Traefik certs dumper, the other related playbooks don't have it,
so there's no conflict. Even if they used it, each one would use its own
instance (different `devture_traefik_certs_dumper_identifier`), so there
wouldn't be a conflict and uninstall tasks can run without any danger.
This allows people wishing to change or unset the resolver,
to have a single variable which they can toggle.
Unsetting the resolver is useful for using your own certificates
(not coming from a certificate resolver).
The newly extracted role also has native Traefik support,
so we no longer need to rely on `matrix-nginx-proxy` for
reverse-proxying to Ntfy.
The new role uses port `80` inside the container (not `8080`, like
before), because that's the default assumption of the officially
published container image. Using a custom port (like `8080`), means the
default healthcheck command (which hardcodes port `80`) doesn't work.
Instead of fiddling to override the healthcheck command, we've decided
to stick to the default port instead. This only affects the
inside-the-container port, not any external ports.
The new role also supports adding the network ranges of the container's
multiple additional networks as "exempt hosts". Previously, only one
network's address range was added to "exempt hosts".
Previously, it had to go through matrix-nginx-proxy.
It's exposed to Traefik directly via container labels now
Serving at a path other than `/` doesn't work well yet.
We'd like to auto-enable traefik-certs-dumper for these setups.
`devture_traefik_certs_dumper_ssl_dir_path` will be empty though,
so the role's validation will point people in the right direction.
This gets us started on adding a Traefik role and hooking Traefik:
- directly to services which support Traefik - we only have a few of
these right now, but the list will grow
- to matrix-nginx-proxy for most services that integrate with
matrix-nginx-proxy right now
Traefik usage should be disabled by default for now and nothing should
change for people just yet.
Enabling these experiments requires additional configuration like this:
```yaml
devture_traefik_ssl_email_address: '.....'
matrix_playbook_traefik_role_enabled: true
matrix_playbook_traefik_labels_enabled: true
matrix_ssl_retrieval_method: none
matrix_nginx_proxy_https_enabled: false
matrix_nginx_proxy_container_http_host_bind_port: ''
matrix_nginx_proxy_container_federation_host_bind_port: ''
matrix_nginx_proxy_trust_forwarded_proto: true
matrix_nginx_proxy_x_forwarded_for: '$proxy_add_x_forwarded_for'
matrix_coturn_enabled: false
```
What currently works is:
reverse-proxying for all nginx-proxy based services **except** for the Matrix homeserver
(both Client-Server an Federation traffic for the homeserver don't work yet)
Related to https://github.com/spantaleev/matrix-docker-ansible-deploy/pull/2427
This just enables the endpoint, which is somewhat helpful, but not
really enough to scrape them. Ideally, we'd be injecting these targets
into the Prometheus scrape config too.
For now, registering targets with Prometheus is very manual
(`matrix_prometheus_scraper_postgres_enabled`, `matrix_prometheus_scraper_hookshot_enabled`, ..).
This should be redone - e.g. a new `matrix_prometheus_scrape_config_jobs_auto` variable,
which is dynamically built in `group_vars/matrix_servers`.
