Certain use-cases may require that people mount additional files
into the matrix-nginx-proxy container. Similarly to how we do it
for Synapse, we are introducing a new variable that makes this
possible (`matrix_nginx_proxy_container_additional_volumes`).
This makes the htpasswd file for Synapse Metrics (introduced in #86,
Github Pull Request) to also perform mounting using this new mechanism.
Hopefully, for such an "extension", keeping htpasswd file-creation and
volume definition in the same place (the tasks file) is better.
All other major volumes' mounting mechanism remains the same (explicit
mounting).
Continuation of 1f0cc92b33.
As an explanation for the problem:
when saying `localhost` on the host, it sometimes gets resolved to `::1`
and sometimes to `127.0.0.1`. On the unfortunate occassions that
it gets resolved to `::1`, the container won't be able to serve the
request, because Docker containers don't have IPv6 enabled by default.
To avoid this problem, we simply prevent any lookups from happening
and explicitly use `127.0.0.1`.
This reverts commit 0dac5ea508.
Relying on pyOpenSSL is the Ansible way of doing things, but is
impractical and annoying for users.
`openssl` is easily available on most servers, even by default.
We'd better use that.
Seems like we unintentionally removed the mounting of certificates
(the `/matrix-config` mount) as part of splitting the playbook into
roles in 51312b8250.
It appears that those certificates weren't necessary for coturn to
funciton though, so we might just get rid of the configuration as well.
We run containers as a non-root user (no effective capabilities).
Still, if a setuid binary is available in a container image, it could
potentially be used to give the user the default capabilities that the
container was started with. For Docker, the default set currently is:
- "CAP_CHOWN"
- "CAP_DAC_OVERRIDE"
- "CAP_FSETID"
- "CAP_FOWNER"
- "CAP_MKNOD"
- "CAP_NET_RAW"
- "CAP_SETGID"
- "CAP_SETUID"
- "CAP_SETFCAP"
- "CAP_SETPCAP"
- "CAP_NET_BIND_SERVICE"
- "CAP_SYS_CHROOT"
- "CAP_KILL"
- "CAP_AUDIT_WRITE"
We'd rather prevent such a potential escalation by dropping ALL
capabilities.
The problem is nicely explained here: https://github.com/projectatomic/atomic-site/issues/203
This is a known/intentional regression since f92c4d5a27.
The new stance on this is that most people would not have
dnspython, but may have the `dig` tool. There's no good
reason for not increasing our chances of success by trying both
methods (Ansible dig lookup and using the `dig` CLI tool).
Fixes#85 (Github issue).
This makes all containers (except mautrix-telegram and
mautrix-whatsapp), start as a non-root user.
We do this, because we don't trust some of the images.
In any case, we'd rather not trust ALL images and avoid giving
`root` access at all. We can't be sure they would drop privileges
or what they might do before they do it.
Because Postfix doesn't support running as non-root,
it had to be replaced by an Exim mail server.
The matrix-nginx-proxy nginx container image is patched up
(by replacing its main configuration) so that it can work as non-root.
It seems like there's no other good image that we can use and that is up-to-date
(https://hub.docker.com/r/nginxinc/nginx-unprivileged is outdated).
Likewise for riot-web (https://hub.docker.com/r/bubuntux/riot-web/),
we patch it up ourselves when starting (replacing the main nginx
configuration).
Ideally, it would be fixed upstream so we can simplify.
We do match the defaults anyway (by default that is),
but people can customize `matrix_user_uid` and `matrix_user_uid`
and it wouldn't be correct then.
In any case, it's better to be explicit about such an important thing.
If this is a brand new server and Postgres had never started,
detecting it before we even start it is not possible.
This moves the logic, so that it happens later on, when Postgres
would have had the chance to start and possibly initialize
a new empty database.
Fixes#82 (Github issue)
The matrix-nginx-proxy role can now be used independently.
This makes it consistent with all other roles, with
the `matrix-base` role remaining as their only dependency.
Separating matrix-nginx-proxy was relatively straightforward, with
the exception of the Mautrix Telegram reverse-proxying configuration.
Mautrix Telegram, being an extension/bridge, does not feel important enough
to justify its own special handling in matrix-nginx-proxy.
Thus, we've introduced the concept of "additional configuration blocks"
(`matrix_nginx_proxy_proxy_matrix_additional_server_configuration_blocks`),
where any module can register its own custom nginx server blocks.
For such dynamic registration to work, the order of role execution
becomes important. To make it possible for each module participating
in dynamic registration to verify that the order of execution is
correct, we've also introduced a `matrix_nginx_proxy_role_executed`
variable.
It should be noted that this doesn't make the matrix-synapse role
dependent on matrix-nginx-proxy. It's optional runtime detection
and registration, and it only happens in the matrix-synapse role
when `matrix_mautrix_telegram_enabled: true`.
With this change, the following roles are now only dependent
on the minimal `matrix-base` role:
- `matrix-corporal`
- `matrix-coturn`
- `matrix-mailer`
- `matrix-mxisd`
- `matrix-postgres`
- `matrix-riot-web`
- `matrix-synapse`
The `matrix-nginx-proxy` role still does too much and remains
dependent on the others.
Wiring up the various (now-independent) roles happens
via a glue variables file (`group_vars/matrix-servers`).
It's triggered for all hosts in the `matrix-servers` group.
According to Ansible's rules of priority, we have the following
chain of inclusion/overriding now:
- role defaults (mostly empty or good for independent usage)
- playbook glue variables (`group_vars/matrix-servers`)
- inventory host variables (`inventory/host_vars/matrix.<your-domain>`)
All roles default to enabling their main component
(e.g. `matrix_mxisd_enabled: true`, `matrix_riot_web_enabled: true`).
Reasoning: if a role is included in a playbook (especially separately,
in another playbook), it should "work" by default.
Our playbook disables some of those if they are not generally useful
(e.g. `matrix_corporal_enabled: false`).
We've previously changed a bunch of lists in `homeserver.yaml.j2`
to be serialized using `|to_nice_yaml`, as that generates a more
readable list in YAML.
`matrix_synapse_federation_domain_whitelist`, however, couldn't have
been changed to that, as it can potentially be an empty list.
We may be able to differentiate between empty and non-empty now
and serialize it accordingly (favoring `|to_nice_yaml` if non-empty),
but it's not important enough to be justified. Thus, always
serializing with `|to_json`.
Fixes#78 (Github issue)
Riot-web parses integrations_widgets_urls as a list, thus causing it to incorrectly think Scalar widgets are non-Scalar and not passing the scalar token
As suggested in #63 (Github issue), splitting the
playbook's logic into multiple roles will be beneficial for
maintainability.
This patch realizes this split. Still, some components
affect others, so the roles are not really independent of one
another. For example:
- disabling mxisd (`matrix_mxisd_enabled: false`), causes Synapse
and riot-web to reconfigure themselves with other (public)
Identity servers.
- enabling matrix-corporal (`matrix_corporal_enabled: true`) affects
how reverse-proxying (by `matrix-nginx-proxy`) is done, in order to
put matrix-corporal's gateway server in front of Synapse
We may be able to move away from such dependencies in the future,
at the expense of a more complicated manual configuration, but
it's probably not worth sacrificing the convenience we have now.
As part of this work, the way we do "start components" has been
redone now to use a loop, as suggested in #65 (Github issue).
This should make restarting faster and more reliable.
This change is provoked by a few different things:
- #54 (Github Pull Request), which rightfully says that we need a
way to support ALL mxisd configuration options easily
- the upcoming mxisd 1.3.0 release, which drops support for
property-style configuration (dot-notation), forcing us to
redo the way we generate the configuration file
With this, mxisd is much more easily configurable now
and much more easily maintaneable by us in the future
(no need to introduce additional playbook variables and logic).
As suggested in #65 (Github issue), this patch switches
cronjob management from using templates to using Ansible's `cron` module.
It also moves the management of the nginx-reload cronjob to `setup_ssl_lets_encrypt.yml`,
which is a more fitting place for it (given that this cronjob is only required when
Let's Encrypt is used).
Pros:
- using a module is more Ansible-ish than templating our own files in
special directories
- more reliable: will fail early (during playbook execution) if `/usr/bin/crontab`
is not available, which is more of a guarantee that cron is working fine
(idea: we should probably install some cron package using the playbook)
Cons:
- invocation schedule is no longer configurable, unless we define individual
variables for everything or do something smart (splitting on ' ', etc.).
Likely not necessary, however.
- requires us to deprecate and clean-up after the old way of managing cronjobs,
because it's not compatible (using the same file as before means appending
additional jobs to it)
This means we no longer have a dependency on the `dig` program,
but we do have a dependency on `dnspython`.
Improves things as suggested in #65 (Github issue).
After having multiple people report issues with retrieving
SSL certificates, we've finally discovered the culprit to be
Ansible 2.5.1 (default and latest version on Ubuntu 18.04 LTS).
As silly as it is, certain distributions ("LTS" even) are 13 bugfix
versions of Ansible behind.
From now on, we try to auto-detect buggy Ansible versions and tell the
user. We also provide some tips for how to upgrade Ansible or
run it from inside a Docker container.
My testing shows that Ansible 2.4.0 and 2.4.6 are OK.
All other intermediate 2.4.x versions haven't been tested, but we
trust they're OK too.
From the 2.5.x releases, only 2.5.0 and 2.5.1 seem to be affected.
Ansible 2.5.2 corrects the problem with `include_tasks` + `with_items`.
This is a simplification and a way to make it consistent with
how we do Postgres imports (see 6d89319822), using
files coming from the server, not from the local machine.
By encouraging people NOT to use local files,
we potentially avoid problems such as #34 (Github issue),
where people would download `media_store` to their Mac's filesystem
and case-sensitivity issues will actually corrupt it.
By not encouraging local files usage, it's less likely that
people would copy (huge) directories to their local machine like that.
This is a simplification and a way to make it consistent with
how we do Postgres imports (see 6d89319822), using
files coming from the server, not from the local machine.
Until now, if the .sql file contained invalid data, psql would
choke on it, but still return an exit code of 0.
This is very misleading.
We need to pass `-v ON_ERROR_STOP=1` to make it exit
with a proper error exit code when failures happen.
We've had that logic in 2 places so far, leading to duplication
and a maintenance burden.
In the future, we'll also have an import-postgres feature,
which will also need Postgres version detection,
leading to more benefit from that logic being reusable.
Fixes#18 (Github issue).
It would probably be better if we serve our own page,
as the Matrix one says:
"To use this server you'll need a Matrix client", which
is true, but we install Riot by default and it'd be better if we mention
that instead.
If uppercase is used, certain tools (like certbot) would cause trouble.
They would retrieve a certificate for the lowercased domain name,
but we'd try to use it from an uppercase-named directory, which will
fail.
Besides certbot, we may experience other trouble too.
(it hasn't been investigated how far the breakage goes).
To fix it all, we lowercase `host_specific_hostname_identity` by default,
which takes care of the general use-case (people only setting that
and relying on us to build the other domain names - `hostname_matrix`
and `hostname_riot`).
For others, who decide to override these other variables directly
(and who may work around us and introduce uppercase there directly),
we also have the sanity-check tool warn if uppercase is detected
in any of the final domains.
Adds support for managing certificates manually and for
having the playbook generate self-signed certificates for you.
With this, Let's Encrypt usage is no longer required.
Fixes Github issue #50.
This is in line with what the recommendation is for matrix-corporal.
A value higher than 30 seconds is required to satisfy Riot
(and other clients') default long-polling behavior.
It looks like SELinux can be left running without any (so far) negative
effects on our Matrix services.
There's no need to use `:z` or `:Z` options when mounting volumes either.
This means that files we create are labeled with a default context
(which may not be ideal if we only want them used from containers),
but it's compatible and doesn't cause issues.
Relabelling files is probably something we wish to stay away from,
especially for things like the media store, which contains lots of
files and is possibly on a fuse-mounted (S3/goofys) filesystem.
The new image is built in a much better way (2-stage build)
and is 10x smaller.
In terms of Goofys version recency, it's about the same..
Both images (and others alike) seem to not use version tags,
but rather some `:latest` (master), with ewoutp/goofys being a bit
more recent than clodproto/goofys.
Not using version tags is good (in this case),
because the last Goofys release seems to be from about a year ago
and there had been a bunch of bugfixes afterwards.
This is described in Github issue #58.
Until now, we had the variable, but if you redefined it, you'd run
into multiple problems:
- we actually always mounted some "storage" directory to the Synapse
container. So if your media store is not there, you're out of luck
- homeserver.yaml always hardcoded the path to the media store,
as a directory called "media-store" inside the storage directory.
Relocating to outside the storage directory was out of the question.
Moreover, even if you had simply renamed the media store directory
(e.g. "media-store" -> "media_store"), it would have also caused trouble.
With this patch, we mount the media store's parent to the Synapse container.
This way, we don't care where the media store is (inside storage or
not). We also don't assume (anymore) that the final part of the path
is called "media-store" -- anything can be used.
The "storage" directory and variable (`matrix_synapse_storage_path`)
still remain for compatibility purposes. People who were previously
overriding `matrix_synapse_storage_path` can continue doing so
and their media store will be at the same place.
The playbook no longer explicitly creates the `matrix_synapse_storage_path` directory
though. It's not necessary. If the media store is specified to be within it, it will
get created when the media store directory is created by the playbook.
Previously, it was more necessary to have it
(because we had a dependency between matrix-synapse and matrix-nginx-proxy)..
But nowadays, it can be removed without negative side effects.
Restarting matrix-nginx-proxy is especially bad when the proxy is not installed at all.
mxisd supports several identity stores. Add support to configure two of them:
* synapseSql (storing identities directly in Synapse's database)
* LDAP
This removed the need to copy `mxisd.yaml.j2` to the inventory in case one wants
to use LDAP as identity store. Note that the previous solution (copying
`mxisd.yaml.j2` was poor because of two reasons:
* The copy remains outdated in case the original is updated in future versions
of this repo.
* The role's configuration should be in one place (configured only through role
variables) instead of in multiple.
Configuring more identity stores through role variables can be supported in the
future.
This is provoked by Github issue #46.
No client had made use of the well-known mechanism
so far, so the set up performed by this playbook was not tested
and turned out to be a little deficient.
Even though /.well-known/matrix/client is usually requested with a
simple request (no preflight), it's still considered cross-origin
and [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS)
applies. Thus, the file always needs to be served with the appropriate
`Access-Control-Allow-Origin` header.
Github issue #46 attempts to fix it at the "reverse-proxying" layer,
which may work, but would need to be done for every server.
It's better if it's done "upstream", so that all reverse-proxy
configurations can benefit.
Trying to:
- stay closer to naming in Synapse (autojoin -> auto_join)
- not create new variable namespaces (`matrix_homeserver_`),
when existing ones (`matrix_synapse_`) are more suitable
- allow `null` (`~`) values for `matrix_riot_web_welcome_user_id`
- render things like `auto_join_rooms` in `homeserver.yaml` more prettily
- fix breakage in `config.json` where `matrix_riot_web_roomdir_servers`
was rendered as YAML and not as JSON
- simplify code (especially in riot-web's `config.json`), which used
`if` statements that could have been omitted
- avoid changing comments in `homeserver.yaml` which are not ours,
so that we can keep closer to the configuration file generated by upstream
Pretty much all variables live in their own `matrix_<whatever>`
prefix now and are grouped closer together in the default
variables file (`roles/matrix-server/defaults/main.yml`).
It should be `/bin/mkdir` and `/bin/chown` on Ubuntu 18.04 for example.
Still, it doesn't seem like we need to create and chown these
directories at all, since the playbook takes care of creating them
and setting appropriate permission by itself.
If a network like `matrix-whatever` already exists for some reason,
the `docker_network` module would not create our `matrix` network.
Working around it by avoiding `docker_network` and doing it manually.
Fixes Github issue #12
`--log-driver=none` is used for all Docker containers now.
All these containers are started through systemd anyway and get logged in journald,
so there's no need for Docker to be logging the same thing using the default `json-file` driver.
Doing that was growing `/var/lib/docker/containers/..` infinitely until service/container restart.
As a result of this, things like `docker logs matrix-synapse` won't work anymore.
`journalctl -u matrix-synapse` is how one can see the logs.
If the playbook were to run with `--tags=setup-nginx-proxy`,
it wouldn't go into `setup_corporal.yml`, which meant it wouldn't
perform a bunch of `set_fact` calls which override important
nginx proxy configuration.
We run these variable overrides on each call now (tagged with `always`)
to avoid such problems in the future.
This disables federation on the 80 port, as it's
not necessary. We also disable the old Angular webclient.
For the federation port (8448), we disable the client APIs
as those are not necessary. Those can even cause trouble
if one doesn't know about them and thinks that guarding the client
APIs at the 80 port is enough.
Moving away from using the default bridge network to using our own.
This isolates our services from other Docker containers running
on the default network on the same host.
The benefits are that:
- isolation is a little better - we no longer share a default
bridge network with any other containers that might be running on the host
- there are no longer hard dependencies - we do service discovery
by DNS name, and not via explicit `--link` usage during container start,
so containers can start out of order and fail without bringing down others
with them
(`matrix-nginx-proxy` can continue running, even if one of the other services dies)
In the future, when other services get introduced,
the increased resilience and simplicity will help as well.
Until now, we were starting from a fresh configuration, as generated
by Synapse and manipulating it with regex and line replacements,
until we made it work.
This is more fragile and less predictable, so we're moving to a static
configuration file generated from a Jinja template.
The upside is that configuration will be stable and predictable.
The downside of this new approach is that any manual configuration changes
after the playbook is done, will be thrown away on future playbook
invocations.
There are 2 ways to work around the need for manual configuration
changes though:
- making them part of this playbook and its default template
configuration files (which benefits everyone)
- going your own way for a given host and overriding the template files
that gets used (that is, the
`matrix_synapse_template_synapse_homeserver` or
`matrix_synapse_template_synapse_log` variables)
This playbook does not set up guest access in Synapse anyway,
so until the need comes (or someone asks for it), guest access
is removed from riot-web's UI too.
As for supporting custom URLs, this is also not something
that seems like it'd be useful to most deployments.
Since cbee084ac1, this playbook supports Postgres 10.x,
but keeps existing Postgres-9.x installs on 9.x.
This playbook can now also be ran with `--tags=upgrade-postgres`
to make it upgrade from Postgres 9.x to 10.x (or other versions
in the future).
This playbook just tries to avoid trying to setup a Postgres 10
database with existing 9.x files, as that makes Postgres complain.
Due to this, existing installs (still on 9.x) are detected
and left on Postgres 9.x.
They need to be upgraded to Postgres 10.x manually.
Switching from from avhost/docker-matrix (silviof/docker-matrix)
to matrixdotorg/synapse.
The avhost/docker-matrix (silviof/docker-matrix) image used to bundle
in the coturn STUN/TURN server, so as part of the move,
we're separating this to a separately-ran service
(matrix-coturn.service, powered by instrumentisto/coturn-docker-image)
A `.log.config` file may be generated with a different
level of indentation depending on which (Docker image, etc.)
generates it.
With this patch, we tolerate different levels of indentation
(2 spaces, 4 spaces, etc.) and don't break the configuration.
When using matrix-nginx-proxy, the file permissions are organized
in a way that matrix-nginx-proxy could read the challenge files
produced by acmetool.
However, when another own/external webserver was used (like nginx
with our generated sample configuration), this could not work.
From on we're proxying the HTTP requests to port :402 in such a case,
which fixes the problem.
The matrix-nginx-proxy was reloaded on the 3rd day of the month (`15 4 3 * *`),
which makes no sense - it's too infrequently.
It's in line with the renewal time now (+5 minutes).
The non-working script is supposed to be fixed
by https://github.com/matrix-org/synapse/pull/2375
To have it work, we'd need an updated Docker image
of `silviof/matrix-riot-docker:latest`, which is not yet available
at the time of this commit.
Still, the previous patched synapse_port_db didn't work well either,
so it's not like we're regressing much by getting rid of it.
As described here (
https://github.com/matrix-org/synapse/issues/2438#issuecomment-327424711
), using own SSL certificates for the federation port is more fragile,
as renewing them could cause federation outages.
The recommended setup is to use the self-signed certificates generated
by Synapse.
On the 443 port (matrix-nginx-proxy) side, we still use the Let's Encrypt
certificates, which ensures API consumers work without having to trust
"our own CA".
Having done this, we also don't need to ever restart Synapse anymore,
as no new SSL certificates need to be applied there.
It's just matrix-nginx-proxy that needs to be restarted, and it doesn't
even need a full restart as an "nginx reload" does the job of swithing
to the new SSL certificates.
Moving keeps everything in the /matrix directory, so that we
wouldn't contaminate anything else on the system or risk
clashing with something else.
Also retrieving certificates separately for the Riot and Matrix domains,
which should help in multiple ways:
- allows them to be very different (completely separate base domain..)
- allows for Riot to be disabled for the playbook some time later
and still have the code not break
Let's let the admin set them as they wish.
We don't care what they are anyway.
If other things run on the same server,
it's also better not to hijack these for our
own purposes, especially when we don't need to.
The timedatectl call also seems to fail on Ubuntu 17.04
for some reason (missing timezones information file?).
The goal is to allow these to be on separate partitions
(including remote ones in the future).
Because the `silviof/docker-matrix` image chowns
everything to MATRIX_UID:MATRIX_GID on startup,
we definitely don't want to include `media_store` in it.
If it's on a remote FS, it would cause a slow startup.
Also, adding some safety checks to the "import media store"
task, after passing a wrong path to it on multiple occassions and
wondering what's wrong.
Also, making logging configurable. The default of keeping 10x100MB
log files is likely excessive and people may want to change that.
Port 8008 is forwarded in our case, so unless we adjust
`x_forwaded` for it, Docker's local network IPs are
logged/displayed for devices.
The TLS port (8448) is not proxied in our setup,
so its `x_forwarded` setting remains `false`.
Otherwise certains values in the config file,
such as `macaroon_secret_key`, would be regenerated,
which is not something that we want.
If `macaroon_secret_key` is regenerated, all users'
auth tokens will become invalid (effectively logging out
all users).
It was trying to omit the `-a` flag, but that wasn't enough,
because the underlying `register_new_matrix_user` command
prompts interactively if it doesn't see the `-a` flag
(it doesn't default to non-admin).
We need to answer such interactive prompts.
Some CentOS 7 hosts may not have firewalld installed.
We shouldn't expect it to be, but should ensure by ourselves that it is.
Docker likes to mess around with iptables forwarding rules,
so it ought to start after firewalld.
matrix-nginx-proxy will be occupying port 80 soon,
so that we can be more user-friendly and have
http->https forwarding for the Riot hostname.
During the playbook run, acmetool also expects to use
port 80 for domain verification.
During an initial playbook run, this wouldn't cause trouble
because matrix-nginx-proxy is not installed yet.
However, on subsequent playbook runs, it would cause trouble.
This ensures that if matrix-nginx-proxy is available
and running, it would be stopped before running acmetool
and started right after.
