finallycoffee/matrix-docker-ansible-deploy
5
1
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Merge branch 'master' into synapse-workers

Browse Source
This commit is contained in:
Slavi Pantaleev
2021-02-14 13:00:57 +02:00
parent e7f3f7c431 894679750e
commit 5cfeae806b
69 changed files with 1694 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
docs/configuring-dns.md
View File

@ -15,20 +15,25 @@ As we discuss in [Server Delegation](howto-server-delegation.md), there are 2 di
This playbook mostly discusses the well-known file method, because it's easier to manage with regard to certificates.
If you decide to go with the alternative method ([Server Delegation via a DNS SRV record (advanced)](howto-server-delegation.md#server-delegation-via-a-dns-srv-record-advanced)), please be aware that the general flow that this playbook guides you through may not match what you need to do.
## General outline of DNS settings you need to do
## Required DNS settings for services enabled by default
| Type | Host | Priority | Weight | Port | Target |
| ----- | ---------------------------- | -------- | ------ | ---- | ---------------------- |
| A | `matrix` | - | - | - | `matrix-server-IP` |
| CNAME | `element` | - | - | - | `matrix.<your-domain>` |
| CNAME | `dimension` (*) | - | - | - | `matrix.<your-domain>` |
| CNAME | `jitsi` (*) | - | - | - | `matrix.<your-domain>` |
| SRV | `_matrix-identity._tcp` | 10 | 0 | 443 | `matrix.<your-domain>` |
Be mindful as to how long it will take for the DNS records to propagate.
DNS records marked with `(*)` above are optional. They refer to services that will not be installed by default (see the section below). If you won't be installing these services, feel free to skip creating these DNS records. Also be mindful as to how long it will take for the DNS records to propagate.
If you are using Cloudflare DNS, make sure to disable the proxy and set all records to `DNS only`. Otherwise, fetching certificates will fail.
## Required DNS settings for optional services
| Type | Host | Priority | Weight | Port | Target |
| ----- | ---------------------------- | -------- | ------ | ---- | ---------------------- |
| CNAME | `dimension` (*) | - | - | - | `matrix.<your-domain>` |
| CNAME | `jitsi` (*) | - | - | - | `matrix.<your-domain>` |
| CNAME | `stats` (*) | - | - | - | `matrix.<your-domain>` |
## Subdomains setup
@ -41,6 +46,8 @@ The `dimension.<your-domain>` subdomain may be necessary, because this playbook
The `jitsi.<your-domain>` subdomain may be necessary, because this playbook could install the [Jitsi video-conferencing platform](https://jitsi.org/) for you. Jitsi installation is disabled by default, because it may be heavy and is not a core required component. To learn how to install it, see our [Jitsi](configuring-playbook-jitsi.md) guide. If you do not wish to set up Jitsi, feel free to skip the `jitsi.<your-domain>` DNS record.
The `stats.<your-domain>` subdomain may be necessary, because this playbook could install [Grafana](https://grafana.com/) and setup performance metrics for you. Grafana installation is disabled by default, it is not a core required component. To learn how to install it, see our [metrics and graphs guide](configuring-playbook-prometheus-grafana.md). If you do not wish to set up Grafana, feel free to skip the `stats.<your-domain>` DNS record. It is possible to install Prometheus without installing Grafana, this would also not require the `stats.<your-domain>` subdomain.
## `_matrix-identity._tcp` SRV record setup

2
docs/configuring-playbook-bridge-mautrix-signal.md
View File

@ -14,7 +14,7 @@ matrix_mautrix_signal_enabled: true
## Set up Double Puppeting
If you'd like to use [Double Puppeting](https://github.com/tulir/mautrix-whatsapp/wiki/Authentication#replacing-whatsapp-accounts-matrix-puppet-with-matrix-account) (hint: you most likely do), you have 2 ways of going about it.
If you'd like to use [Double Puppeting](https://github.com/tulir/mautrix-signal/wiki/Authentication#double-puppeting) (hint: you most likely do), you have 2 ways of going about it.
### Method 1: automatically, by enabling Shared Secret Auth

31
docs/configuring-playbook-etherpad.md Normal file
View File

@ -0,0 +1,31 @@
# Setting up Etherpad (optional)
[Etherpad](https://etherpad.org) is is an open source collaborative text editor that can be embedded in a Matrix chat room using the [Dimension integrations manager](https://dimension.t2bot.io)
When enabled together with the Jitsi audio/video conferencing system (see [our docs on Jitsi](configuring-playbook-jitsi.md)), it will be made available as an option during the conferences.
## Prerequisites
For the self-hosted Etherpad instance to be available to your users, you must first enable and configure the **Dimension integrations manager** as described in [the playbook documentation](configuring-playbook-dimension.md)
## Installing
[Etherpad](https://etherpad.org) installation is disabled by default. You can enable it in your configuration file (`inventory/host_vars/matrix.<your-domain>/vars.yml`):
```yaml
matrix_etherpad_enabled: true
```
## Set Dimension default to the self-hosted Etherpad
The Dimension administrator users can configure the default URL template. The Dimension configuration menu can be accessed with the sprocket icon as you begin to add a widget to a room in Element. There you will find the Etherpad Widget Configuration action beneath the _Widgets_ tab. Replace `scalar.vector.im` with your own Dimension domain.
### Removing the integrated Etherpad chat
If you wish to disable the Etherpad chat button, you can do it by appending `?showChat=false` to the end of the pad URL, or the template.
Example: `https://dimension.<your-domain>/etherpad/p/$roomId_$padName?showChat=false`
## Known issues
If your Etherpad widget fails to load, this might be due to Dimension generating a Pad name so long, the Etherpad app rejects it.
`$roomId_$padName` can end up being longer than 50 characters. You can avoid having this problem by altering the template so it only contains the three word random identifier `$padName`.

26
docs/configuring-playbook-nginx.md
View File

@ -34,8 +34,7 @@ Possible values are:
- `"intermediate"` (**default**) - Recommended configuration for a general-purpose server
- `"old"` - Services accessed by very old clients or libraries, such as Internet Explorer 8 (Windows XP), Java 6, or OpenSSL 0.9.8
**Be really carefull when setting it to `"modern"`**. This could break comunication with other Matrix servers, limiting your federation posibilities. The
[Federarion tester](https://federationtester.matrix.org/) also won't work.
**Be really carefull when setting it to `"modern"`**. This could break comunication with other Matrix servers, limiting your federation posibilities.
Besides changing the preset (`matrix_nginx_proxy_ssl_preset`), you can also directly override these 3 variables:
@ -60,3 +59,26 @@ This will disable the access logging for nginx.
```yaml
matrix_nginx_proxy_access_log_enabled: false
```
## Additional configuration
This playbook also allows for additional configuration to be applied to the nginx server.
If you want this playbook to obtain and renew certificates for other domains, then you can set the `matrix_ssl_additional_domains_to_obtain_certificates_for` variable (as mentioned in the [Obtaining SSL certificates for additional domains](configuring-playbook-ssl-certificates.md#obtaining-ssl-certificates-for-additional-domains) documentation as well). Make sure that you have set the DNS configuration for the domains you want to include to point at your server.
```yaml
matrix_ssl_additional_domains_to_obtain_certificates_for:
- domain.one.example
- domain.two.example
```
You can include additional nginx configuration by setting the `matrix_nginx_proxy_proxy_http_additional_server_configuration_blocks` variable.
```yaml
matrix_nginx_proxy_proxy_http_additional_server_configuration_blocks:
- |
# These lines will be included in the nginx configuration.
# This is at the top level of the file, so you will need to define all of the `server { ... }` blocks.
- |
# For advanced use, have a look at the template files in `roles/matrix-nginx-proxy/templates/nginx/conf.d`
```

66
docs/configuring-playbook-prometheus-grafana.md Normal file
View File

@ -0,0 +1,66 @@
# Enabling metrics and graphs for your Matrix server (optional)
It can be useful to have some (visual) insight into the performance of your homeserver.
You can enable this with the following settings in your configuration file (`inventory/host_vars/matrix.<your-domain>/vars.yml`):
```yaml
matrix_prometheus_enabled: true
matrix_prometheus_node_exporter_enabled: true
matrix_grafana_enabled: true
matrix_grafana_anonymous_access: false
# This has no relation to your Matrix user id. It can be any username you'd like.
# Changing the username subsequently won't work.
matrix_grafana_default_admin_user: some_username_chosen_by_you
# Passwords containing special characters may be troublesome.
# Changing the password subsequently won't work.
matrix_grafana_default_admin_password: some_strong_password_chosen_by_you
```
By default, a [Grafana](https://grafana.com/) web user-interface will be available at `https://stats.<your-domain>`.
## What does it do?
Name | Description
-----|----------
`matrix_prometheus_enabled`|[Prometheus](https://prometheus.io) is a time series database. It holds all the data we're going to talk about.
`matrix_prometheus_node_exporter_enabled`|[Node Exporter](https://prometheus.io/docs/guides/node-exporter/) is an addon of sorts to Prometheus that collects generic system information such as CPU, memory, filesystem, and even system temperatures
`matrix_grafana_enabled`|[Grafana](https://grafana.com/) is the visual component. It shows (on the `stats.<your-domain>` subdomain) the dashboards with the graphs that we're interested in
`matrix_grafana_anonymous_access`|By default you need to log in to see graphs. If you want to publicly share your graphs (e.g. when asking for help in [`#synapse:matrix.org`](https://matrix.to/#/#synapse:matrix.org?via=matrix.org&via=privacytools.io&via=mozilla.org)) you'll want to enable this option.
`matrix_grafana_default_admin_user`<br>`matrix_grafana_default_admin_password`|By default Grafana creates a user with `admin` as the username and password. If you feel this is insecure and you want to change it beforehand, you can do that here
## Security and privacy
Metrics and resulting graphs can contain a lot of information. This includes system specs but also usage patterns. This applies especially to small personal/family scale homeservers. Someone might be able to figure out when you wake up and go to sleep by looking at the graphs over time. Think about this before enabling anonymous access. And you should really not forget to change your Grafana password.
Most of our docker containers run with limited system access, but the `prometheus-node-exporter` has access to the host network stack and (readonly) root filesystem. This is required to report on them. If you don't like that, you can set `matrix_prometheus_node_exporter_enabled: false` (which is actually the default). You will still get Synapse metrics with this container disabled. Both of the dashboards will always be enabled, so you can still look at historical data after disabling either source.
## Collecting metrics to an external Prometheus server
If you wish, you could expose homeserver metrics without enabling (installing) Prometheus and Grafana via the playbook. This may be useful for hooking Matrix services to an external Prometheus/Grafana installation.
To do this, you may be interested in the following variables:
Name | Description
-----|----------
`matrix_synapse_metrics_enabled`|Set this to `true` to make Synapse expose metrics (locally, on the container network)
`matrix_nginx_proxy_proxy_synapse_metrics`|Set this to `true` to make matrix-nginx-proxy expose the Synapse metrics at `https://matrix.DOMAIN/_synapse/metrics`
`matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_enabled`|Set this to `true` to password-protect (using HTTP Basic Auth) `https://matrix.DOMAIN/_synapse/metrics` (the username is always `prometheus`, the password is defined in `matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key`)
`matrix_nginx_proxy_proxy_synapse_metrics_basic_auth_key`|Set this to a password to use for HTTP Basic Auth for protecting `https://matrix.DOMAIN/_synapse/metrics` (the username is always `prometheus` - it's not configurable)
## More inforation
- [Understanding Synapse Performance Issues Through Grafana Graphs](https://github.com/matrix-org/synapse/wiki/Understanding-Synapse-Performance-Issues-Through-Grafana-Graphs) at the Synapse Github Wiki
- [The Prometheus scraping rules](https://github.com/matrix-org/synapse/tree/master/contrib/prometheus) (we use v2)
- [The Synapse Grafana dashboard](https://github.com/matrix-org/synapse/tree/master/contrib/grafana)
- [The Node Exporter dashboard](https://github.com/rfrail3/grafana-dashboards) (for generic non-synapse performance graphs)

135
docs/configuring-playbook-s3.md
View File

@ -6,6 +6,11 @@ If that's alright, you can skip this.
If you'd like to store Synapse's content repository (`media_store`) files on Amazon S3 (or other S3-compatible service),
you can let this playbook configure [Goofys](https://github.com/kahing/goofys) for you.
Using a Goofys-backed media store works, but performance may not be ideal. If possible, try to use a region which is close to your Matrix server.
If you'd like to move your locally-stored media store data to Amazon S3 (or another S3-compatible object store), we also provide some migration instructions below.
## Amazon S3
You'll need an Amazon S3 bucket and some IAM user credentials (access key + secret key) with full write access to the bucket. Example security policy:
@ -50,3 +55,133 @@ matrix_s3_media_store_custom_endpoint_enabled: true
# Example: "https://storage.googleapis.com"
matrix_s3_media_store_custom_endpoint: "your-custom-endpoint"
```
### Backblaze B2
To use [Backblaze B2](https://www.backblaze.com/b2/cloud-storage.html):
- create a new **private** bucket through its user interface (you can call it something like `matrix-DOMAIN-media-store`)
- note the **Endpoint** for your bucket (something like `s3.us-west-002.backblazeb2.com`)
- adjust its lifecycle rules to use the following **custom** rules:
- File Path: *empty value*
- Days Till Hide: *empty value*
- Days Till Delete: `1`
- go to [App Keys](https://secure.backblaze.com/app_keys.htm) and use the **Add a New Application Key** to create a new one
- restrict it to the previously created bucket (e.g. `matrix-DOMAIN-media-store`)
- give it *Read & Write* access
Copy the `keyID` and `applicationKey`.
You need the following *additional* playbook configuration (on top of what you see above):
```yaml
matrix_s3_media_store_bucket_name: "YOUR_BUCKET_NAME_GOES_HERE"
matrix_s3_media_store_aws_access_key: "YOUR_keyID_GOES_HERE"
matrix_s3_media_store_aws_secret_key: "YOUR_applicationKey_GOES_HERE"
matrix_s3_media_store_custom_endpoint_enabled: true
matrix_s3_media_store_custom_endpoint: "https://s3.us-west-002.backblazeb2.com" # this may be different for your bucket
```
If you have local media store files and wish to migrate to Backblaze B2 subsequently, follow our [migration guide to Backblaze B2](#migrating-to-backblaze-b2) below instead of applying this configuration as-is.
## Migrating from local filesystem storage to S3
It's a good idea to [make a complete server backup](faq.md#how-do-i-backup-the-data-on-my-server) before migrating your local media store to an S3-backed one.
Follow one of the guides below for a migration path from a locally-stored media store to one stored on S3-compatible storage:
- [Migrating to any S3-compatible storage (universal, but likely slow)](#migrating-to-any-s3-compatible-storage-universal-but-likely-slow)
- [Migrating to Backblaze B2](#migrating-to-backblaze-b2)
### Migrating to any S3-compatible storage (universal, but likely slow)
It's a good idea to [make a complete server backup](faq.md#how-do-i-backup-the-data-on-my-server) before doing this.
1. Proceed with the steps below without stopping Matrix services
2. Start by adding the base S3 configuration in your `vars.yml` file (seen above, may be different depending on the S3 provider of your choice)
3. In addition to the base configuration you see above, add this to your `vars.yml` file:
```yaml
matrix_s3_media_store_path: /matrix/s3-media-store
```
This enables S3 support, but mounts the S3 storage bucket to `/matrix/s3-media-store` without hooking it to your homeserver yet. Your homeserver will still continue using your local filesystem for its media store.
5. Run the playbook to apply the changes: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start`
6. Do an **initial sync of your files** by running this **on the server** (it may take a very long time):
```sh
sudo -u matrix -- rsync --size-only --ignore-existing -avr /matrix/synapse/storage/media-store/. /matrix/s3-media-store/.
```
You may need to install `rsync` manually.
7. Stop all Matrix services (`ansible-playbook -i inventory/hosts setup.yml --tags=stop`)
8. Start the S3 service by running this **on the server**: `systemctl start matrix-goofys`
9. Sync the files again by re-running the `rsync` command you see in step #6
10. Stop the S3 service by running this **on the server**: `systemctl stop matrix-goofys`
11. Get the old media store out of the way by running this command on the server:
```sh
mv /matrix/synapse/storage/media-store /matrix/synapse/storage/media-store-local-backup
```
12. Remove the `matrix_s3_media_store_path` configuration from your `vars.yml` file (undoing step #3 above)
13. Run the playbook: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start`
14. You're done! Verify that loading existing (old) media files works and that you can upload new ones.
15. When confident that it all works, get rid of the local media store directory: `rm -rf /matrix/synapse/storage/media-store-local-backup`
### Migrating to Backblaze B2
It's a good idea to [make a complete server backup](faq.md#how-do-i-backup-the-data-on-my-server) before doing this.
1. While all Matrix services are running, run the following command on the server:
(you need to adjust the 3 `--env` line below with your own data)
```sh
docker run -it --rm -w /work \
--env='B2_KEY_ID=YOUR_KEY_GOES_HERE' \
--env='B2_KEY_SECRET=YOUR_SECRET_GOES_HERE' \
--env='B2_BUCKET_NAME=YOUR_BUCKET_NAME_GOES_HERE' \
-v /matrix/synapse/storage/media-store/:/work \
--entrypoint=/bin/sh \
docker.io/tianon/backblaze-b2:2.1.0 \
-c 'b2 authorize-account $B2_KEY_ID $B2_KEY_SECRET > /dev/null && b2 sync /work/ b2://$B2_BUCKET_NAME'
```
This is some initial file sync, which may take a very long time.
2. Stop all Matrix services (`ansible-playbook -i inventory/hosts setup.yml --tags=stop`)
3. Run the command from step #1 again.
Doing this will sync any new files that may have been created locally in the meantime.
Now that Matrix services aren't running, we're sure to get Backblaze B2 and your local media store fully in sync.
4. Get the old media store out of the way by running this command on the server:
```sh
mv /matrix/synapse/storage/media-store /matrix/synapse/storage/media-store-local-backup
```
5. Put the [Backblaze B2 settings seen above](#backblaze-b2) in your `vars.yml` file
6. Run the playbook: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-all,start`
7. You're done! Verify that loading existing (old) media files works and that you can upload new ones.
8. When confident that it all works, get rid of the local media store directory: `rm -rf /matrix/synapse/storage/media-store-local-backup`

15
docs/configuring-playbook-ssl-certificates.md
View File

@ -74,15 +74,12 @@ If you are hosting other domains on the Matrix machine, you can make the playboo
To do that, simply define your own custom configuration like this:
```yaml
# Note: we need to explicitly list the aforementioned Matrix domains that you use (Matrix, Element, Dimension).
# In this example, we retrieve an extra certificate - one for the base domain (in the `matrix_domain` variable).
# In this example, we retrieve 2 extra certificates,
# one for the base domain (in the `matrix_domain` variable) and one for a hardcoded domain.
# Adding any other additional domains (hosted on the same machine) is possible.
matrix_ssl_domains_to_obtain_certificates_for:
- '{{ matrix_server_fqn_matrix }}'
- '{{ matrix_server_fqn_element }}'
- '{{ matrix_server_fqn_dimension }}'
- '{{ matrix_server_fqn_jitsi }}'
matrix_ssl_additional_domains_to_obtain_certificates_for:
- '{{ matrix_domain }}'
- 'another.domain.example.com'
```
After redefining `matrix_ssl_domains_to_obtain_certificates_for`, to actually obtain certificates you should:
@ -91,9 +88,9 @@ After redefining `matrix_ssl_domains_to_obtain_certificates_for`, to actually ob
- re-run the SSL part of the playbook and restart all services: `ansible-playbook -i inventory/hosts setup.yml --tags=setup-ssl,start`
The certificate files would be available in `/matrix/ssl/config/live/<your-other-domain>/...`.
The certificate files would be made available in `/matrix/ssl/config/live/<your-other-domain>/...`.
For automated certificate renewal to work, each port `80` vhost for each domain you are obtaining certificates for needs to forward requests for `/.well-known/acme-challenge` to the certbot container we use for renewal.
See how this is configured for the `matrix.` subdomain in `/matrix/nginx-proxy/conf.d/matrix-synapse.conf`
Don't be alarmed if the above configuraiton file says port `8080`, instead of port `80`. It's due to port mapping due to our use of containers.
Don't be alarmed if the above configuration file says port `8080`, instead of port `80`. It's due to port mapping due to our use of containers.

2
docs/configuring-playbook.md
View File

@ -35,6 +35,8 @@ When you're done with all the configuration you'd like to do, continue with [Ins
- [Setting up Dynamic DNS](configuring-playbook-dynamic-dns.md) (optional)
- [Enabling metrics and graphs (Prometheus, Grafana) for your Matrix server](configuring-playbook-prometheus-grafana.md) (optional)
### Core service adjustments
- [Configuring Synapse](configuring-playbook-synapse.md) (optional)

8
docs/container-images.md
View File

@ -32,6 +32,8 @@ These services are not part of our default installation, but can be enabled by [
- [ewoutp/goofys](https://hub.docker.com/r/ewoutp/goofys/) - the [Goofys](https://github.com/kahing/goofys) Amazon [S3](https://aws.amazon.com/s3/) file-system-mounting program (optional)
- [etherpad/etherpad](https://hub.docker.com/r/etherpad/etherpad/) - the [Etherpad](https://etherpad.org) realtime collaborative text editor that can be used in a Jitsi audio/video call or integrated as a widget into Matrix chat rooms via the Dimension integration manager (optional)
- [devture/email2matrix](https://hub.docker.com/r/devture/email2matrix/) - the [Email2Matrix](https://github.com/devture/email2matrix) email server, which can relay email messages to Matrix rooms (optional)
- [devture/matrix-corporal](https://hub.docker.com/r/devture/matrix-corporal/) - [Matrix Corporal](https://github.com/devture/matrix-corporal): reconciliator and gateway for a managed Matrix server (optional)
@ -83,3 +85,9 @@ These services are not part of our default installation, but can be enabled by [
- [anoa/matrix-reminder-bot](https://hub.docker.com/r/anoa/matrix-reminder-bot) - the [matrix-reminder-bot](https://github.com/anoadragon453/matrix-reminder-bot) bot for one-off & recurring reminders and alarms (optional)
- [awesometechnologies/synapse-admin](https://hub.docker.com/r/awesometechnologies/synapse-admin) - the [synapse-admin](https://github.com/Awesome-Technologies/synapse-admin) web UI tool for administrating users and rooms on your Matrix server (optional)
- [prom/prometheus](https://hub.docker.com/r/prom/prometheus/) - [Prometheus](https://github.com/prometheus/prometheus/) is a systems and service monitoring system
- [prom/node-exporter](https://hub.docker.com/r/prom/node-exporter/) - [Prometheus Node Exporter](https://github.com/prometheus/node_exporter/) is an addon for Prometheus that gathers standard system metrics
- [grafana/grafana](https://hub.docker.com/r/grafana/grafana/) - [Grafana](https://github.com/grafana/grafana/) is a graphing tool that works well with the above two images. Our playbook also adds two dashboards for [Synapse](https://github.com/matrix-org/synapse/tree/master/contrib/grafana) and [Node Exporter](https://github.com/rfrail3/grafana-dashboards)

2
docs/prerequisites.md
View File

@ -2,7 +2,7 @@
To install Matrix services using this Ansible playbook, you need:
- (Recommended) An **x86** server running one of these operating systems:
- (Recommended) An **x86** server ([What kind of server specs do I need?](faq.md#what-kind-of-server-specs-do-i-need)) running one of these operating systems:
- **CentOS** (7 only for now; [8 is not yet supported](https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/300))
- **Debian** (9/Stretch or newer)
- **Ubuntu** (16.04 or newer, although [20.04 may be problematic](ansible.md#supported-ansible-versions))