Initial work on Synapse 0.99/1.0 preparation

docs/configuring-dns.md

@@ -5,26 +5,50 @@ To set up Matrix on your domain, you'd need to do some DNS configuration.
 To use an identifier like `@<username>:<your-domain>`, you don't actually need
 to install anything on the actual `<your-domain>` server.
 
+
+## General outline of DNS settings you need to do
+
+| Type  | Host                    | Priority | Weight | Port | Target                 |
+| ----- | ----------------------- | -------- | ------ | ---- | ---------------------- |
+| A     | `matrix`                | -        | -      | -    | `matrix-server-IP`     |
+| CNAME | `riot`                  | -        | -      | -    | `matrix.<your-domain>` |
+| SRV   | `_matrix._tcp`          | 10       | 0      | 8448 | `matrix.<your-domain>` |
+| SRV   | `_matrix-identity._tcp` | 10       | 0      | 443  | `matrix.<your-domain>` |
+
+The `_matrix._tcp` SRV record is a temporary measure and will not be necessary in the near future.
+In fact, it will have to be removed at some point. To learn more about that, read below.
+
+
+## Subdomains setup
+
+As the table above illustrates, you need to create 2 subdomains (`matrix.<your-domain>` and `riot.<your-domain>`) and point both of them to your new server's IP address (DNS `A` record or `CNAME` record is fine).
+
+The `riot.<your-domain>` subdomain is necessary, because this playbook installs the Riot web client for you.
+If you'd rather instruct the playbook not to install Riot (`matrix_riot_web_enabled: false` when [Configuring the playbook](configuring-playbook.md) later), feel free to skip the `riot.<your-domain>` DNS record.
+
+
+## `_matrix._tcp` SRV record setup (temporary requirement)
+
 All services created by this playbook are meant to be installed on their own server (such as `matrix.<your-domain>`).
 
-To accomplish such a "redirect", you need to instruct the Matrix network of this by setting up a DNS SRV record.
+To use a Matrix user identifier like `@<username>:<your-domain>` while hosting services on `matrix.<your-domain>`, we need to instruct the Matrix network of such a delegation/redirection by means of setting up a DNS SRV record.
+
 The SRV record should look like this:
 - Name: `_matrix._tcp` (use this text as-is)
 - Content: `10 0 8448 matrix.<your-domain>` (replace `<your-domain>` with your own)
 
+A [new file-based mechanism for Federation Server Discovery](configuring-well-known.md#introduction-to-federation-server-discovery) is superseding the `_matrix._tcp` SRV record. **During the transition phase, you'll need to set up both mechanisms**. We'll instruct you how to set up the file-based mechanism after the [installation phase](installing.md) for this playbook.
+
+Doing delegation/redirection of Matrix services using a DNS SRV record (`_matrix._tcp`) is a **temporary measure** that is only necessary before Synapse v1.0 is released.
+
+As more and more people upgrade to the Synapse v0.99 transitional release and just before the final Synapse v1.0 gets released, at some point in the near future **you will need to remove the `_matrix._tcp` SRV record** and leave only the [new file-based mechanism for Federation Server Discovery](configuring-well-known.md#introduction-to-federation-server-discovery) in place.
+
+
+## `_matrix-identity._tcp` SRV record setup
+
 To make the [mxisd](https://github.com/kamax-io/mxisd) Identity Server (which this playbook installs for you) be authoritative for your domain name, set up one more SRV record that looks like this:
 - Name: `_matrix-identity._tcp` (use this text as-is)
 - Content: `10 0 443 matrix.<your-domain>` (replace `<your-domain>` with your own)
 
-Once you've set up these DNS SRV records, you should create 2 other domain names (`matrix.<your-domain>` and `riot.<your-domain>`) and point both of them to your new server's IP address (DNS `A` record or `CNAME` is fine).
 
-This playbook can then install all the services on that new server and you'll be able to join the Matrix network as `@<username>:<your-domain>`.
+When you're done with the DNS configuration and ready to proceed, continue with [Configuring this Ansible playbook](configuring-playbook.md).
-
-| Type  | Host                    | Priority | Weight | Port | Target                 |
-| ----- | ----------------------- | -------- | ------ | ---- | ---------------------- |
-| SRV   | `_matrix._tcp`          | 10       | 0      | 8448 | `matrix.<your-domain>` |
-| SRV   | `_matrix-identity._tcp` | 10       | 0      | 443  | `matrix.<your-domain>` |
-| A     | `matrix`                | -        | -      | -    | `server-IP`            |
-| CNAME | `riot`                  | -        | -      | -    | `matrix.<your-domain>` |
-
-When ready to proceed, continue with [Configuring this Ansible playbook](configuring-playbook.md).

docs/configuring-playbook.md

@@ -1,6 +1,6 @@
 # Configuring the Ansible playbook
 
-Once you have your server and you have [configured your DNS records](configuring-dns.md#configuring-dns), you can proceed with configuring this playbook, so that it knows what to install and where.
+Once you have your server and you have [configured your DNS records](configuring-dns.md), you can proceed with configuring this playbook, so that it knows what to install and where.
 
 You can follow these steps:
 

docs/configuring-well-known.md

@@ -1,44 +1,66 @@
-# Configuring service discovery via .well-known
+# Configuring Service Discovery via .well-known
+
+Service discovery is a way for the Matrix network to discover where a Matrix server is.
+
+There are 2 types of well-known service discovery that Matrix makes use of:
+
+- (important) **Federation Server discovery** (`/.well-known/matrix/server`) -- assists other servers in the Matrix network with finding your server. Without a proper configuration, your server will effectively not be part of the Matrix network. Learn more in [Introduction to Federation Server Discovery](#introduction-to-federation-server-discovery)
+
+- (not that important) **Client Server discovery** (`/.well-known/matrix/client`) -- assists programs that you use to connect to your server (e.g. Riot), so that they can make it more convenient for you by automatically configuring the "Homeserver URL" and "Identity Server URL" addresses. Learn more in [Introduction to Client Server Discovery](#introduction-to-client-server-discovery)
 
 
-## Introduction
+## Introduction to Federation Server Discovery
 
-Service discovery lets various client programs which support it, to receive a full user id (e.g. `@username:example.com`) and determine where the Matrix server is automatically (e.g. `https://matrix.example.com`).
+All services created by this playbook are meant to be installed on their own server (such as `matrix.<your-domain>`).
 
-This lets your users easily connect to your Matrix server without having to customize connection URLs.
+As [per the Server-Server specification](https://matrix.org/docs/spec/server_server/r0.1.0.html#server-discovery), to use a Matrix user identifier like `@<username>:<your-domain>` while hosting services on a subdomain like `matrix.<your-domain>`, we need to instruct the Matrix network of such a delegation/redirection by means of setting up a `/.well-known/matrix/server` file on the base domain (`<your-domain.com>).
 
-As [per the specification](https://matrix.org/docs/spec/client_server/r0.4.0.html#server-discovery) Matrix does service discovery using a `/.well-known/matrix/client` file hosted on the base domain (e.g. `example.com`).
+We have discussed this same thing already in the "`_matrix._tcp` SRV record setup (temporary requirement)" section of [Configuring DNS](configuring-dns.md).
+
+In short, you are required to set up both a `_matrix._tcp` DNS SRV record and the `/.well-known/matrix/server` file at the moment.
+
+As the Synapse server progresses towards v1.0, only the `/.well-known/matrix/server` file will be used. At that future moment, you would need to remove the `_matrix._tcp` SRV record because Synapse v1.0+ will do the wrong thing if a SRV record exists. During the transitional phase (before Synapse 1.0), we do need to have both a SRV record and a `/.well-known/matrix/server` file, in order to federate correctly with v0.99 and older Synapse versions.
+
+To learn how to set it up, read the Installing section below.
+
+
+## Introduction to Client Server Discovery
+
+Client Server Service discovery lets various client programs which support it, to receive a full user id (e.g. `@username:example.com`) and determine where the Matrix server is automatically (e.g. `https://matrix.example.com`).
+
+This lets you (and your users) easily connect to your Matrix server without having to customize connection URLs. When using client programs that support it, you won't need to point them to `https://matrix.example.com` in Custom Server options manually anymore. The connection URL would be discovered automatically from your full username.
+
+As [per the Client-Server specification](https://matrix.org/docs/spec/client_server/r0.4.0.html#server-discovery) Matrix does Client Server service discovery using a `/.well-known/matrix/client` file hosted on the base domain (e.g. `example.com`).
 
 However, this playbook installs your Matrix server on another domain (e.g. `matrix.example.com`) and not on the base domain (e.g. `example.com`), so it takes a little extra manual effort to set up the file.
 
-
+To learn how to set it up, read the Installing section below.
-## Prerequisites
-
-To implement service discovery, your base domain's server (e.g. `example.com`) needs to support HTTPS.
 
 
-## Setting it up
+## Installing well-known files on the base domain's server
 
-To make things easy for you to set up, this playbook generates and hosts the well-known file on the Matrix domain's server (e.g. `https://matrix.example.com/.well-known/matrix/client`), even though this is the wrong place to host it.
+To implement the two service discovery mechanisms, your base domain's server (e.g. `example.com`) needs to support HTTPS.
 
-You have 2 options when it comes to installing the file on the base domain's server:
+To make things easy for you to set up, this playbook generates and hosts 2 well-known files on the Matrix domain's server (e.g. `https://matrix.example.com/.well-known/matrix/server` and `https://matrix.example.com/.well-known/matrix/client`), even though this is the wrong place to host them.
+
+You have 2 options when it comes to installing the files on the base domain's server:
 
 
-### (Option 1): **Copying the file manually** to your base domain's server
+### (Option 1): **Copying the files manually** to your base domain's server
 
 **Hint**: Option 2 (below) is generally a better way to do this. Make sure to go with that one, if possible.
 
 All you need to do is:
 
-- copy the `/.well-known/matrix/client` from the Matrix server (e.g. `matrix.example.com`) to your base domain's server (`example.com`).
+- copy `/.well-known/matrix/server` and `/.well-known/matrix/client` from the Matrix server (e.g. `matrix.example.com`) to your base domain's server (`example.com`).
 
-- set up the server at your base domain (e.g. `example.com`) so that it adds an extra HTTP header when serving the `/.well-known/matrix/client` file. [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS), the `Access-Control-Allow-Origin` header should be set with a value of `*`. If you don't do this step, web-based Matrix clients (like Riot) may fail to work.
+- set up the server at your base domain (e.g. `example.com`) so that it adds an extra HTTP header when serving the `/.well-known/matrix/client` file. [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS), the `Access-Control-Allow-Origin` header should be set with a value of `*`. If you don't do this step, web-based Matrix clients (like Riot) may fail to work. Setting up headers for the `/.well-known/matrix/server` file is not necessary, as this file is only consumed by non-browsers, which don't care about CORS.
 
 This is relatively easy to do and possibly your only choice if you can only host static files from the base domain's server.
-It is, however, **a little fragile**, as future updates performed by this playbook may regenerate the well-known file and you may need to notice that and copy it again.
+It is, however, **a little fragile**, as future updates performed by this playbook may regenerate the well-known files and you may need to notice that and copy them over again.
 
 
-### (Option 2): **Setting up reverse-proxying** of the well-known file from the base domain's server to the Matrix server
+### (Option 2): **Setting up reverse-proxying** of the well-known files from the base domain's server to the Matrix server
 
 This option is less fragile and generally better.
 
@@ -91,6 +113,9 @@ Make sure to:
 
 ## Confirming it works
 
-No matter which method you've used to set up the well-known file, if you've done it correctly you should be able to see a JSON file at a URL like this: `https://<domain>/.well-known/matrix/client`.
+No matter which method you've used to set up the well-known files, if you've done it correctly you should be able to see a JSON file at both of these URLs:
+
+- `https://<domain>/.well-known/matrix/server`
+- `https://<domain>/.well-known/matrix/client`
 
 You can also check if everything is configured correctly, by [checking if services work](maintenance-checking-services.md).

docs/installing.md

@@ -34,7 +34,7 @@ ansible-playbook -i inventory/hosts setup.yml --tags=start
 
 Now that the services are running, you might want to:
 
-- [create your first user account](registering-users.md)
+- **finalize the installation process** (required for federation to work!) by [Configuring Service Discovery via .well-known](configuring-well-known.md)
-- or **finalize the installation process** by [Configuring service discovery via .well-known](configuring-well-known.md)
+- or [create your first user account](registering-users.md)
-- or [Check if services work](maintenance-checking-services.md)
+- or [check if services work](maintenance-checking-services.md)
 - or learn how to [upgrade your services when new versions are released](maintenance-upgrading-services.md)

docs/prerequisites.md

@@ -6,11 +6,9 @@
 
 - the [Ansible](http://ansible.com/) program being installed on your own computer. It's used to run this playbook and configures your server for you. Take a look at [our guide about Ansible](ansible.md) for version requirements or alternative ways to run Ansible.
 
-- properly configured DNS SRV record for `<your-domain>` (details in [Configuring DNS](configuring-dns.md#configuring-dns) below)
+- an HTTPS-capable web server at the base domain name (`<your-domain>`) which is capable of serving static files
 
-- `matrix.<your-domain>` domain name pointing to your new server - this is where the Matrix Synapse server will live (details in [Configuring DNS](configuring-dns.md#configuring-dns) below)
+- properly configured DNS records for `<your-domain>` (details in [Configuring DNS](configuring-dns.md))
-
-- `riot.<your-domain>` domain name pointing to your new server - this is where the Riot web UI will live (details in [Configuring DNS](configuring-dns.md#configuring-dns) below)
 
 - some TCP/UDP ports open. This playbook configures the server's internal firewall for you. In most cases, you don't need to do anything special. But **if your server is running behind another firewall**, you'd need to open these ports: `80/tcp` (HTTP webserver), `443/tcp` (HTTPS webserver), `3478/tcp`  (STUN over TCP), `3478/udp` (STUN over UDP), `8448/tcp` (Matrix federation HTTPS webserver), `49152-49172/udp` (TURN over UDP).
 

group_vars/matrix-servers

@@ -161,7 +161,7 @@ matrix_mxisd_systemd_wanted_services_list: |
 #
 ######################################################################
 
-# By default, this playbook sets up a reverse-proxy nginx proxy server on port 80/443.
+# By default, this playbook sets up a reverse-proxy nginx proxy server on TCP ports 80, 443 and 8448.
 # This is fine if you're dedicating the whole server to Matrix.
 # If that's not the case, you may wish to disable this and take care of proxying yourself.
 matrix_nginx_proxy_enabled: true
@@ -181,6 +181,11 @@ matrix_nginx_proxy_proxy_matrix_identity_api_enabled: "{{ matrix_mxisd_enabled }
 matrix_nginx_proxy_proxy_matrix_identity_api_addr_with_container: "matrix-mxisd:8090"
 matrix_nginx_proxy_proxy_matrix_identity_api_addr_sans_container: "127.0.0.1:8090"
 
+# By default, we do TLS termination for the Matrix Federation API (port 8448) at matrix-nginx-proxy.
+matrix_nginx_proxy_proxy_matrix_federation_api_enabled: true
+matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container: "matrix-synapse:8048"
+matrix_nginx_proxy_proxy_matrix_federation_api_addr_sans_container: "127.0.0.1:8048"
+
 matrix_nginx_proxy_proxy_synapse_metrics: "{{ matrix_synapse_metrics_enabled }}"
 matrix_nginx_proxy_proxy_synapse_metrics_addr_with_container: "matrix-synapse:{{ matrix_synapse_metrics_port }}"
 matrix_nginx_proxy_proxy_synapse_metrics_addr_sans_container: "127.0.0.1:{{ matrix_synapse_metrics_port }}"
@@ -272,9 +277,13 @@ matrix_riot_web_default_is_url: "{{ matrix_identity_server_url }}"
 matrix_synapse_trusted_third_party_id_servers: "{{ [hostname_matrix] if matrix_mxisd_enabled else matrix_synapse_id_servers_public }}"
 
 # Normally, matrix-nginx-proxy is enabled and nginx can reach Synapse over the container network.
-# If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose
+# If matrix-nginx-proxy is not enabled, or you otherwise have a need for it, you can expose its ports
-# the Client/Server API's port to the local host (`127.0.0.1:8008`).
+# to the local host.
-matrix_synapse_container_expose_client_server_api_port: "{{ not matrix_nginx_proxy_enabled }}"
+#
+# For exposing the Matrix Client API's port (plain HTTP) to the local host (`127.0.0.1:8008`).
+matrix_synapse_container_expose_client_api_port: "{{ not matrix_nginx_proxy_enabled }}"
+# For exposing the Matrix Federation API's port (plain HTTP) to the local host (`127.0.0.1:8048`).
+matrix_synapse_container_expose_federation_api_port: "{{ not matrix_nginx_proxy_enabled }}"
 
 matrix_synapse_container_expose_metrics_port: "{{ not matrix_nginx_proxy_enabled }}"
 
@@ -283,6 +292,10 @@ matrix_synapse_database_user: "{{ matrix_postgres_connection_username }}"
 matrix_synapse_database_password: "{{ matrix_postgres_connection_password }}"
 matrix_synapse_database_database: "{{ matrix_postgres_db_name }}"
 
+# We do not enable TLS in Synapse by default.
+# TLS is handled by the matrix-nginx-proxy, which proxies the requests to Synapse.
+matrix_synapse_no_tls: true
+
 matrix_synapse_email_enabled: "{{ matrix_mailer_enabled }}"
 matrix_synapse_email_smtp_host: "matrix-mailer"
 matrix_synapse_email_smtp_port: 8025

roles/matrix-base/tasks/setup_well_known.yml

@@ -12,10 +12,13 @@
   with_items:
     - "{{ matrix_static_files_base_path }}/.well-known/matrix"
 
-- name: Ensure Matrix /.well-known/matrix/client configured
+- name: Ensure Matrix /.well-known/matrix files configured
   template:
-    src: "{{ role_path }}/templates/static-files/well-known/matrix-client.j2"
+    src: "{{ role_path }}/templates/static-files/well-known/matrix-{{ item }}.j2"
-    dest: "{{ matrix_static_files_base_path }}/.well-known/matrix/client"
+    dest: "{{ matrix_static_files_base_path }}/.well-known/matrix/{{ item }}"
     mode: 0644
     owner: "{{ matrix_user_username }}"
     group: "{{ matrix_user_username }}"
+  with_items:
+    - "client"
+    - "server"

roles/matrix-base/templates/static-files/well-known/matrix-server.j2

@@ -0,0 +1,3 @@
+{
+	"m.server": "{{ hostname_matrix }}:8448"
+}

roles/matrix-corporal/tasks/setup_corporal.yml

@@ -36,6 +36,7 @@
     mode: 0644
   when: "matrix_corporal_enabled"
 
+
 #
 # Tasks related to getting rid of matrix-corporal (if it was previously enabled)
 #

roles/matrix-nginx-proxy/defaults/main.yml

@@ -57,8 +57,14 @@ matrix_nginx_proxy_proxy_matrix_client_api_addr_sans_container: "127.0.0.1:8008"
 # This needs to be equal or higher than the maximum upload size accepted by Synapse.
 matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb: 25
 
+# Controls whether proxying for the Matrix Federation API should be done.
+matrix_nginx_proxy_proxy_matrix_federation_api_enabled: false
+matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container: "matrix-synapse:8048"
+matrix_nginx_proxy_proxy_matrix_federation_api_addr_sans_container: "localhost:8048"
+matrix_nginx_proxy_proxy_matrix_federation_api_client_max_body_size_mb: "{{ matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb * 3 }}"
+
 # The tmpfs at /tmp needs to be large enough to handle multiple concurrent file uploads.
-matrix_nginx_proxy_tmp_directory_size_mb: "{{ matrix_nginx_proxy_proxy_matrix_client_api_client_max_body_size_mb * 50 }}"
+matrix_nginx_proxy_tmp_directory_size_mb: "{{ matrix_nginx_proxy_proxy_matrix_federation_api_client_max_body_size_mb * 50 }}"
 
 # A list of strings containing additional configuration blocks to add to the matrix domain's server configuration.
 matrix_nginx_proxy_proxy_matrix_additional_server_configuration_blocks: []

roles/matrix-nginx-proxy/tasks/self_check_well_known.yml

@@ -1,65 +1,13 @@
 ---
 
-- set_fact:
+- name: Perform well-known checks
-    well_known_url_matrix: "https://{{ hostname_matrix }}/.well-known/matrix/client"
+  include_tasks: "{{ role_path }}/tasks/self_check_well_known_file.yml"
-    well_known_url_identity: "https://{{ hostname_identity }}/.well-known/matrix/client"
+  with_items:
-
+    - path: /.well-known/matrix/server
-# These well-known files may be served without a `Content-Type: application/json` header,
+      purpose: Server Discovery
-# so we can't rely on the uri module's automatic parsing of JSON.
+      cors: false
-- name: Check .well-known on the matrix hostname
+    - path: /.well-known/matrix/client
-  uri:
+      purpose: Client Discovery
-    url: "{{ well_known_url_matrix }}"
+      cors: true
-    follow_redirects: false
+  loop_control:
-    return_content: true
+    loop_var: well_known_file_check
-  register: result_well_known_matrix
-  ignore_errors: true
-
-- name: Fail if .well-known not working on the matrix hostname
-  fail:
-    msg: "Failed checking that well-known is configured at `{{ hostname_matrix }}` (checked endpoint: `{{ well_known_url_matrix }}`). Is port 443 open in your firewall? Full error: {{ result_well_known_matrix }}"
-  when: "result_well_known_matrix.failed"
-
-- name: Parse JSON for well-known payload at the matrix hostname
-  set_fact:
-    well_known_matrix_payload: "{{ result_well_known_matrix.content|from_json }}"
-
-- name: Fail if .well-known not CORS-aware on the matrix hostname
-  fail:
-    msg: "Well-known serving for `{{ hostname_matrix }}` (checked endpoint: `{{ well_known_url_matrix }}`) is not CORS-aware. The file needs to be served with an Access-Control-Allow-Origin header set."
-  when: "'access_control_allow_origin' not in result_well_known_matrix"
-
-- name: Report working .well-known on the matrix hostname
-  debug:
-    msg: "well-known is configured correctly for `{{ hostname_matrix }}` (checked endpoint: `{{ well_known_url_matrix }}`)"
-
-- name: Check .well-known on the identity hostname
-  uri:
-    url: "{{ well_known_url_identity }}"
-    follow_redirects: false
-    return_content: true
-  register: result_well_known_identity
-  ignore_errors: true
-
-- name: Fail if .well-known not working on the identity hostname
-  fail:
-    msg: "Failed checking that well-known is configured at `{{ hostname_identity }}` (checked endpoint: `{{ well_known_url_identity }}`). Is port 443 open in your firewall? Full error: {{ result_well_known_identity }}"
-  when: "result_well_known_identity.failed"
-
-- name: Parse JSON for well-known payload at the identity hostname
-  set_fact:
-    well_known_identity_payload: "{{ result_well_known_identity.content|from_json }}"
-
-- name: Fail if .well-known not CORS-aware on the identity hostname
-  fail:
-    msg: "Well-known serving for `{{ hostname_identity }}` (checked endpoint: `{{ well_known_url_identity }}`) is not CORS-aware. The file needs to be served with an Access-Control-Allow-Origin header set. See docs/configuring-well-known.md"
-  when: "'access_control_allow_origin' not in result_well_known_identity"
-
-# For people who manually copy the well-known file, try to detect if it's outdated
-- name: Fail if well-known is different on matrix hostname and identity hostname
-  fail:
-    msg: "The well-known files at `{{ hostname_matrix }}` and `{{ hostname_identity }}` are different. Perhaps you copied the file manually before and now it's outdated?"
-  when: "well_known_matrix_payload != well_known_identity_payload"
-
-- name: Report working .well-known on the identity hostname
-  debug:
-    msg: "well-known is configured correctly for `{{ hostname_identity }}` (checked endpoint: `{{ well_known_url_identity }}`)"

roles/matrix-nginx-proxy/tasks/self_check_well_known_file.yml

@@ -0,0 +1,65 @@
+---
+
+- set_fact:
+    well_known_url_matrix: "https://{{ hostname_matrix }}{{ well_known_file_check.path }}"
+    well_known_url_identity: "https://{{ hostname_identity }}{{ well_known_file_check.path }}"
+
+# These well-known files may be served without a `Content-Type: application/json` header,
+# so we can't rely on the uri module's automatic parsing of JSON.
+- name: Check .well-known on the matrix hostname
+  uri:
+    url: "{{ well_known_url_matrix }}"
+    follow_redirects: false
+    return_content: true
+  register: result_well_known_matrix
+  ignore_errors: true
+
+- name: Fail if .well-known not working on the matrix hostname
+  fail:
+    msg: "Failed checking that the well-known file for {{ well_known_file_check.purpose }} is configured at `{{ hostname_matrix }}` (checked endpoint: `{{ well_known_url_matrix }}`). Is port 443 open in your firewall? Full error: {{ result_well_known_matrix }}"
+  when: "result_well_known_matrix.failed"
+
+- name: Parse JSON for well-known payload at the matrix hostname
+  set_fact:
+    well_known_matrix_payload: "{{ result_well_known_matrix.content|from_json }}"
+
+- name: Fail if .well-known not CORS-aware on the matrix hostname
+  fail:
+    msg: "The well-known file for {{ well_known_file_check.purpose }} on `{{ hostname_matrix }}` (checked endpoint: `{{ well_known_url_matrix }}`) is not CORS-aware. The file needs to be served with an Access-Control-Allow-Origin header set."
+  when: "well_known_file_check.cors and 'access_control_allow_origin' not in result_well_known_matrix"
+
+- name: Report working .well-known on the matrix hostname
+  debug:
+    msg: "well-known for {{ well_known_file_check.purpose }} is configured correctly for `{{ hostname_matrix }}` (checked endpoint: `{{ well_known_url_matrix }}`)"
+
+- name: Check .well-known on the identity hostname
+  uri:
+    url: "{{ well_known_url_identity }}"
+    follow_redirects: false
+    return_content: true
+  register: result_well_known_identity
+  ignore_errors: true
+
+- name: Fail if .well-known not working on the identity hostname
+  fail:
+    msg: "Failed checking that the well-known file for {{ well_known_file_check.purpose }} is configured at `{{ hostname_identity }}` (checked endpoint: `{{ well_known_url_identity }}`). Is port 443 open in your firewall? Full error: {{ result_well_known_identity }}"
+  when: "result_well_known_identity.failed"
+
+- name: Parse JSON for well-known payload at the identity hostname
+  set_fact:
+    well_known_identity_payload: "{{ result_well_known_identity.content|from_json }}"
+
+- name: Fail if .well-known not CORS-aware on the identity hostname
+  fail:
+    msg: "The well-known file for {{ well_known_file_check.purpose }} on `{{ hostname_identity }}` (checked endpoint: `{{ well_known_url_identity }}`) is not CORS-aware. The file needs to be served with an Access-Control-Allow-Origin header set. See docs/configuring-well-known.md"
+  when: "well_known_file_check.cors and 'access_control_allow_origin' not in result_well_known_identity"
+
+# For people who manually copy the well-known file, try to detect if it's outdated
+- name: Fail if well-known is different on matrix hostname and identity hostname
+  fail:
+    msg: "The well-known files for {{ well_known_file_check.purpose }} at `{{ hostname_matrix }}` and `{{ hostname_identity }}` are different. Perhaps you copied the file ({{ well_known_file_check.path }}) manually before and now it's outdated?"
+  when: "well_known_matrix_payload != well_known_identity_payload"
+
+- name: Report working .well-known on the identity hostname
+  debug:
+    msg: "well-known for {{ well_known_file_check.purpose }} ({{ well_known_file_check.path }}) is configured correctly for `{{ hostname_identity }}` (checked endpoint: `{{ well_known_url_identity }}`)"

roles/matrix-nginx-proxy/templates/nginx/conf.d/matrix-synapse.conf.j2

@@ -39,7 +39,7 @@ server {
 	ssl_prefer_server_ciphers on;
 	ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
 
-	location /.well-known/matrix/client {
+	location /.well-known/matrix {
 		root {{ matrix_static_files_base_path }};
 		expires 1m;
 		default_type application/json;
@@ -101,6 +101,10 @@ server {
 		{{- configuration_block }}
 	{% endfor %}
 
+	{#
+		This handles the Matrix Client API only.
+		The Matrix Federation API is handled by a separate vhost.
+	#}
 	location /_matrix {
 		{% if matrix_nginx_proxy_enabled %}
 			{# Use the embedded DNS resolver in Docker containers to discover the service #}
@@ -146,3 +150,43 @@ server {
 		rewrite ^/$ /_matrix/static/ last;
 	}
 }
+
+{% if matrix_nginx_proxy_proxy_matrix_federation_api_enabled %}
+server {
+	listen 8448 ssl http2;
+	listen [::]:8448 ssl http2;
+
+	server_name {{ matrix_nginx_proxy_proxy_matrix_hostname }};
+
+	server_tokens off;
+	root /dev/null;
+
+	gzip on;
+	gzip_types text/plain application/json;
+
+	ssl_certificate {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_matrix_hostname }}/fullchain.pem;
+	ssl_certificate_key {{ matrix_ssl_config_dir_path }}/live/{{ matrix_nginx_proxy_proxy_matrix_hostname }}/privkey.pem;
+	ssl_protocols {{ matrix_nginx_proxy_ssl_protocols }};
+	ssl_prefer_server_ciphers on;
+	ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
+
+	location / {
+		{% if matrix_nginx_proxy_enabled %}
+			{# Use the embedded DNS resolver in Docker containers to discover the service #}
+			resolver 127.0.0.11 valid=5s;
+			set $backend "{{ matrix_nginx_proxy_proxy_matrix_federation_api_addr_with_container }}";
+			proxy_pass http://$backend;
+		{% else %}
+			{# Generic configuration for use outside of our container setup #}
+			proxy_pass http://{{ matrix_nginx_proxy_proxy_matrix_federation_api_addr_sans_container }};
+		{% endif %}
+
+		proxy_set_header Host $host;
+		proxy_set_header X-Forwarded-For $remote_addr;
+
+		client_body_buffer_size 25M;
+		client_max_body_size {{ matrix_nginx_proxy_proxy_matrix_federation_api_client_max_body_size_mb }}M;
+		proxy_max_temp_file_size 0;
+	}
+}
+{% endif %}

roles/matrix-nginx-proxy/templates/systemd/matrix-nginx-proxy.service.j2

@@ -22,6 +22,9 @@ ExecStart=/usr/bin/docker run --rm --name matrix-nginx-proxy \
 			--network={{ matrix_docker_network }} \
 			-p 80:8080 \
 			-p 443:8443 \
+			{% if matrix_nginx_proxy_proxy_matrix_federation_api_enabled %}
+			-p 8448:8448 \
+			{% endif %}
 			-v {{ matrix_nginx_proxy_data_path }}/nginx.conf:/etc/nginx/nginx.conf:ro \
 			-v {{ matrix_nginx_proxy_confd_path }}:/etc/nginx/conf.d:ro \
 			-v {{ matrix_ssl_config_dir_path }}:{{ matrix_ssl_config_dir_path }}:ro \

roles/matrix-synapse/defaults/main.yml

@@ -8,7 +8,12 @@ matrix_synapse_media_store_path: "{{ matrix_synapse_storage_path }}/media-store"
 matrix_synapse_ext_path: "{{ matrix_synapse_base_path }}/ext"
 
 # Controls whether the Synapse container exposes the Client/Server API port (tcp/8008).
-matrix_synapse_container_expose_client_server_api_port: false
+matrix_synapse_container_expose_client_api_port: false
+
+# Controls whether the Synapse container exposes the Server/Server (Federation) API port (tcp/8048).
+# This is for the plain HTTP API. If you need Synapse to handle TLS encryption,
+# that would be on another port (tcp/8448) controlled by `matrix_synapse_tls_federation_listener_enabled`.
+matrix_synapse_container_expose_federation_api_port: false
 
 # Controls whether the Appservice IRC container exposes the Client/Server API port (tcp/9999).
 matrix_appservice_irc_container_expose_client_server_api_port: false
@@ -60,6 +65,17 @@ matrix_synapse_root_log_level: "INFO"
 matrix_synapse_rc_messages_per_second: 0.2
 matrix_synapse_rc_message_burst_count: 10.0
 
+# If you're serving Synapse behind an HTTPS-capable reverse-proxy,
+# you can disable TLS completely (`matrix_synapse_no_tls: true`).
+# Otherwise, you would need to provide certificate files to it.
+matrix_synapse_no_tls: false
+# Controls whether the TLS federation listener is enabled (tcp/8448).
+# Note that federation may potentially be enabled on tcp/8008 as well.
+# Only makes sense if federation is not disabled (`matrix_synapse_federation_enabled`).
+matrix_synapse_tls_federation_listener_enabled: "{{ not matrix_synapse_no_tls }}"
+matrix_synapse_tls_certificate_path: "/data/{{ hostname_matrix }}.tls.crt"
+matrix_synapse_tls_private_key_path: "/data/{{ hostname_matrix }}.tls.key"
+
 # Enable this to allow Synapse to report utilization statistics about your server to matrix.org
 # (things like number of users, number of messages sent, uptime, load, etc.)
 matrix_synapse_report_stats: false
@@ -95,6 +111,8 @@ matrix_synapse_cache_factor: 0.5
 
 # Controls whether Matrix Synapse will federate at all.
 # Disable this to completely isolate your server from the rest of the Matrix network.
+# Also see: `matrix_synapse_tls_federation_listener_enabled` if you wish to keep federation enabled,
+# but want to stop the TLS listener (port 8448).
 matrix_synapse_federation_enabled: true
 
 # A list of domain names that are allowed to federate with the given Matrix Synapse server.

roles/matrix-synapse/tasks/setup_synapse_main.yml

@@ -58,6 +58,61 @@
     dest: "{{ matrix_synapse_config_dir_path }}/{{ hostname_matrix }}.log.config"
     mode: 0644
 
+#
+# To make Synapse 0.99 happy, we need to generate a valid (self-signed is OK) certificate file that we provide to it.
+# It won't be used for anything important, but it needs to be there.
+# See https://github.com/matrix-org/synapse/issues/4554
+#
+# Previously, Synapse would generate such certificate files and actually use them.
+# So existing installations already have them.
+#
+
+- name: Check if Synapse certificate exists
+  stat:
+    path: "{{ matrix_synapse_config_dir_path }}/{{ hostname_matrix }}.tls.crt"
+  register: matrix_synapse_certificate_stat
+
+- name: Ensure OpenSSL installed (RedHat)
+  yum:
+    name:
+      - openssl
+    state: present
+    update_cache: no
+  when: "not matrix_synapse_certificate_stat.stat.exists and ansible_os_family == 'RedHat'"
+
+- name: Ensure OpenSSL installed (Debian)
+  apt:
+    name:
+      - openssl
+    state: present
+    update_cache: no
+  when: "not matrix_synapse_certificate_stat.stat.exists and ansible_os_family == 'Debian'"
+
+# The proper way to do this is by using a sequence of
+# `openssl_privatekey`, `openssl_csr` and `openssl_certificate`.
+#
+# Unfortunately, `openssl_csr` and `openssl_certificate` require `PyOpenSSL>=0.15` to work,
+# which is not available on CentOS 7 (at least).
+#
+# We'll do it in a more manual way.
+- name: Generate SSL certificate
+  command: |
+    openssl req -x509 \
+    -sha256 \
+    -newkey rsa:4096 \
+    -nodes \
+    -subj "/CN={{ hostname_matrix }}" \
+    -keyout {{ matrix_synapse_config_dir_path }}/{{ hostname_matrix }}.tls.key \
+    -out {{ matrix_synapse_config_dir_path }}/{{ hostname_matrix }}.tls.crt \
+    -days 3650
+  become: true
+  become_user: "{{ matrix_user_username }}"
+  when: "not matrix_synapse_certificate_stat.stat.exists"
+
+#
+# End of tasks related to making Synapse 0.99 happy.
+#
+
 - name: Ensure matrix-synapse.service installed
   template:
     src: "{{ role_path }}/templates/synapse/systemd/matrix-synapse.service.j2"

roles/matrix-synapse/tasks/validate_config.yml

@@ -7,3 +7,13 @@
   when: "vars[item] == ''"
   with_items:
     - "matrix_synapse_macaroon_secret_key"
+
+- name: (Deprecation) Catch and report renamed settings
+  fail:
+    msg: >
+      Your configuration contains a variable, which now has a different name.
+      Please change your configuration to rename the variable (`{{ item.old }}` -> `{{ item.new }}`).
+  when: "item.old in vars"
+  with_items:
+    - {'old': 'matrix_synapse_container_expose_api_port', 'new': 'matrix_synapse_container_expose_client_api_port'}
+

roles/matrix-synapse/templates/synapse/homeserver.yaml.j2

@@ -1,19 +1,70 @@
 # vim:ft=yaml
-# PEM encoded X509 certificate for TLS.
+# PEM-encoded X509 certificate for TLS.
-# You can replace the self-signed certificate that synapse
+# This certificate, as of Synapse 1.0, will need to be a valid and verifiable
-# autogenerates on launch with your own SSL certificate + key pair
+# certificate, signed by a recognised Certificate Authority.
-# if you like.  Any required intermediary certificates can be
+#
-# appended after the primary certificate in hierarchical order.
+# See 'ACME support' below to enable auto-provisioning this certificate via
-tls_certificate_path: "/data/{{ hostname_matrix }}.tls.crt"
+# Let's Encrypt.
+#
+tls_certificate_path: "{{ matrix_synapse_tls_certificate_path }}"
 
-# PEM encoded private key for TLS
+# PEM-encoded private key for TLS
-tls_private_key_path: "/data/{{ hostname_matrix }}.tls.key"
+tls_private_key_path: "{{ matrix_synapse_tls_private_key_path }}"
 
-# PEM dh parameters for ephemeral keys
+# ACME support: This will configure Synapse to request a valid TLS certificate
-tls_dh_params_path: "/data/{{ hostname_matrix }}.tls.dh"
+# for your configured `server_name` via Let's Encrypt.
+#
+# Note that provisioning a certificate in this way requires port 80 to be
+# routed to Synapse so that it can complete the http-01 ACME challenge.
+# By default, if you enable ACME support, Synapse will attempt to listen on
+# port 80 for incoming http-01 challenges - however, this will likely fail
+# with 'Permission denied' or a similar error.
+#
+# There are a couple of potential solutions to this:
+#
+#  * If you already have an Apache, Nginx, or similar listening on port 80,
+#    you can configure Synapse to use an alternate port, and have your web
+#    server forward the requests. For example, assuming you set 'port: 8009'
+#    below, on Apache, you would write:
+#
+#    ProxyPass /.well-known/acme-challenge http://localhost:8009/.well-known/acme-challenge
+#
+#  * Alternatively, you can use something like `authbind` to give Synapse
+#    permission to listen on port 80.
+#
+acme:
+    # ACME support is disabled by default. Uncomment the following line
+    # to enable it.
+    #
+    # enabled: true
 
-# Don't bind to the https port
+    # Endpoint to use to request certificates. If you only want to test,
-no_tls: False
+    # use Let's Encrypt's staging url:
+    #     https://acme-staging.api.letsencrypt.org/directory
+    #
+    # url: https://acme-v01.api.letsencrypt.org/directory
+
+    # Port number to listen on for the HTTP-01 challenge. Change this if
+    # you are forwarding connections through Apache/Nginx/etc.
+    #
+    # port: 80
+
+    # Local addresses to listen on for incoming connections.
+    # Again, you may want to change this if you are forwarding connections
+    # through Apache/Nginx/etc.
+    #
+    # bind_addresses: ['::', '0.0.0.0']
+
+    # How many days remaining on a certificate before it is renewed.
+    #
+    # reprovision_threshold: 30
+
+# If your server runs behind a reverse-proxy which terminates TLS connections
+# (for both client and federation connections), it may be useful to disable
+# All TLS support for incoming connections. Setting no_tls to True will
+# do so (and avoid the need to give synapse a TLS private key).
+#
+no_tls: {{ matrix_synapse_no_tls|to_json }}
 
 # List of allowed TLS fingerprints for this server to publish along
 # with the signing keys for this server. Other matrix servers that
@@ -133,7 +184,8 @@ listeners:
     bind_addresses:
       - '0.0.0.0'
 {% endif %}
-{% if matrix_synapse_federation_enabled %}
+
+{% if matrix_synapse_federation_enabled and matrix_synapse_tls_federation_listener_enabled %}
   # Main HTTPS listener
   # For when matrix traffic is sent directly to synapse.
   -
@@ -168,7 +220,7 @@ listeners:
     #     config: {}
 {% endif %}
 
-  # Unsecure HTTP listener,
+  # Unsecure HTTP listener for the Client API,
   # For when matrix traffic passes through loadbalancer that unwraps TLS.
   - port: 8008
     tls: false
@@ -181,6 +233,21 @@ listeners:
       - names: [client]
         compress: false
 
+{% if matrix_synapse_federation_enabled %}
+  # Unsecure HTTP listener for the Federation API,
+  # For when matrix traffic passes through loadbalancer that unwraps TLS.
+  - port: 8048
+    tls: false
+    bind_addresses: ['::']
+    type: http
+
+    x_forwarded: true
+
+    resources:
+      - names: [federation]
+        compress: false
+{% endif %}
+
   # Turn on the twisted ssh manhole service on localhost on the given
   # port.
   # - port: 9000

roles/matrix-synapse/templates/synapse/systemd/matrix-synapse.service.j2

@@ -28,11 +28,14 @@ ExecStart=/usr/bin/docker run --rm --name matrix-synapse \
 			--tmpfs=/tmp:rw,noexec,nosuid,size={{ matrix_synapse_tmp_directory_size_mb }}m \
 			--network={{ matrix_docker_network }} \
 			-e SYNAPSE_CACHE_FACTOR={{ matrix_synapse_cache_factor }} \
-			{% if matrix_synapse_federation_enabled %}
+			{% if matrix_synapse_container_expose_client_api_port %}
+			-p 127.0.0.1:8008:8008 \
+			{% endif %}
+			{% if matrix_synapse_federation_enabled and matrix_synapse_tls_federation_listener_enabled %}
 			-p 8448:8448 \
 			{% endif %}
-			{% if matrix_synapse_container_expose_client_server_api_port %}
+			{% if matrix_synapse_federation_enabled and matrix_synapse_container_expose_federation_api_port %}
-			-p 127.0.0.1:8008:8008 \
+			-p 127.0.0.1:8048:8048 \
 			{% endif %}
 			{% if matrix_synapse_container_expose_metrics_port %}
 			-p 127.0.0.1:{{ matrix_synapse_metrics_port }}:{{ matrix_synapse_metrics_port }} \