Completely redo how mxisd configuration gets generated

This change is provoked by a few different things:

- #54 (Github Pull Request), which rightfully says that we need a
way to support ALL mxisd configuration options easily

- the upcoming mxisd 1.3.0 release, which drops support for
property-style configuration (dot-notation), forcing us to
redo the way we generate the configuration file

With this, mxisd is much more easily configurable now
and much more easily maintaneable by us in the future
(no need to introduce additional playbook variables and logic).

CHANGELOG.md

@@ -1,3 +1,43 @@
+# 2019-01-11
+
+## (BC Break) mxisd configuration changes
+
+To be more flexible and to support the upcoming [mxisd](https://github.com/kamax-io/mxisd) 1.3.0 (when it gets released),
+we've had to redo how mxisd gets configured.
+
+The following variables are no longer supported by this playbook:
+
+- `matrix_mxisd_ldap_enabled`
+- `matrix_mxisd_ldap_connection_host`
+- `matrix_mxisd_ldap_connection_tls`
+- `matrix_mxisd_ldap_connection_port`
+- `matrix_mxisd_ldap_connection_baseDn`
+- `matrix_mxisd_ldap_connection_baseDns`
+- `matrix_mxisd_ldap_connection_bindDn`
+- `matrix_mxisd_ldap_connection_bindDn`
+- `matrix_mxisd_ldap_connection_bindPassword`
+- `matrix_mxisd_ldap_filter`
+- `matrix_mxisd_ldap_attribute_uid_type`
+- `matrix_mxisd_ldap_attribute_uid_value`
+- `matrix_mxisd_ldap_connection_bindPassword`
+- `matrix_mxisd_ldap_attribute_name`
+- `matrix_mxisd_ldap_attribute_threepid_email`
+- `matrix_mxisd_ldap_attribute_threepid_msisdn`
+- `matrix_mxisd_ldap_identity_filter`
+- `matrix_mxisd_ldap_identity_medium`
+- `matrix_mxisd_ldap_auth_filter`
+- `matrix_mxisd_ldap_directory_filter`
+- `matrix_mxisd_template_config`
+
+You are encouraged to use the `matrix_mxisd_configuration_extension_yaml` variable to define your own mxisd configuration additions and overrides.
+Refer to the [default variables file](roles/matrix-server/defaults/main.yml) for more information.
+
+This new way of configuring mxisd is beneficial because:
+
+- it lets us support all mxisd configuration options, as the playbook simply forwards them to mxisd without needing to care or understand them
+- it lets you upgrade to newer mxisd versions and make use of their features, without us having to add support for them explicitly
+
+
 # 2019-01-08
 
 ## (BC Break) Cronjob schedule no longer configurable

docs/configuring-playbook-mxisd.md

@@ -22,10 +22,14 @@ matrix_mxisd_matrixorg_forwarding_enabled: true
 
 What this playbook configures for your is some bare minimum Identity Server functionality, so that you won't need to rely on external 3rd party services.
 
-Still, mxisd can do much more.
+A few variables can be toggled in this playbook to alter the mxisd configuration that gets generated.
-You can refer to the [mxisd website](https://github.com/kamax-io/mxisd) for more details.
 
-You can override the `matrix_mxisd_template_config` variable and use your own custom configuration template.
+Still, mxisd can do much more.
+You can refer to the [mxisd website](https://github.com/kamax-io/mxisd) for more details and configuration options.
+
+To use a more custom configuration, you can define a `matrix_mxisd_configuration_extension_yaml` string variable
+and put your configuration in it.
+To learn more about how to do this, refer to the information about `matrix_mxisd_configuration_extension_yaml` in the [default variables file](../roles/matrix-server/defaults/main.yml).
 
 
 ## Troubleshooting

roles/matrix-server/defaults/main.yml

@@ -239,44 +239,73 @@ matrix_mxisd_data_path: "{{ matrix_mxisd_base_path }}/data"
 matrix_mxisd_matrixorg_forwarding_enabled: false
 
 # mxisd has serveral supported identity stores.
-# One of them is storing identities directly in Synapse's database.
+# One of them (which we enable by default) is storing identities directly in Synapse's database.
 # Learn more here: https://github.com/kamax-matrix/mxisd/blob/master/docs/stores/synapse.md
+#
+# If you need to disable this in favor of some other store, you can toggle it to disabled here
+# and add your own mxisd configuration for the other store in `matrix_mxisd_configuration_extension_yaml`.
 matrix_mxisd_synapsesql_enabled: true
 matrix_mxisd_synapsesql_type: postgresql
 matrix_mxisd_synapsesql_connection: //{{ matrix_postgres_connection_hostname }}/{{ matrix_postgres_db_name }}?user={{ matrix_postgres_connection_username }}&password={{ matrix_postgres_connection_password }}
 
-# LDAP is another identity store that's supported by mxisd.
+# Default mxisd configuration template which covers the generic use case.
-# Learn more here: https://github.com/kamax-matrix/mxisd/blob/master/docs/stores/ldap.md
+# You can customize it by controlling the various variables inside it.
-matrix_mxisd_ldap_enabled: false
+#
-matrix_mxisd_ldap_connection_host: ldapHostnameOrIp
+# For a more advanced customization, you can extend the default (see `matrix_mxisd_configuration_extension_yaml`)
-matrix_mxisd_ldap_connection_tls: false
+# or completely replace this variable with your own template.
-matrix_mxisd_ldap_connection_port: 389
+matrix_mxisd_configuration_yaml: |
-matrix_mxisd_ldap_connection_baseDns: ['OU=Users,DC=example,DC=org']
+  matrix:
-matrix_mxisd_ldap_connection_bindDn: CN=My Mxisd User,OU=Users,DC=example,DC=org
+    domain: {{ hostname_identity }}
-matrix_mxisd_ldap_connection_bindPassword: TheUserPassword
-# The following keys are optional:
-# matrix_mxisd_ldap_filter: ""
-# matrix_mxisd_ldap_attribute_uid_type: uid
-# matrix_mxisd_ldap_attribute_uid_value: sAMAccountName
-# matrix_mxisd_ldap_attribute_name: cn
-# matrix_mxisd_ldap_attribute_threepid_email:
-#   - mail
-#   - otherMailAttribute
-# matrix_mxisd_ldap_attribute_threepid_msisdn:
-#   - phone
-#   - otherPhoneAttribute
-# matrix_mxisd_ldap_identity_filter: ""
-# matrix_mxisd_ldap_identity_medium: ""
-# matrix_mxisd_ldap_auth_filter: ""
-# matrix_mxisd_ldap_directory_filter: ""
 
+  server:
+    name: {{ hostname_matrix }}
 
-# Specifies which template files to use when configuring mxisd.
+  key:
-# If you'd like to have your own different configuration, feel free to copy and paste
+    path: /var/mxisd/sign.key
-# the original files into your inventory (e.g. in `inventory/host_vars/<host>/`)
+
-# and then change the specific host's `vars.yaml` file like this:
+  storage:
-# matrix_mxisd_template_config: "{{ playbook_dir }}/inventory/host_vars/<host>/mxisd.yaml.j2"
+    provider:
-matrix_mxisd_template_config: "{{ role_path }}/templates/mxisd/mxisd.yaml.j2"
+      sqlite:
+        database: /var/mxisd/mxisd.db
+
+  {% if matrix_mxisd_matrixorg_forwarding_enabled %}
+  forward:
+    servers: ['matrix-org']
+  {% endif %}
+
+  synapseSql:
+    enabled: {{ matrix_mxisd_synapsesql_enabled }}
+    type: {{ matrix_mxisd_synapsesql_type }}
+    connection: {{ matrix_mxisd_synapsesql_connection }}
+
+matrix_mxisd_configuration_extension_yaml: |
+  # Your custom YAML configuration for mxisd goes here.
+  # This configuration extends the default starting configuration (`matrix_mxisd_configuration_yaml`).
+  #
+  # You can override individual variables from the default configuration, or introduce new ones.
+  #
+  # If you need something more special, you can take full control by
+  # completely redefining `matrix_mxisd_configuration_yaml`.
+  #
+  # Example configuration extension follows:
+  #
+  # ldap:
+  #   enabled: true
+  #   connection:
+  #     host: ldapHostnameOrIp
+  #     tls: false
+  #     port: 389
+  #     baseDns: ['OU=Users,DC=example,DC=org']
+  #     bindDn: CN=My Mxisd User,OU=Users,DC=example,DC=org
+  #     bindPassword: TheUserPassword
+
+# Doing `|from_yaml` when the extension contains nothing yields an empty string ("").
+# We need to ensure it's a dictionary or `|combine` (when building `matrix_mxisd_configuration`) will fail later.
+matrix_mxisd_configuration_extension: "{{ matrix_mxisd_configuration_extension_yaml|from_yaml if matrix_mxisd_configuration_extension_yaml|from_yaml else {} }}"
+
+# Holds the final mxisd configuration (a combination of the default and its extension).
+# You most likely don't need to touch this variable. Instead, see `matrix_mxisd_configuration_yaml`.
+matrix_mxisd_configuration: "{{ matrix_mxisd_configuration_yaml|from_yaml|combine(matrix_mxisd_configuration_extension, recursive=True) }}"
 
 
 # Enable this to add support for matrix-corporal.

roles/matrix-server/tasks/setup/setup_mxisd.yml

@@ -4,14 +4,47 @@
 # Tasks related to setting up mxisd
 #
 
-- name: (Deprecation) Fail if using outdated configuration
+- name: (Deprecation) Warn about mxisd variables that are not used anymore
   fail:
-    msg: "You're using the `matrix_mxisd_ldap_connection_baseDn` variable (single string), which has been superseded by `matrix_mxisd_ldap_connection_baseDns` (array of strings). See https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/CHANGELOG.md#bc-break-mxisd-upgrade-with-multiple-base-dn-support"
+    msg: >
-  when: "matrix_mxisd_ldap_connection_baseDn is defined"
+      The `{{ item }}` variable defined in your configuration is not used by this playbook anymore!
+      You'll need to adapt to the new way of extending mxisd configuration.
+      See the CHANGELOG and the `matrix_mxisd_configuration_extension_yaml` variable for more information and examples.
+  when: "matrix_mxisd_enabled and item in vars"
+  with_items:
+    - 'matrix_mxisd_ldap_enabled'
+    - 'matrix_mxisd_ldap_connection_host'
+    - 'matrix_mxisd_ldap_connection_tls'
+    - 'matrix_mxisd_ldap_connection_port'
+    - 'matrix_mxisd_ldap_connection_baseDn'
+    - 'matrix_mxisd_ldap_connection_baseDns'
+    - 'matrix_mxisd_ldap_connection_bindDn'
+    - 'matrix_mxisd_ldap_connection_bindPassword'
+    - 'matrix_mxisd_ldap_filter'
+    - 'matrix_mxisd_ldap_attribute_uid_type'
+    - 'matrix_mxisd_ldap_attribute_uid_value'
+    - 'matrix_mxisd_ldap_connection_bindPassword'
+    - 'matrix_mxisd_ldap_attribute_name'
+    - 'matrix_mxisd_ldap_attribute_threepid_email'
+    - 'matrix_mxisd_ldap_attribute_threepid_msisdn'
+    - 'matrix_mxisd_ldap_identity_filter'
+    - 'matrix_mxisd_ldap_identity_medium'
+    - 'matrix_mxisd_ldap_auth_filter'
+    - 'matrix_mxisd_ldap_directory_filter'
+    - 'matrix_mxisd_template_config'
+
+- name: Ensure mxisd configuration does not contain any dot-notation keys
+  fail:
+    msg: >
+      Since version 1.3.0, mxisd will not accept property-style configuration keys.
+      You have defined a key (`{{ item.key }}`) which contains a dot.
+      Instead, use nesting. See: https://github.com/kamax-matrix/mxisd/wiki/Upgrade#v130
+  when: "matrix_mxisd_enabled and '.' in item.key"
+  with_dict: "{{ matrix_mxisd_configuration }}"
 
 - name: Fail if mailer is not enabled
   fail:
-    msg: "You need to enable the mailer service (matrix_mailer_enabled) to install mxisd"
+    msg: "You need to enable the mailer service (`matrix_mailer_enabled`) to install mxisd"
   when: "matrix_mxisd_enabled and not matrix_mailer_enabled"
 
 - name: Ensure mxisd paths exist
@@ -32,8 +65,8 @@
   when: matrix_mxisd_enabled
 
 - name: Ensure mxisd config installed
-  template:
+  copy:
-    src: "{{ matrix_mxisd_template_config }}"
+    content: "{{ matrix_mxisd_configuration|to_nice_yaml }}"
     dest: "{{ matrix_mxisd_config_path }}/mxisd.yaml"
     mode: 0644
     owner: "{{ matrix_user_username }}"

roles/matrix-server/templates/mxisd/mxisd.yaml.j2

@@ -1,69 +0,0 @@
-matrix.domain: {{ hostname_identity }}
-server.name: {{ hostname_matrix }}
-
-key.path: /var/mxisd/sign.key
-
-storage.provider.sqlite.database: /var/mxisd/mxisd.db
-
-threepid.medium.email.identity.from: {{ matrix_mailer_sender_address }}
-threepid.medium.email.connectors.smtp.host: matrix-mailer
-threepid.medium.email.connectors.smtp.port: 587
-threepid.medium.email.connectors.smtp.tls: 0
-
-{% if matrix_mxisd_matrixorg_forwarding_enabled %}
-forward.servers: ['matrix-org']
-{% endif %}
-
-synapseSql.enabled: {{ matrix_mxisd_synapsesql_enabled }}
-synapseSql.type: {{ matrix_mxisd_synapsesql_type }}
-synapseSql.connection: {{ matrix_mxisd_synapsesql_connection }}
-
-ldap.enabled: {{ matrix_mxisd_ldap_enabled }}
-ldap.connection.host: {{ matrix_mxisd_ldap_connection_host }}
-ldap.connection.tls: {{ matrix_mxisd_ldap_connection_tls }}
-ldap.connection.port: {{ matrix_mxisd_ldap_connection_port }}
-ldap.connection.baseDns:
-{{ matrix_mxisd_ldap_connection_baseDns|to_nice_yaml }}
-ldap.connection.bindDn: {{ matrix_mxisd_ldap_connection_bindDn }}
-ldap.connection.bindPassword: {{ matrix_mxisd_ldap_connection_bindPassword }}
-
-{% if matrix_mxisd_ldap_filter is defined %}
-ldap.filter: {{ matrix_mxisd_ldap_filter }}
-{% endif %}
-
-{% if matrix_mxisd_ldap_attribute_uid_type is defined %}
-ldap.attribute.uid.type: {{ matrix_mxisd_ldap_attribute_uid_type }}
-{% endif %}
-
-{% if matrix_mxisd_ldap_attribute_uid_value is defined %}
-ldap.attribute.uid.value: {{ matrix_mxisd_ldap_attribute_uid_value }}
-{% endif %}
-
-{% if matrix_mxisd_ldap_attribute_name is defined %}
-ldap.attribute.name: {{ matrix_mxisd_ldap_attribute_name }}
-{% endif %}
-
-{% if matrix_mxisd_ldap_attribute_threepid_email is defined %}
-ldap.attribute.threepid.email: {{ matrix_mxisd_ldap_attribute_threepid_email|to_yaml }}
-{% endif %}
-
-{% if matrix_mxisd_ldap_attribute_threepid_msisdn is defined %}
-ldap.attribute.threepid.msisdn: {{ matrix_mxisd_ldap_attribute_threepid_msisdn|to_yaml }}
-{% endif %}
-
-{% if matrix_mxisd_ldap_identity_filter is defined %}
-ldap.identity.filter: {{ matrix_mxisd_ldap_identity_filter }}
-{% endif %}
-
-{% if matrix_mxisd_ldap_identity_medium is defined %}
-ldap.identity.medium: {{ matrix_mxisd_ldap_identity_medium }}
-{% endif %}
-
-{% if matrix_mxisd_ldap_auth_filter is defined %}
-ldap.auth.filter: {{ matrix_mxisd_ldap_auth_filter }}
-{% endif %}
-
-{% if matrix_mxisd_ldap_directory_filter is defined %}
-ldap.directory.filter: {{ matrix_mxisd_ldap_directory_filter }}
-{% endif %}
-