Increase Synapse caches and enable cache-autotuning by default (#3017)

* Modify Synapse Cache Factor to use Auto Tune

Synapse has the ability to as it calls in its config auto tune caches.

This ability lets us set very high cache factors and then instead limit our resource use.

Defaults for this commit are 1/10th of what Element apparently runs for EMS stuff and matrix.org on Cache Factor and upstream documentation defaults for auto tune.

* Add vars to Synapse main.yml to control cache related config

This commit adds various cache related vars to main.yml for Synapse.

Some are auto tune and some are just adding explicit ways to control upstream vars.

* Updated Auto Tune figures

Autotuned figures have been bumped in consultation with other community members as to a reasonable level. Please note these defaults are more on the one of each workers side than they are on the monolith Side.

* Fix YML Error

The playbook is not happy with the previous state of this patch so this commit hopefully fixes it

* Add to_json to various Synapse tuning related configs

* Fix incorrect indication in homeserver.yaml.j2

* Minor cleanups

* Synapse Cache Autotuning Documentation

* Upgrade Synapse Cache Autotune to auto configure memory use

* Update Synapse Tuning docs to reflect automatic memory use configuration

* Fix Linting errors in synapses main.yml

* Rename variables for consistency (matrix_synapse_caches_autotuning_* -> matrix_synapse_cache_autotuning_*)

* Remove FIX ME comment about Synapse's `cache_autotuning`

`docs/maintenance-synapse.md` and `roles/custom/matrix-synapse/defaults/main.yml`
already contains documentation about these variables and the default values we set.

* Improve "Tuning caches and cache autotuning" documentation for Synapse

* Announce larger Synapse caches and cache auto-tuning

---------

Co-authored-by: Slavi Pantaleev <slavi@devture.com>

roles/custom/matrix-synapse/defaults/main.yml

@@ -547,8 +547,23 @@ matrix_synapse_event_cache_size: "100K"
 
 # Controls cache sizes for Synapse.
 # Raise this to increase cache sizes or lower it to potentially lower memory use.
-# To learn more, see: https://github.com/matrix-org/synapse/issues/3939
-matrix_synapse_caches_global_factor: 0.5
+# To learn more, see:
+# - https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#caching
+# - https://github.com/matrix-org/synapse/issues/3939
+# Defaults for timings of caches is from https://tcpipuk.github.io/synapse/deployment/synapse.html
+# The idea with the timings used is that you get to evict soon but also you keep stuff around for a long time when its not forced out.
+# Long cache lifetimes together with the low minimum TTL allows autotune to be the primary eviction method assuming size of cache is hit before we hit other caps.
+matrix_synapse_caches_global_factor: 10
+matrix_synapse_caches_expire_caches: true
+matrix_synapse_caches_cache_entry_ttl: "1080m"
+matrix_synapse_caches_sync_response_cache_duration: "2m"
+matrix_synapse_cache_autotuning_min_cache_ttl: "30s"
+# The Cache tune math used here is a derivative of the same math used to autotune sizes for postgres.
+# The memtotal variable can in theory be overiden to make Synapse think it has less ram to work with.
+# But if your at the point of considering that just override the math or put static values in.
+matrix_synapse_memtotal_kb: "{{ ansible_memtotal_mb*1024|int }}"
+matrix_synapse_cache_autotuning_max_cache_memory_usage: "{{ 2097152 if (matrix_synapse_memtotal_kb|int/8)/1024 >= 2048 else matrix_synapse_memtotal_kb|int/8 }}"
+matrix_synapse_cache_autotuning_target_cache_memory_usage: "{{ 1048576 if (matrix_synapse_memtotal_kb|int/16)/1024 >= 1024 else matrix_synapse_memtotal_kb|int/16 }}"
 
 # Controls whether Synapse will federate at all.
 # Disable this to completely isolate your server from the rest of the Matrix network.