Sign in Subscribe
matrix

Debugging Matrix Synapse Room Creation: The Tale of Encoded Hashes and Traefik v3

How to Solve "Room Address is Already in Use" with Matrix Synapse

Michael

• 8 min read
Debugging Matrix Synapse Room Creation: The Tale of Encoded Hashes and Traefik v3

The Problem

After deploying a self-hosted Matrix Synapse server using the matrix-docker-ansible-deploy playbook, I encountered a frustrating issue: I couldn't create any public rooms in Element. This affected both the Element web client and the iOS app - no matter what room address I entered, Element would report that the address was already in use, even for completely random strings I'd never used before.

The fact that this issue appeared on both clients ruled out any client-specific bugs and pointed to a server-side problem.

My Infrastructure Setup

Before diving into the debugging process, it's important to understand my specific configuration, which added some complexity to the troubleshooting:

Architecture Overview:

Internet → Nginx LXC (10.0.0.10) → Traefik LXC (10.0.0.20:81) → Synapse Container
  • External Nginx Reverse Proxy: Running in a separate LXC container at 10.0.0.10, handling SSL termination and reverse proxying for ALL my services
  • Matrix Stack: Deployed via ansible playbook in another LXC at 10.0.0.20
  • Traefik (internal, port 81): Acts as an entrypoint/router for Matrix services
  • Synapse: The actual Matrix homeserver
  • Element: The web client
  • PostgreSQL, Coturn, etc.

This multi-layered setup meant requests flowed: Client → Nginx → Traefik → Synapse, which would prove crucial to understanding the problem.

Nginx Configuration

My Nginx was configured to proxy everything to Traefik on port 81:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name matrix.example.com element.example.com;

    # SSL config omitted for brevity
    
    location / {
        proxy_pass         http://10.0.0.20:81;
        proxy_set_header   Host $host;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_set_header   X-Real-IP $remote_addr;
        
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection "upgrade";
        
        proxy_buffering off;
        proxy_request_buffering off;
    }
}

Initial Investigation: Browser Console

The first clue came from the browser's developer console when attempting to create a room with address #123:example.com:

Access to fetch at 'https://matrix.example.com/_matrix/client/v3/directory/room/%23123%3Aexample.com' 
from origin 'https://element.example.com' has been blocked by CORS policy: 
Response to preflight request doesn't pass access control check: 
No 'Access-Control-Allow-Origin' header is present on the requested resource.

First assumption: CORS issue. This seemed straightforward - Element at element.example.com couldn't make requests to matrix.example.com due to missing CORS headers.

Red Herring #1: CORS Configuration

I added CORS headers to my Nginx configuration for the /_matrix location:

location /_matrix {
    # Hide CORS headers from upstream to prevent duplicates
    proxy_hide_header Access-Control-Allow-Origin;
    proxy_hide_header Access-Control-Allow-Methods;
    proxy_hide_header Access-Control-Allow-Headers;
    
    # Set our own CORS headers
    add_header 'Access-Control-Allow-Origin' 'https://element.example.com' always;
    add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS' always;
    add_header 'Access-Control-Allow-Headers' 'Origin, X-Requested-With, Content-Type, Accept, Authorization' always;
    add_header 'Access-Control-Max-Age' '1728000' always;

    # Handle preflight OPTIONS requests
    if ($request_method = 'OPTIONS') {
        add_header 'Access-Control-Allow-Origin' 'https://element.example.com' always;
        add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS' always;
        add_header 'Access-Control-Allow-Headers' 'Origin, X-Requested-With, Content-Type, Accept, Authorization' always;
        add_header 'Content-Type' 'text/plain; charset=utf-8' always;
        add_header 'Content-Length' 0 always;
        return 204;
    }

    proxy_pass http://10.0.0.20:81;
    # ... rest of proxy config
}

This fixed the CORS issue, but the room creation still failed! Now the browser showed:

GET https://matrix.example.com/_matrix/client/v3/directory/room/%23123%3Aexample.com 400 (Bad Request)

A 400 error with zero bytes in the response body. Very unusual - Synapse normally returns detailed JSON error messages.

Debugging Step 1: Enable Docker Logging

The ansible playbook had disabled Docker logging for containers by default (using --log-driver=none). This made debugging nearly impossible.

Enabling Synapse Logs

I edited /matrix/vars.yml (the playbook's configuration file):

matrix_synapse_container_extra_arguments:
  - "--log-driver=json-file"
  - "--log-opt max-size=10m"
  - "--log-opt max-file=3"

Then re-ran the playbook:

cd /opt/matrix-docker-ansible-deploy
ansible-playbook -i inventory/hosts setup.yml --tags=setup-synapse,start

Now I could view Synapse logs:

docker logs -f matrix-synapse

Result: Nothing appeared in Synapse logs when I tried to create a room. The request wasn't reaching Synapse at all!

Debugging Step 2: Nginx Access Logs

I checked Nginx access logs on the reverse proxy LXC:

tail -f /var/log/nginx/access.log | grep "_matrix"

The logs showed:

10.0.0.1 - - [11/Dec/2025:06:51:29 +0000] "GET /_matrix/client/v3/directory/room/%23123%3Aexample.com HTTP/2.0" 400 0

The request was hitting Nginx and returning 400, but with 0 bytes. The Nginx error log showed:

connect() failed (111: Connection refused) while connecting to upstream

But this turned out to be from old log entries when containers were restarting. Current requests weren't showing errors.

Debugging Step 3: Testing the Chain

I needed to understand where the request was failing. Testing from the Matrix server:

# Direct to Synapse (bypassing Traefik)
docker exec matrix-synapse curl http://localhost:8008/_matrix/client/v3/directory/room/%23123%3Aexample.com

Result:

{"errcode":"M_NOT_FOUND","error":"Room alias #123:example.com not found"}

Perfect! Synapse was working correctly. The problem was between Nginx and Synapse.

Testing Traefik connectivity:

# Without Host header - returns 404
curl http://10.0.0.20:81/_matrix/client/versions
# 404 page not found

# With Host header - works!
curl -H "Host: matrix.example.com" http://10.0.0.20:81/_matrix/client/versions
# Returns proper JSON response

So Traefik was working and Nginx was correctly passing the Host header. But the room alias endpoint was still failing.

Debugging Step 4: Enable Traefik Debug Logging

Time to see what Traefik was doing. I edited /matrix/vars.yml again:

traefik_config_log_level: DEBUG
traefik_config_accessLog_enabled: true

# Enable Docker logging for Traefik
traefik_container_extra_arguments:
  - "--log-driver=json-file"
  - "--log-opt max-size=10m"
  - "--log-opt max-file=3"

Re-ran the playbook:

ansible-playbook -i inventory/hosts setup.yml --tags=setup-traefik,start

Now I could see Traefik logs:

docker logs -f matrix-traefik

The Smoking Gun

When I attempted to create a room, this appeared in Traefik's debug logs:

2025-12-11T07:02:15Z DBG Rejecting request because it contains encoded character %23 in the URL path: 
/_matrix/client/v3/directory/room/%23123%3Aexample.com

There it was! Traefik v3 was rejecting the request because Matrix room aliases contain # (hash), which gets URL-encoded as %23. Traefik v3 has security features that reject encoded special characters by default to prevent path traversal attacks.

The Solution: Traefik v3 Encoded Character Configuration

In Traefik v3, encoded character filtering is controlled via the encodedCharacters configuration under each entrypoint. By default, encoded hashes (%23) are rejected.

I needed to explicitly allow encoded hashes for the web entrypoint (port 81) that Nginx was proxying to.

Final Configuration

I added this to /matrix/vars.yml:

traefik_configuration_extension_yaml: |
  entryPoints:
    web:
      http:
        encodedCharacters:
          allowEncodedHash: true

Applied the changes:

cd /opt/matrix-docker-ansible-deploy
ansible-playbook -i inventory/hosts setup.yml --tags=setup-traefik,start

Verified the configuration was applied:

docker exec matrix-traefik cat /config/traefik.yml

The generated config now included:

entryPoints:
  web:
    address: ":8080"
    http:
      encodedCharacters:
        allowEncodedHash: true
    forwardedHeaders:
      trustedIPs: 10.0.0.10
    # ... rest of config

Testing

Attempted to create a room with address #test123:example.com:

Success! The room was created without any errors.

Key Lessons Learned

  1. Multi-layer debugging is essential: With Nginx → Traefik → Synapse, I had to systematically test each layer to isolate the problem.
  2. Enable logging first: The default --log-driver=none in the ansible playbook made initial debugging nearly impossible. Always enable logging in production environments.
  3. 400 errors with empty bodies are suspicious: They often indicate the request is being rejected by a proxy/gateway before reaching the application.
  4. Traefik v3's security defaults: The encoded character filtering in Traefik v3 is a good security feature, but it requires explicit configuration for legitimate use cases like Matrix room aliases.
  5. Browser console + server logs = complete picture: The browser console showed the client-side CORS error initially, but only server-side logs revealed the real issue.

Debugging Checklist for Similar Issues

If you're experiencing similar problems with Matrix or other services behind Traefik v3:

  1. Check browser console for client-side errors (CORS, network failures)
  2. Enable Docker logging for all containers in your stack
  3. Check Nginx/reverse proxy logs (both access and error logs)
  4. Test each layer independently:
  5. Direct container access (bypassing all proxies)
  6. Internal proxy (e.g., Traefik) with proper headers
  7. External proxy (e.g., Nginx) access
  8. Enable debug logging in Traefik to see detailed rejection reasons
  9. Review Traefik v3's encoded character defaults if dealing with special characters in URLs

Complete Configuration Reference

For reference, here's the complete relevant configuration:

/matrix/vars.yml (Ansible Playbook)

matrix_domain: example.com
matrix_homeserver_implementation: synapse

# Enable Traefik as internal reverse proxy
matrix_playbook_reverse_proxy_type: "playbook-managed-traefik"
traefik_container_web_host_bind_port: '0.0.0.0:81'
traefik_config_entrypoint_web_forwardedHeaders_trustedIPs: '10.0.0.10'

# Enable logging
traefik_config_log_level: DEBUG
traefik_config_accessLog_enabled: true

traefik_container_extra_arguments:
  - "--log-driver=json-file"
  - "--log-opt max-size=10m"
  - "--log-opt max-file=3"

matrix_synapse_container_extra_arguments:
  - "--log-driver=json-file"
  - "--log-opt max-size=10m"
  - "--log-opt max-file=3"

# Allow encoded hashes in URLs (for Matrix room aliases)
traefik_configuration_extension_yaml: |
  entryPoints:
    web:
      http:
        encodedCharacters:
          allowEncodedHash: true

Nginx Configuration (/etc/nginx/sites-available/matrix.conf)

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name matrix.example.com element.example.com;

    ssl_certificate     /etc/letsencrypt/live/matrix.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/matrix.example.com/privkey.pem;
    
    client_max_body_size 100M;

    # CORS headers for Matrix API
    location /_matrix {
        proxy_hide_header Access-Control-Allow-Origin;
        proxy_hide_header Access-Control-Allow-Methods;
        proxy_hide_header Access-Control-Allow-Headers;
        
        add_header 'Access-Control-Allow-Origin' 'https://element.example.com' always;
        add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS' always;
        add_header 'Access-Control-Allow-Headers' 'Origin, X-Requested-With, Content-Type, Accept, Authorization' always;
        add_header 'Access-Control-Max-Age' '1728000' always;

        if ($request_method = 'OPTIONS') {
            add_header 'Access-Control-Allow-Origin' 'https://element.example.com' always;
            add_header 'Access-Control-Allow-Methods' 'GET, POST, PUT, DELETE, OPTIONS' always;
            add_header 'Access-Control-Allow-Headers' 'Origin, X-Requested-With, Content-Type, Accept, Authorization' always;
            add_header 'Content-Type' 'text/plain; charset=utf-8' always;
            add_header 'Content-Length' 0 always;
            return 204;
        }

        proxy_pass         http://10.0.0.20:81;
        proxy_set_header   Host $host;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_set_header   X-Real-IP $remote_addr;
        
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection "upgrade";
        
        proxy_buffering off;
        proxy_request_buffering off;
    }

    # Everything else (Element client)
    location / {
        proxy_pass         http://10.0.0.20:81;
        proxy_set_header   Host $host;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_set_header   X-Real-IP $remote_addr;
        
        proxy_http_version 1.1;
        proxy_set_header   Upgrade $http_upgrade;
        proxy_set_header   Connection "upgrade";
        
        proxy_buffering off;
        proxy_request_buffering off;
    }

    # Well-known delegation
    location /.well-known/matrix/server {
        return 200 '{"m.server": "matrix.example.com:443"}';
        default_type application/json;
        add_header Access-Control-Allow-Origin * always;
    }

    location /.well-known/matrix/client {
        return 200 '{"m.homeserver": {"base_url": "https://matrix.example.com"}}';
        default_type application/json;
        add_header Access-Control-Allow-Origin * always;
    }
}

Conclusion

What started as a seemingly simple "room address already in use" error turned into a deep dive through multiple layers of reverse proxies, CORS policies, and Traefik v3's security features. The key to solving it was systematic debugging: enabling logging at each layer, testing components independently, and reading the actual debug output rather than making assumptions.

If you're running Matrix Synapse behind Traefik v3 (especially with an additional Nginx layer like my setup), make sure to configure allowEncodedHash: true in your Traefik entrypoint configuration. Matrix room aliases inherently contain # characters that get URL-encoded, and Traefik v3's default security settings will reject these requests.

Happy Matrix hosting!

Bonus: Encrypted Key Backups and Encoded Slashes

After fixing the room creation issue, I discovered another related problem when using the Element desktop app (macOS). The app would fail to sync encrypted key backups, and Traefik logs revealed:

2025-12-11T17:34:29Z DBG Rejecting request because it contains encoded character %2F in the URL path: 
/_matrix/client/v3/room_keys/keys/!CNDCABmKHxcB0jCNcwco4rjRigPdXQY8DAX2xz1amew/DC9Hqnmarz3MsneWzGyhb9iiES%2B7cqYN7qj3%2FQlanU0

The issue: Matrix's end-to-end encryption key identifiers use base64 encoding, which can include / (forward slash) characters. When these get URL-encoded to %2F, Traefik v3 rejects them by default.

The Fix

You need to allow both encoded hashes and encoded slashes. Update your configuration:

yaml

traefik_configuration_extension_yaml: |
  entryPoints:
    web:
      http:
        encodedCharacters:
          allowEncodedHash: true      # For room aliases (#room:domain)
          allowEncodedSlash: true     # For encryption key IDs (base64)

Re-run the playbook:

bash

cd /opt/matrix-docker-ansible-deploy
ansible-playbook -i inventory/hosts setup.yml --tags=setup-traefik,start

This ensures that both room creation and encrypted key backup/sync functionality work properly.

Why This Matters

Without allowEncodedSlash: true:

  • Encrypted key backups will fail to sync
  • Cross-device verification may have issues
  • Element desktop/mobile apps may show sync errors
  • Room keys might not be properly backed up

The iOS app worked fine for basic functionality, but the desktop app's key backup features revealed this additional requirement.

Environment Details:

  • Matrix Synapse: 1.144.0
  • Traefik: v3 (via matrix-docker-ansible-deploy playbook)
  • Nginx: 1.24.0
  • Element Web: 1.12.6
  • Element iOS App: Latest version (as of December 2025)
  • OS: Ubuntu 24 (LXC containers)