INFO Server SYS-3  CPU: Intel Xeon-E 2288G - 8c/16t - 3.7 GHz/5 GHz: dedicated server(resurrection)(vehicle)(duel)OVH
ip: 51.91.80.158
SSH password: c9DVSPntoWquzp1X
user: jeff
port: 222

# MatrixRTC / Element Call Troubleshooting Summary

## Infrastructure

- **cyper-proxy**: nginx, livekit, lk-jwt-service
- **cyper-controller**: matrix-synapse 1.151.0, postgresql, element-web, cinny, fluffychat

---

## Changes Made

### synapse.nix (controller)

- Added `msc3401_enabled`, `msc4143_enabled`, `msc4195_enabled` to `experimental_features`

### livekit.nix (proxy)

- `livekitUrl` changed from `ws://127.0.0.1:7880` to `wss://cyperpunk.de/livekit/sfu`
- Added `settings.rtc.use_external_ip = true`
- Added `settings.rtc.tcp_port`, `udp_port`, port ranges
- Added `MATRIX_BASE_URL` and `LIVEKIT_FULL_ACCESS_HOMESERVERS` to environment

### nginx.nix (proxy)

- Fixed `/livekit/jwt` location to use trailing slash on both location and proxyPass to strip prefix correctly
- Added trailing slash to `/livekit/sfu/` proxyPass to strip prefix before forwarding to LiveKit
- Added WebSocket timeouts to `/livekit/sfu` location

### clients.nix (controller)

- Fixed `element_call.url` from JWT service URL to Element Call frontend URL
- Added `livekit.livekit_service_url` pointing to JWT service
- Set `feature_disable_call_per_sender_encryption = true`
- Added `elementCallConfigured` derivation self-hosting Element Call

---

## What Works

- nginx correctly routes `/livekit/jwt/` to lk-jwt-service (confirmed via 405 on GET)
- nginx correctly routes `/livekit/sfu/` to LiveKit (confirmed via 400)
- lk-jwt-service authenticates Matrix users correctly
- lk-jwt-service issues JWTs successfully
- LiveKit SFU WebSocket connection works when JWT contains correct public URL
- Synapse MSC4143 `/rtc/transports` endpoint responds correctly when authenticated
- Hairpin NAT works on cyper-proxy (confirmed via curl returning 400 from LiveKit)
- well-known serves correct `org.matrix.msc4143.rtc_foci`

---

## Remaining Problems

### Problem 1: Room Creation Fails (critical)

**Error:** `Unable to create room on SFU`

**Root cause:** `LIVEKIT_URL` in lk-jwt-service serves two purposes:

1. Used internally by lk-jwt-service to call the LiveKit RoomService API to create rooms
2. Embedded in the JWT token returned to the client for WebSocket connection

These cannot be split in lk-jwt-service v0.4.3 — there is no separate env var for each purpose.

- `ws://127.0.0.1:7880` → room creation works, but client receives `ws://127.0.0.1:7880` in JWT and cannot connect
- `wss://cyperpunk.de/livekit/sfu` → client receives correct URL in JWT, but room creation fails

Room creation fails despite hairpin NAT working. The exact error from lk-jwt-service is only `Unable to create room on SFU` with no further detail in logs. LiveKit logs nothing when this happens, suggesting the request never reaches LiveKit's RoomService. The LiveKit Go SDK converts `wss://` to `https://` for the RoomService Twirp API — it is unclear whether nginx correctly handles this Twirp HTTP/2 traffic through the `/livekit/sfu/` proxy path.

### Problem 2: Element Web Bundles Wrong Element Call Build (critical)

**Error:** `Unknown or unsupported from-widget action: io.element.join`

**Root cause:** The nixpkgs build of Element Web 1.12.10 bundles the **standalone/spa** build of Element Call (`spa-3dms9gnk.js`) at `/widgets/element-call/`, but the correct build for widget embedding is `@element-hq/element-call-embedded@0.16.3`. This is a nixpkgs packaging bug. The standalone build does not use the widget API, so it makes direct Matrix API calls which fail with 401 (no access token) and does not support `io.element.join`/`io.element.close` widget actions.

This affects both Element Web and Cinny.

Using `call.element.io` as the frontend avoids this problem as it serves the correct embedded build, but Problem 1 (room creation) must be resolved first.

---

## Next Steps

1. Determine why lk-jwt-service room creation fails when using `wss://cyperpunk.de/livekit/sfu` — specifically whether nginx is correctly proxying Twirp HTTP/2 traffic to LiveKit's RoomService on port 7880, or whether a separate internal proxy path is needed for the API vs WebSocket traffic.
    
2. Once room creation works with the public URL, use `call.element.io` as `element_call.url` since it serves the correct embedded build.
    
3. Long term: track the nixpkgs fix for the embedded Element Call packaging bug, or override the Element Web derivation to inject the correct embedded build.g