diff --git a/README.md b/README.md index 3188b86..8beac9a 100644 --- a/README.md +++ b/README.md @@ -1,163 +1,474 @@ -# xonotic_exporter +# xonotic-exporter -Prometheus exporter for [Xonotic](https://xonotic.org/) game servers. -Scrapes metrics via RCON (`sv_public` + `status 1`) and exposes them -in Prometheus exposition format. +A production-grade [Prometheus](https://prometheus.io/) exporter for [Xonotic](https://xonotic.org/) game servers. It scrapes server metrics via RCON — using all three Xonotic authentication modes — and exposes them in Prometheus exposition format on a single HTTP endpoint. One process monitors as many servers as you like. + +--- + +## Table of Contents + +- [Features](#features) +- [Architecture](#architecture) +- [Requirements](#requirements) +- [Installation](#installation) +- [Configuration](#configuration) +- [Running the exporter](#running-the-exporter) +- [CLI reference](#cli-reference) +- [HTTP endpoints](#http-endpoints) +- [Prometheus scrape configuration](#prometheus-scrape-configuration) +- [Metrics reference](#metrics-reference) +- [RCON modes explained](#rcon-modes-explained) +- [Project structure](#project-structure) + +--- ## Features -- **Python 3.11+** (fully compatible with 3.13 — no deprecated asyncio `loop=` args) -- **TOML configuration** — one file, all servers, one systemd unit -- **All three RCON modes** — nonsecure, secure-time, secure-challenge (MD4) -- **Blackbox-style multi-target** — one exporter, many game servers -- **CLI test interface** — human-readable or raw Prometheus output -- **Zero-downtime config reload** — `kill -HUP $PID` or `POST /-/reload` +- **Multi-server** — monitor any number of Xonotic servers from a single process and config file +- **All three RCON modes** — nonsecure, secure-time (HMAC-MD4), and secure-challenge (HMAC-MD4) — matching whatever your `server.cfg` uses +- **Blackbox-style multi-target** — Prometheus passes `?target=` per scrape, one exporter serves all servers +- **GeoIP player tracking** — optional `/geo` endpoint resolves player IP addresses to country/city via ip-api.com (free, cached 24 h) +- **Match data endpoint** — `/match` exposes per-player frags, spectator status, and match metadata (timelimit, fraglimit, teamplay) +- **Zero-downtime config reload** — send `SIGHUP` or `POST /-/reload` without restarting +- **CLI test interface** — query any server from the command line, human-readable or raw Prometheus format +- **Retry logic** — configurable retry count and per-attempt timeout on every RCON scrape +- **No external HTTP framework** — HTTP server is built on `asyncio.start_server` from the standard library +- **Python 3.11+** — uses `tomllib` (stdlib), fully compatible with 3.13 + +--- + +## Architecture + +``` + ┌─────────────────────────────────────────────┐ + │ xonotic-exporter │ + │ │ + Prometheus ───────► HTTP :9260 │ + scrapes │ GET /metrics?target=vehicles ───────────► UDP RCON ──► Xonotic srv A + /metrics?target=N │ GET /metrics?target=dm ───────────► UDP RCON ──► Xonotic srv B + │ GET /geo?target=vehicles ───────────► ip-api.com │ + │ GET /match?target=vehicles ───────────► UDP RCON │ + │ POST /-/reload │ │ + │ GET /-/healthy │ │ + │ │ + │ TOML config ◄── SIGHUP reloads on disk │ + └─────────────────────────────────────────────┘ +``` + +### How a single scrape works + +``` +Prometheus xonotic-exporter Xonotic server + │ │ │ + │ GET /metrics?target=dm │ │ + │───────────────────────────►│ │ + │ │ UDP: ping packet │ + │ │──────────────────────────────►│ + │ │ UDP: getchallenge (mode 2) │ + │ │──────────────────────────────►│ + │ │ challenge token │ + │ │◄──────────────────────────────│ + │ │ UDP: RCON "sv_public\0status 1" + │ │ (HMAC-MD4 signed) │ + │ │──────────────────────────────►│ + │ │ pong │ + │ │◄──────────────────────────────│ + │ │ RCON response chunks │ + │ │◄──────────────────────────────│ + │ │ parse → build Prometheus reg │ + │ 200 OK + metrics text │ │ + │◄───────────────────────────│ │ +``` + +Both the ping and the RCON query are sent concurrently. The parser uses an adaptive wait: it reads the first chunk, notes the RTT, then waits up to `2× RTT` for additional packets (Xonotic responses can span multiple UDP datagrams). + +--- + +## Requirements + +- Python **3.11** or later (3.12 / 3.13 fully supported) +- [`xrcon`](https://pypi.org/project/xrcon/) ≥ 0.1 — Xonotic RCON packet helpers +- [`prometheus-client`](https://pypi.org/project/prometheus-client/) ≥ 0.20 + +Both dependencies are installed automatically via `pip`. + +--- ## Installation +### From source (recommended) + ```bash +# 1. Create a dedicated virtualenv +python3 -m venv /opt/xonotic_exporter/venv + +# 2. Install the package and its dependencies +/opt/xonotic_exporter/venv/bin/pip install . + +# 3. Verify +/opt/xonotic_exporter/venv/bin/xonotic-exporter --help +``` + +### From a zip archive + +```bash +unzip Xonotic-Exporter-v1_0.zip +cd xonotic-exporter python3 -m venv /opt/xonotic_exporter/venv /opt/xonotic_exporter/venv/bin/pip install . ``` +The `xonotic-exporter` command will now be available inside the venv. + +--- + ## Configuration -Copy `examples/xonotic_exporter.toml` to `/etc/xonotic_exporter/xonotic_exporter.toml` -and fill in your `rcon_password` values. - -```toml -[exporter] -host = "0.0.0.0" -port = 9260 - -[[servers]] -name = "vehicles" -host = "localhost" -port = 26010 -rcon_password = "secret" -rcon_mode = 2 # 2 = secure-challenge (MD4) — recommended - -[[servers]] -name = "resurrection" -host = "localhost" -port = 26015 -rcon_password = "secret" -rcon_mode = 2 - -[[servers]] -name = "insurrection" -host = "localhost" -port = 26016 -rcon_password = "secret" -rcon_mode = 2 -``` - -**rcon_mode values** match your server's `server.cfg`: - -| Value | Name | server.cfg setting | -|-------|-------------------|--------------------| -| `0` | nonsecure | `rcon_restricted 0` | -| `1` | secure-time | `rcon_secure 1` | -| `2` | secure-challenge | `rcon_secure 2` | - -## Running - -### As a systemd service +Copy the example config and edit it: ```bash -sudo cp examples/xonotic_exporter.service /etc/systemd/system/ -sudo systemctl daemon-reload -sudo systemctl enable --now xonotic_exporter +sudo mkdir -p /etc/xonotic_exporter +sudo cp examples/xonotic_exporter.toml /etc/xonotic_exporter/xonotic_exporter.toml +sudo $EDITOR /etc/xonotic_exporter/xonotic_exporter.toml ``` -Reload config without restarting: +### Full config reference + +```toml +# /etc/xonotic_exporter/xonotic_exporter.toml + +[exporter] +host = "0.0.0.0" # bind address; use "127.0.0.1" to listen locally only +port = 9260 # Prometheus scrapes http://:9260/metrics?target= + +# Add one [[servers]] block per Xonotic game server. +[[servers]] +name = "vehicles" # unique label; used as the Prometheus `instance` label +host = "localhost" # hostname or IP of the game server +port = 26010 # game server UDP port (default 26000) +rcon_password = "CHANGEME" # must match rcon_password in server.cfg +rcon_mode = 2 # see RCON modes section below + +[[servers]] +name = "deathmatch" +host = "game.example.com" +port = 26000 +rcon_password = "CHANGEME" +rcon_mode = 2 +``` + +#### `[exporter]` options + +| Key | Default | Description | +|-----|---------|-------------| +| `host` | `"0.0.0.0"` | Address the HTTP server binds to | +| `port` | `9260` | Port the HTTP server listens on | + +#### `[[servers]]` options + +| Key | Default | Required | Description | +|-----|---------|----------|-------------| +| `name` | — | yes | Unique server identifier; becomes the `instance` Prometheus label | +| `host` | — | yes | Hostname or IP of the Xonotic game server | +| `port` | `26000` | no | UDP port of the game server | +| `rcon_password` | `""` | yes | RCON password (matches `rcon_password` in `server.cfg`) | +| `rcon_mode` | `2` | no | RCON authentication mode: `0`, `1`, or `2` | + +### Validating your config + +```bash +xonotic-exporter validate /etc/xonotic_exporter/xonotic_exporter.toml +# OK — 2 server(s) configured: +# vehicles localhost:26010 rcon_mode=2 (secure-challenge (MD4)) +# deathmatch game.example.com:26000 rcon_mode=2 (secure-challenge (MD4)) +``` + +--- + +## Running the exporter + +### As a systemd service (recommended for production) + +```bash +# Create a dedicated system user +sudo useradd -r -s /sbin/nologin monitoring + +# Install the unit file +sudo cp examples/xonotic_exporter.service /etc/systemd/system/ + +# Enable and start +sudo systemctl daemon-reload +sudo systemctl enable --now xonotic_exporter + +# Check status +sudo systemctl status xonotic_exporter +``` + +The unit file already includes hardening options (`NoNewPrivileges`, `ProtectHome`, `ProtectSystem=strict`, capability stripping). + +**Reload config without restarting the process:** ```bash sudo systemctl reload xonotic_exporter -# or -curl -XPOST http://localhost:9260/-/reload +# or equivalently: +curl -s -XPOST http://localhost:9260/-/reload ``` ### Manually ```bash -xonotic-exporter serve /etc/xonotic_exporter/xonotic_exporter.toml +/opt/xonotic_exporter/venv/bin/xonotic-exporter serve \ + /etc/xonotic_exporter/xonotic_exporter.toml + +# Override listen address/port at runtime: +xonotic-exporter serve config.toml --host 127.0.0.1 --port 9261 ``` -## CLI Testing +--- + +## CLI reference + +The exporter ships with three sub-commands. + +### `serve` — start the HTTP server + +``` +xonotic-exporter serve CONFIG.toml [--host HOST] [--port PORT] +``` + +| Flag | Description | +|------|-------------| +| `CONFIG.toml` | Path to the TOML configuration file | +| `-l`, `--host` | Override the listen host from config | +| `-p`, `--port` | Override the listen port from config | + +### `query` — scrape one server and print results + +Use this to test connectivity and inspect what Prometheus would receive. + +**From a config file (by server name):** ```bash -# Human-readable output (default) -xonotic-exporter query vehicles --config xonotic_exporter.toml - -# Raw Prometheus exposition -xonotic-exporter query vehicles --config xonotic_exporter.toml --prometheus - -# Ad-hoc (no config file needed) -xonotic-exporter query --host localhost --port 26010 --password secret --mode 2 - -# Ad-hoc with verbose logging -xonotic-exporter -v query --host localhost --port 26010 --password secret - -# Validate config -xonotic-exporter validate /etc/xonotic_exporter/xonotic_exporter.toml +xonotic-exporter query vehicles --config /etc/xonotic_exporter/xonotic_exporter.toml ``` -Human-readable output example: +**Ad-hoc (no config file needed):** + +```bash +xonotic-exporter query --host localhost --port 26010 --password secret --mode 2 +``` + +**Raw Prometheus exposition format:** + +```bash +xonotic-exporter query vehicles --config config.toml --prometheus +``` + +**With verbose (debug) logging:** + +```bash +xonotic-exporter -v query --host localhost --port 26010 --password secret +``` + +**Example human-readable output:** ``` RCON mode : secure-challenge (MD4) -──────────────────────────────────────────────────── +──────────────────────────────────────────────────────────────────── Server : vehicles - Hostname : My Xonotic Server + Hostname : My Awesome Xonotic Server Map : warfare Public : yes (sv_public=1) -──────────────────────────────────────────────────── +──────────────────────────────────────────────────────────────────── Ping : 2.3 ms CPU : 1.4 % Lost : 0.0 % Offset : avg=0.02ms max=1.20ms sdev=0.10ms -──────────────────────────────────────────────────── +──────────────────────────────────────────────────────────────────── Players : 4/16 Active : 3 Spectators : 1 Bots : 0 -──────────────────────────────────────────────────── +──────────────────────────────────────────────────────────────────── ``` -## Prometheus Configuration +#### `query` flags -See `examples/prometheus.yml` for a ready-to-use scrape config. +| Flag | Default | Description | +|------|---------|-------------| +| `SERVER_NAME` | — | Named server from config (requires `--config`) | +| `--config` | — | Path to TOML config file | +| `--host` | — | Ad-hoc server hostname | +| `--port` | `26000` | Ad-hoc server UDP port | +| `--password` | `""` | Ad-hoc RCON password | +| `--mode` | `2` | Ad-hoc RCON mode (0, 1, or 2) | +| `--prometheus` | false | Print raw Prometheus text instead of human-readable output | +| `--retries` | `3` | Number of scrape attempts before giving up | +| `--timeout` | `5.0` | Per-attempt timeout in seconds | -## Metrics +### `validate` — check a config file -| Metric | Description | -|--------|-------------| -| `xonotic_up` | 1 if server reachable | -| `xonotic_sv_public` | Value of sv_public cvar | -| `xonotic_ping_seconds` | Round-trip time to server | -| `xonotic_timing_cpu_percent` | Server CPU usage % | -| `xonotic_timing_lost_percent` | Packet loss % | -| `xonotic_timing_offset_avg_ms` | Average timing offset ms | -| `xonotic_timing_offset_max_ms` | Max timing offset ms | -| `xonotic_timing_offset_sdev_ms` | Timing offset std dev ms | -| `xonotic_players_active` | Active (scoring) players | -| `xonotic_players_spectators` | Spectators | -| `xonotic_players_bots` | Bots | -| `xonotic_players_total` | Total connected | -| `xonotic_players_max` | Max player slots | +```bash +xonotic-exporter validate /etc/xonotic_exporter/xonotic_exporter.toml +``` -All metrics carry an `instance` label set to the server name from TOML. +Exits with code `0` on success, `1` on any config error. -## Endpoints +--- + +## HTTP endpoints | Endpoint | Method | Description | |----------|--------|-------------| -| `/` | GET | HTML index with all configured servers | -| `/metrics?target=` | GET | Prometheus metrics for one server | -| `/-/reload` | POST | Reload config from disk | -| `/-/healthy` | GET | Liveness probe | -======= -# Xonotic-Exporter +| `/` | `GET` | HTML index page listing all configured servers with links | +| `/metrics?target=` | `GET` | Prometheus metrics for the named server | +| `/geo?target=` | `GET` | Player geolocation metrics (country, city, lat/lon) | +| `/match?target=` | `GET` | Match data: frags, spectators, timelimit, fraglimit, teamplay | +| `/-/reload` | `POST` | Reload config from disk; returns `200` on success, `500` on failure | +| `/-/healthy` | `GET` | Liveness probe — always returns `200 OK` | + +--- + +## Prometheus scrape configuration + +The exporter uses the [blackbox exporter pattern](https://prometheus.io/docs/guides/multi-target-exporter/): Prometheus sends a `?target=` query parameter, and the exporter queries that specific server on demand. This means one exporter process and one HTTP port serves all your game servers. + +Copy `examples/prometheus.yml` or add this to your existing `prometheus.yml`: + +```yaml +scrape_configs: + - job_name: "xonotic" + scrape_interval: 15s + scrape_timeout: 10s + + relabel_configs: + # Pass the server name as ?target= + - source_labels: [__address__] + target_label: __param_target + # Use the server name as the instance label in Grafana + - source_labels: [__param_target] + target_label: instance + # Point all scrapes at the exporter + - target_label: __address__ + replacement: "127.0.0.1:9260" + + static_configs: + - targets: + - "vehicles" # must match a [[servers]] name in the TOML + - "deathmatch" +``` + +The `relabel_configs` block is required — without it, Prometheus would try to scrape the server name directly as an HTTP address. + +For GeoIP and match data, add separate scrape jobs pointing at `/geo` and `/match`: + +```yaml + - job_name: "xonotic_geo" + metrics_path: /geo + scrape_interval: 30s + # ... same relabel_configs and targets as above ... + + - job_name: "xonotic_match" + metrics_path: /match + scrape_interval: 15s + # ... same relabel_configs and targets as above ... +``` + +--- + +## Metrics reference + +### Core server metrics (`/metrics`) + +All metrics carry an `instance` label set to the server name from the TOML config. + +| Metric | Type | Description | +|--------|------|-------------| +| `xonotic_up` | Gauge | `1` if the server was reachable on this scrape, `0` otherwise | +| `xonotic_sv_public` | Gauge | Value of the `sv_public` cvar (`1` = listed on master server) | +| `xonotic_ping_seconds` | Gauge | Round-trip time to the server in seconds | +| `xonotic_timing_cpu_percent` | Gauge | Server CPU usage % (from `status 1`) | +| `xonotic_timing_lost_percent` | Gauge | Packet loss % (from `status 1`) | +| `xonotic_timing_offset_avg_ms` | Gauge | Average timing offset in milliseconds | +| `xonotic_timing_offset_max_ms` | Gauge | Maximum timing offset in milliseconds | +| `xonotic_timing_offset_sdev_ms` | Gauge | Standard deviation of timing offset in milliseconds | +| `xonotic_players_active` | Gauge | Number of active (scoring) players | +| `xonotic_players_spectators` | Gauge | Number of spectators | +| `xonotic_players_bots` | Gauge | Number of bots | +| `xonotic_players_total` | Gauge | Total connected (active + spectators + bots) | +| `xonotic_players_max` | Gauge | Maximum player slots on the server | + +### Player geolocation metrics (`/geo`) + +Labels: `instance`, `ip`, `port`, `slot`, `name`, `country`, `city`, `lat`, `lon`, `ping`. + +| Metric | Description | +|--------|-------------| +| `xonotic_player_info` | Always `1`; carries player identity labels (name, country, city) | +| `xonotic_player_geo` | Always `1`; carries geolocation labels including `lat` and `lon` | +| `xonotic_player_ping` | Per-player ping to the server in milliseconds | +| `xonotic_player_packetloss_percent` | Per-player packet loss percentage | + +Private/loopback IPs are never resolved or emitted. GeoIP results are cached for 24 hours using ip-api.com's free batch endpoint (up to 100 IPs per request, 15 req/min rate limit). + +### Match data metrics (`/match`) + +| Metric | Labels | Description | +|--------|--------|-------------| +| `xonotic_match_timelimit_seconds` | `instance` | Time limit converted to seconds (timelimit × 60) | +| `xonotic_match_fraglimit` | `instance` | Frag limit for the current match | +| `xonotic_match_teamplay` | `instance` | Teamplay mode (`0` = free-for-all) | +| `xonotic_match_info` | `instance`, `map` | Always `1`; carries current map name as a label | +| `xonotic_match_player_frags` | `instance`, `name`, `slot` | Current frag count per player | +| `xonotic_match_spectator` | `instance`, `name`, `slot` | `1` if the player is spectating | +| `xonotic_match_player_connection` | `instance`, `name`, `slot`, `ping`, `packetloss`, `time_seconds` | Always `1`; carries connection stats as labels | + +--- + +## RCON modes explained + +Xonotic supports three RCON authentication modes. The mode in your exporter config **must match** the `rcon_secure` / `rcon_restricted` setting in each game server's `server.cfg`. + +| Mode | Name | `server.cfg` | Security | Notes | +|------|------|--------------|----------|-------| +| `0` | nonsecure | `rcon_restricted 0` | Low — password sent in plaintext | Avoid in production; use only on localhost | +| `1` | secure-time | `rcon_secure 1` | Medium — HMAC-MD4 signed with current timestamp | Requires clocks to be in sync (within a few seconds) | +| `2` | secure-challenge | `rcon_secure 2` | High — HMAC-MD4 signed with a server-issued challenge | **Recommended.** Not susceptible to replay attacks | + +In mode `2` (the default), the exporter first requests a challenge token from the server, then signs the RCON command with `HMAC-MD4(password, challenge + command)`. This exchange happens automatically before every RCON command. + +--- + +## Project structure + +``` +xonotic-exporter/ +├── pyproject.toml # build config, dependencies, entry point +├── examples/ +│ ├── xonotic_exporter.toml # annotated example configuration +│ ├── prometheus.yml # ready-to-use Prometheus scrape config +│ └── xonotic_exporter.service # hardened systemd unit file +└── xonotic_exporter/ + ├── __init__.py # package version + ├── cli.py # argument parser + serve/query/validate sub-commands + ├── config.py # TOML loader and dataclasses (ExporterConfig, ServerConfig) + ├── rcon.py # async UDP protocol: ping, challenge, RCON send/receive + ├── metrics_parser.py # state-machine parser for sv_public + status 1 output + ├── prometheus.py # per-scrape CollectorRegistry builders + ├── server.py # asyncio HTTP server, routing, reload, GeoIP/match handlers + └── geoip.py # async GeoIP cache backed by ip-api.com batch API +``` + +### Module responsibilities + +`cli.py` is the entry point. It parses arguments and dispatches to one of three sub-commands: `serve`, `query`, or `validate`. The `serve` command calls `run_server()` in `server.py`, which creates an `asyncio` HTTP server that handles every incoming request. + +`rcon.py` owns all network I/O. It implements `_XonoticProtocol`, a low-level `asyncio.DatagramProtocol` that handles ping, challenge, and RCON packet framing. The public `scrape_server()` coroutine opens a UDP endpoint, fires ping and RCON commands concurrently, and retries up to the configured limit. + +`metrics_parser.py` contains `XonoticMetricsParser`, a state machine that processes the raw bytes returned by `sv_public` and `status 1` line by line. It handles Xonotic color codes (`^1`, `^x1A2`), IPv4, IPv6, and bot clients. + +`prometheus.py` builds a fresh `CollectorRegistry` on every scrape request — this is the correct multi-target pattern, as it avoids any shared global metric state between servers. + +`geoip.py` maintains a long-lived, process-wide IP → GeoResult cache. Cache entries expire after 24 hours. Lookups for private and loopback addresses are skipped entirely without a network call. + +--- -A simple Prometheus Xonotic Exporter written in Python