Xonotic-Exporter/README.md

# xonotic-exporter

A production-grade [Prometheus](https://prometheus.io/) exporter for [Xonotic](https://xonotic.org/) game servers. It scrapes server metrics via RCON — using all three Xonotic authentication modes — and exposes them in Prometheus exposition format on a single HTTP endpoint. One process monitors as many servers as you like.

---

## Table of Contents

- [Features](#features)
- [Architecture](#architecture)
- [Requirements](#requirements)
- [Installation](#installation)
- [Configuration](#configuration)
- [Running the exporter](#running-the-exporter)
- [CLI reference](#cli-reference)
- [HTTP endpoints](#http-endpoints)
- [Prometheus scrape configuration](#prometheus-scrape-configuration)
- [Metrics reference](#metrics-reference)
- [RCON modes explained](#rcon-modes-explained)
- [Project structure](#project-structure)

---

## Features

- **Multi-server** — monitor any number of Xonotic servers from a single process and config file
- **All three RCON modes** — nonsecure, secure-time (HMAC-MD4), and secure-challenge (HMAC-MD4) — matching whatever your `server.cfg` uses
- **Blackbox-style multi-target** — Prometheus passes `?target=<name>` per scrape, one exporter serves all servers
- **GeoIP player tracking** — optional `/geo` endpoint resolves player IP addresses to country/city via ip-api.com (free, cached 24 h)
- **Match data endpoint** — `/match` exposes per-player frags, spectator status, and match metadata (timelimit, fraglimit, teamplay)
- **Zero-downtime config reload** — send `SIGHUP` or `POST /-/reload` without restarting
- **CLI test interface** — query any server from the command line, human-readable or raw Prometheus format
- **Retry logic** — configurable retry count and per-attempt timeout on every RCON scrape
- **No external HTTP framework** — HTTP server is built on `asyncio.start_server` from the standard library
- **Python 3.11+** — uses `tomllib` (stdlib), fully compatible with 3.13

---

## Architecture

```
                    ┌─────────────────────────────────────────────┐
                    │              xonotic-exporter                │
                    │                                              │
  Prometheus ───────►  HTTP :9260                                  │
  scrapes             │   GET /metrics?target=vehicles  ───────────►  UDP RCON ──► Xonotic srv A
  /metrics?target=N   │   GET /metrics?target=dm        ───────────►  UDP RCON ──► Xonotic srv B
                    │   GET /geo?target=vehicles        ───────────►  ip-api.com  │
                    │   GET /match?target=vehicles      ───────────►  UDP RCON    │
                    │   POST /-/reload                             │              │
                    │   GET  /-/healthy                            │              │
                    │                                              │
                    │  TOML config  ◄── SIGHUP reloads on disk     │
                    └─────────────────────────────────────────────┘
```

### How a single scrape works

```
Prometheus                 xonotic-exporter                Xonotic server
    │                            │                               │
    │  GET /metrics?target=dm    │                               │
    │───────────────────────────►│                               │
    │                            │  UDP: ping packet             │
    │                            │──────────────────────────────►│
    │                            │  UDP: getchallenge (mode 2)   │
    │                            │──────────────────────────────►│
    │                            │          challenge token      │
    │                            │◄──────────────────────────────│
    │                            │  UDP: RCON "sv_public\0status 1"
    │                            │  (HMAC-MD4 signed)            │
    │                            │──────────────────────────────►│
    │                            │          pong                 │
    │                            │◄──────────────────────────────│
    │                            │          RCON response chunks  │
    │                            │◄──────────────────────────────│
    │                            │  parse → build Prometheus reg  │
    │    200 OK + metrics text   │                               │
    │◄───────────────────────────│                               │
```

Both the ping and the RCON query are sent concurrently. The parser uses an adaptive wait: it reads the first chunk, notes the RTT, then waits up to `2× RTT` for additional packets (Xonotic responses can span multiple UDP datagrams).

---

## Requirements

- Python **3.11** or later (3.12 / 3.13 fully supported)
- [`xrcon`](https://pypi.org/project/xrcon/) ≥ 0.1 — Xonotic RCON packet helpers
- [`prometheus-client`](https://pypi.org/project/prometheus-client/) ≥ 0.20

Both dependencies are installed automatically via `pip`.

---

## Installation

### From source (recommended)

```bash
# 1. Create a dedicated virtualenv
python3 -m venv /opt/xonotic_exporter/venv

# 2. Install the package and its dependencies
/opt/xonotic_exporter/venv/bin/pip install .

# 3. Verify
/opt/xonotic_exporter/venv/bin/xonotic-exporter --help
```

### From a zip archive

```bash
unzip Xonotic-Exporter-v1_0.zip
cd xonotic-exporter
python3 -m venv /opt/xonotic_exporter/venv
/opt/xonotic_exporter/venv/bin/pip install .
```

The `xonotic-exporter` command will now be available inside the venv.

---

## Configuration

Copy the example config and edit it:

```bash
sudo mkdir -p /etc/xonotic_exporter
sudo cp examples/xonotic_exporter.toml /etc/xonotic_exporter/xonotic_exporter.toml
sudo $EDITOR /etc/xonotic_exporter/xonotic_exporter.toml
```

### Full config reference

```toml
# /etc/xonotic_exporter/xonotic_exporter.toml

[exporter]
host = "0.0.0.0"   # bind address; use "127.0.0.1" to listen locally only
port = 9260        # Prometheus scrapes http://<host>:9260/metrics?target=<name>

# Add one [[servers]] block per Xonotic game server.
[[servers]]
name          = "vehicles"      # unique label; used as the Prometheus `instance` label
host          = "localhost"     # hostname or IP of the game server
port          = 26010           # game server UDP port (default 26000)
rcon_password = "CHANGEME"      # must match rcon_password in server.cfg
rcon_mode     = 2               # see RCON modes section below

[[servers]]
name          = "deathmatch"
host          = "game.example.com"
port          = 26000
rcon_password = "CHANGEME"
rcon_mode     = 2
```

#### `[exporter]` options

| Key | Default | Description |
|-----|---------|-------------|
| `host` | `"0.0.0.0"` | Address the HTTP server binds to |
| `port` | `9260` | Port the HTTP server listens on |

#### `[[servers]]` options

| Key | Default | Required | Description |
|-----|---------|----------|-------------|
| `name` | — | yes | Unique server identifier; becomes the `instance` Prometheus label |
| `host` | — | yes | Hostname or IP of the Xonotic game server |
| `port` | `26000` | no | UDP port of the game server |
| `rcon_password` | `""` | yes | RCON password (matches `rcon_password` in `server.cfg`) |
| `rcon_mode` | `2` | no | RCON authentication mode: `0`, `1`, or `2` |

### Validating your config

```bash
xonotic-exporter validate /etc/xonotic_exporter/xonotic_exporter.toml
# OK — 2 server(s) configured:
#   vehicles              localhost:26010  rcon_mode=2 (secure-challenge (MD4))
#   deathmatch            game.example.com:26000  rcon_mode=2 (secure-challenge (MD4))
```

---

## Running the exporter

### As a systemd service (recommended for production)

```bash
# Create a dedicated system user
sudo useradd -r -s /sbin/nologin monitoring

# Install the unit file
sudo cp examples/xonotic_exporter.service /etc/systemd/system/

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable --now xonotic_exporter

# Check status
sudo systemctl status xonotic_exporter
```

The unit file already includes hardening options (`NoNewPrivileges`, `ProtectHome`, `ProtectSystem=strict`, capability stripping).

**Reload config without restarting the process:**

```bash
sudo systemctl reload xonotic_exporter
# or equivalently:
curl -s -XPOST http://localhost:9260/-/reload
```

### Manually

```bash
/opt/xonotic_exporter/venv/bin/xonotic-exporter serve \
    /etc/xonotic_exporter/xonotic_exporter.toml

# Override listen address/port at runtime:
xonotic-exporter serve config.toml --host 127.0.0.1 --port 9261
```

---

## CLI reference

The exporter ships with three sub-commands.

### `serve` — start the HTTP server

```
xonotic-exporter serve CONFIG.toml [--host HOST] [--port PORT]
```

| Flag | Description |
|------|-------------|
| `CONFIG.toml` | Path to the TOML configuration file |
| `-l`, `--host` | Override the listen host from config |
| `-p`, `--port` | Override the listen port from config |

### `query` — scrape one server and print results

Use this to test connectivity and inspect what Prometheus would receive.

**From a config file (by server name):**

```bash
xonotic-exporter query vehicles --config /etc/xonotic_exporter/xonotic_exporter.toml
```

**Ad-hoc (no config file needed):**

```bash
xonotic-exporter query --host localhost --port 26010 --password secret --mode 2
```

**Raw Prometheus exposition format:**

```bash
xonotic-exporter query vehicles --config config.toml --prometheus
```

**With verbose (debug) logging:**

```bash
xonotic-exporter -v query --host localhost --port 26010 --password secret
```

**Example human-readable output:**

```
  RCON mode : secure-challenge (MD4)
────────────────────────────────────────────────────────────────────
  Server   : vehicles
  Hostname : My Awesome Xonotic Server
  Map      : warfare
  Public   : yes (sv_public=1)
────────────────────────────────────────────────────────────────────
  Ping     : 2.3 ms
  CPU      : 1.4 %
  Lost     : 0.0 %
  Offset   : avg=0.02ms  max=1.20ms  sdev=0.10ms
────────────────────────────────────────────────────────────────────
  Players  : 4/16
    Active      : 3
    Spectators  : 1
    Bots        : 0
────────────────────────────────────────────────────────────────────
```

#### `query` flags

| Flag | Default | Description |
|------|---------|-------------|
| `SERVER_NAME` | — | Named server from config (requires `--config`) |
| `--config` | — | Path to TOML config file |
| `--host` | — | Ad-hoc server hostname |
| `--port` | `26000` | Ad-hoc server UDP port |
| `--password` | `""` | Ad-hoc RCON password |
| `--mode` | `2` | Ad-hoc RCON mode (0, 1, or 2) |
| `--prometheus` | false | Print raw Prometheus text instead of human-readable output |
| `--retries` | `3` | Number of scrape attempts before giving up |
| `--timeout` | `5.0` | Per-attempt timeout in seconds |

### `validate` — check a config file

```bash
xonotic-exporter validate /etc/xonotic_exporter/xonotic_exporter.toml
```

Exits with code `0` on success, `1` on any config error.

---

## HTTP endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/` | `GET` | HTML index page listing all configured servers with links |
| `/metrics?target=<name>` | `GET` | Prometheus metrics for the named server |
| `/geo?target=<name>` | `GET` | Player geolocation metrics (country, city, lat/lon) |
| `/match?target=<name>` | `GET` | Match data: frags, spectators, timelimit, fraglimit, teamplay |
| `/-/reload` | `POST` | Reload config from disk; returns `200` on success, `500` on failure |
| `/-/healthy` | `GET` | Liveness probe — always returns `200 OK` |

---

## Prometheus scrape configuration

The exporter uses the [blackbox exporter pattern](https://prometheus.io/docs/guides/multi-target-exporter/): Prometheus sends a `?target=<name>` query parameter, and the exporter queries that specific server on demand. This means one exporter process and one HTTP port serves all your game servers.

Copy `examples/prometheus.yml` or add this to your existing `prometheus.yml`:

```yaml
scrape_configs:
  - job_name: "xonotic"
    scrape_interval: 15s
    scrape_timeout:  10s

    relabel_configs:
      # Pass the server name as ?target=
      - source_labels: [__address__]
        target_label:  __param_target
      # Use the server name as the instance label in Grafana
      - source_labels: [__param_target]
        target_label:  instance
      # Point all scrapes at the exporter
      - target_label: __address__
        replacement:  "127.0.0.1:9260"

    static_configs:
      - targets:
          - "vehicles"       # must match a [[servers]] name in the TOML
          - "deathmatch"
```

The `relabel_configs` block is required — without it, Prometheus would try to scrape the server name directly as an HTTP address.

For GeoIP and match data, add separate scrape jobs pointing at `/geo` and `/match`:

```yaml
  - job_name: "xonotic_geo"
    metrics_path: /geo
    scrape_interval: 30s
    # ... same relabel_configs and targets as above ...

  - job_name: "xonotic_match"
    metrics_path: /match
    scrape_interval: 15s
    # ... same relabel_configs and targets as above ...
```

---

## Metrics reference

### Core server metrics (`/metrics`)

All metrics carry an `instance` label set to the server name from the TOML config.

| Metric | Type | Description |
|--------|------|-------------|
| `xonotic_up` | Gauge | `1` if the server was reachable on this scrape, `0` otherwise |
| `xonotic_sv_public` | Gauge | Value of the `sv_public` cvar (`1` = listed on master server) |
| `xonotic_ping_seconds` | Gauge | Round-trip time to the server in seconds |
| `xonotic_timing_cpu_percent` | Gauge | Server CPU usage % (from `status 1`) |
| `xonotic_timing_lost_percent` | Gauge | Packet loss % (from `status 1`) |
| `xonotic_timing_offset_avg_ms` | Gauge | Average timing offset in milliseconds |
| `xonotic_timing_offset_max_ms` | Gauge | Maximum timing offset in milliseconds |
| `xonotic_timing_offset_sdev_ms` | Gauge | Standard deviation of timing offset in milliseconds |
| `xonotic_players_active` | Gauge | Number of active (scoring) players |
| `xonotic_players_spectators` | Gauge | Number of spectators |
| `xonotic_players_bots` | Gauge | Number of bots |
| `xonotic_players_total` | Gauge | Total connected (active + spectators + bots) |
| `xonotic_players_max` | Gauge | Maximum player slots on the server |

### Player geolocation metrics (`/geo`)

Labels: `instance`, `ip`, `port`, `slot`, `name`, `country`, `city`, `lat`, `lon`, `ping`.

| Metric | Description |
|--------|-------------|
| `xonotic_player_info` | Always `1`; carries player identity labels (name, country, city) |
| `xonotic_player_geo` | Always `1`; carries geolocation labels including `lat` and `lon` |
| `xonotic_player_ping` | Per-player ping to the server in milliseconds |
| `xonotic_player_packetloss_percent` | Per-player packet loss percentage |

Private/loopback IPs are never resolved or emitted. GeoIP results are cached for 24 hours using ip-api.com's free batch endpoint (up to 100 IPs per request, 15 req/min rate limit).

### Match data metrics (`/match`)

| Metric | Labels | Description |
|--------|--------|-------------|
| `xonotic_match_timelimit_seconds` | `instance` | Time limit converted to seconds (timelimit × 60) |
| `xonotic_match_fraglimit` | `instance` | Frag limit for the current match |
| `xonotic_match_teamplay` | `instance` | Teamplay mode (`0` = free-for-all) |
| `xonotic_match_info` | `instance`, `map` | Always `1`; carries current map name as a label |
| `xonotic_match_player_frags` | `instance`, `name`, `slot` | Current frag count per player |
| `xonotic_match_spectator` | `instance`, `name`, `slot` | `1` if the player is spectating |
| `xonotic_match_player_connection` | `instance`, `name`, `slot`, `ping`, `packetloss`, `time_seconds` | Always `1`; carries connection stats as labels |

---

## RCON modes explained

Xonotic supports three RCON authentication modes. The mode in your exporter config **must match** the `rcon_secure` / `rcon_restricted` setting in each game server's `server.cfg`.

| Mode | Name | `server.cfg` | Security | Notes |
|------|------|--------------|----------|-------|
| `0` | nonsecure | `rcon_restricted 0` | Low — password sent in plaintext | Avoid in production; use only on localhost |
| `1` | secure-time | `rcon_secure 1` | Medium — HMAC-MD4 signed with current timestamp | Requires clocks to be in sync (within a few seconds) |
| `2` | secure-challenge | `rcon_secure 2` | High — HMAC-MD4 signed with a server-issued challenge | **Recommended.** Not susceptible to replay attacks |

In mode `2` (the default), the exporter first requests a challenge token from the server, then signs the RCON command with `HMAC-MD4(password, challenge + command)`. This exchange happens automatically before every RCON command.

---

## Project structure

```
xonotic-exporter/
├── pyproject.toml                  # build config, dependencies, entry point
├── examples/
│   ├── xonotic_exporter.toml       # annotated example configuration
│   ├── prometheus.yml              # ready-to-use Prometheus scrape config
│   └── xonotic_exporter.service    # hardened systemd unit file
└── xonotic_exporter/
    ├── __init__.py                 # package version
    ├── cli.py                      # argument parser + serve/query/validate sub-commands
    ├── config.py                   # TOML loader and dataclasses (ExporterConfig, ServerConfig)
    ├── rcon.py                     # async UDP protocol: ping, challenge, RCON send/receive
    ├── metrics_parser.py           # state-machine parser for sv_public + status 1 output
    ├── prometheus.py               # per-scrape CollectorRegistry builders
    ├── server.py                   # asyncio HTTP server, routing, reload, GeoIP/match handlers
    └── geoip.py                    # async GeoIP cache backed by ip-api.com batch API
```

### Module responsibilities

`cli.py` is the entry point. It parses arguments and dispatches to one of three sub-commands: `serve`, `query`, or `validate`. The `serve` command calls `run_server()` in `server.py`, which creates an `asyncio` HTTP server that handles every incoming request.

`rcon.py` owns all network I/O. It implements `_XonoticProtocol`, a low-level `asyncio.DatagramProtocol` that handles ping, challenge, and RCON packet framing. The public `scrape_server()` coroutine opens a UDP endpoint, fires ping and RCON commands concurrently, and retries up to the configured limit.

`metrics_parser.py` contains `XonoticMetricsParser`, a state machine that processes the raw bytes returned by `sv_public` and `status 1` line by line. It handles Xonotic color codes (`^1`, `^x1A2`), IPv4, IPv6, and bot clients.

`prometheus.py` builds a fresh `CollectorRegistry` on every scrape request — this is the correct multi-target pattern, as it avoids any shared global metric state between servers.

`geoip.py` maintains a long-lived, process-wide IP → GeoResult cache. Cache entries expire after 24 hours. Lookups for private and loopback addresses are skipped entirely without a network call.

---