Self-diagnosing server
When a service goes down, the server collects evidence, an LLM diagnoses it, and I get one message with the cause and the fix.
Privilege separationRedactionLLMsystemdUptime Kuma
Problem
An alert that says only "DOWN" at 3 a.m. starts a long search through logs. I wanted the first ten minutes of that search done before I even look at my phone.
Architecture
- 01 Monitor: DOWN
- 02 Re-check after 2 min
- 03 Whitelisted evidence + redaction
- 04 LLM with no tools
- 05 Push: cause + fix
Key decisions
- Privilege separation: a deterministic collector runs a fixed whitelist of read-only commands; the config can only select built-in checks, never commands.
- Secrets are redacted before anything leaves the collector (tokens, passwords, e-mail addresses, push topics, long blobs), and the redaction has tests.
- The model runs as an unprivileged user with no tools. It diagnoses only; it never executes a fix.
- Flap protection, rate limits and its own monthly cost cap inside the shared AI budget.
What I learned
- Letting an LLM help with operations is safe when the system around it is designed so that it cannot do anything.
- Good evidence matters more than a bigger model: filtering logs to warnings and errors halved the input and improved the answers.