16 October 2026 · Self-hosting · Observability · Alerting · 9 min read

Watching a one-box server

My home lab runs every service on a single machine. This post covers what I chose to watch on it, the small stack that does the watching, and the outage that showed me which alerts were still missing.

Fewer tools, more signal

I started with the usual self-hosting pair: Uptime Kuma for uptime checks and Dozzle for live container logs. Both are good tools, and I removed both. Each tool is another container, another UI, another login and another thing to update. One log and metrics platform plus one external service now cover what they did, plus a lot they couldn't.

A monitor can't report its own death

Uptime Kuma ran on the machine it was watching. If that machine lost power, lost its network or booted into another OS, the monitor went down with it and said nothing. Every check that answers "is it up?" has to live somewhere else, so the design splits in two:

The heartbeat is the part I like most. A cron job runs the platform's preflight check (data disk mounted, Docker up, edge proxy healthy) and pings the external service only if everything passes. Silence means "down, offline or unhealthy", and the external side raises the alert. Because the server only ever pings on success, this works with any heartbeat provider.

The external HTTPS check covers the rest of the path a visitor uses: DNS, the router, the proxy and TLS. Between them, the two outside checks answer the one question the inside can't.

What to watch on a small server

Before writing any rule I made a checklist of what can actually go wrong on one machine that serves websites:

1. Reachability from outside. DNS, router, proxy and TLS together: one external URL check.
2. Liveness of the box. A heartbeat gated on a health check.
3. Capacity. Disk, memory and load, with thresholds you'd act on.
4. Hardware health. Kernel I/O errors, SMART warnings, temperature.
5. The network itself. Link drops, a changed public IP, a failing dynamic-DNS job.
6. Security signals. SSH logins from places they shouldn't come from.
7. App health. The 5xx rate per site, read from the proxy, so it works for every app without touching its code.
8. Containers. Unhealthy, crash-looping or stopped.
9. Certificates. Renewal failures from the inside, days left from both sides.
10. The monitoring itself. An alert when logs stop arriving.

The pipeline

• Vector collects everything. It tags each container's lines with its project and service, parses JSON log lines into fields, and sends the proxy's access log to its own stream, so "which site is returning 5xx right now" is one query.
• The collector has its own Docker API proxy that allows reading logs and nothing else. The edge proxy's own API proxy stays without log access.
• Everything runs on an internal network with no internet access. Telemetry can reach OpenObserve and nothing else.

Noise first

The first hour of data was mostly about the monitoring itself. OpenObserve logs every ingest request it receives, and Vector shipped those lines straight back in: a loop of about 32,000 lines an hour that said nothing. Every container start and stop also produced around ten kernel lines about virtual network interfaces. Both are filtered at the collector, before they cost storage or attention.

Container metrics without cAdvisor

Host metrics are easy. Per-container CPU and memory are the hard part. The usual answer, cAdvisor, wants privileged access to the host and the raw Docker socket, and this platform doesn't give that to anything.

Docker's built-in metrics endpoint looked like the clean alternative, but it's a trap. The daemon opens that listener before it creates its own networks. Bind it to an internal network address and Docker refuses to start at boot. The remaining choices are localhost, which containers can't reach, or every interface.

So it's a 40-line shell script, run every minute by cron. It reads a handful of fields per container, writes them in Prometheus format to a file, and node-exporter's textfile collector picks the file up:

docker_container_running{name="traefik",stack="traefik"}   1
docker_container_healthy{name="traefik",stack="traefik"}   1
docker_container_restarts{name="traefik",stack="traefik"}  0
docker_container_cpu_percent{name="openobserve"}           8.09
docker_container_memory_bytes{name="openobserve"}          705691852

No new container, no extra privileges, and it moves to any other host unchanged. It also skips the container's environment, where secrets can live.

Certificates came almost free. Traefik can expose Prometheus metrics on an internal port that is never published, and one of them is the expiry date of every certificate it serves. That covers client domains too, which the external check doesn't watch.

Alerts as code

Every rule is a JSON file in the platform's Git repository, applied through OpenObserve's API by an idempotent script. Nothing is created by clicking around the UI, so a fresh install gets every rule back with one command.

Each rule has a silence period, so a disk that stays full sends one message every six hours instead of one every ten minutes. A rule that fires all the time just trains you to ignore it.

Dashboards, SLOs and retention, also as code

The overview dashboard is generated by a Python file and pushed through the same API, and every panel query was run and checked before it shipped. From top to bottom it shows health tiles, host trends (CPU, memory, disk, I/O, load, temperature), traffic by status class, top sites, error tables, p95 response time and bandwidth per site, SSH activity by source, the paths scanner bots probe, and per-container CPU and memory against each container's limit.

One panel was cut on purpose. A host network chart would have needed node-exporter on the host's network, which breaks the platform's isolation rules for one chart. Bandwidth per site, from the proxy's access log, answers the more useful question anyway.

There's also one SLO: each site answers without a server error 99.5 % of the time over 30 days. It's computed from the proxy's access log and grouped by hostname, so every new site gets its own error budget the day it goes live, with no setup and no change to its code.

Retention follows value. Security and hardware logs stay 90 days, because you only look for them after something went wrong. Request logs stay 30 days. Container logs are the noisiest and least useful later, so they stay 14.

Small details that paid off

• Credentials never leave a throwaway container. The API helper that applies rules and dashboards starts a short-lived container on the internal network, loads the credentials and the Discord webhook from root-owned files inside it, and prints only the HTTP status. No secret ever shows up in a terminal, a log or the repository.
• OpenObserve refuses private addresses as alert destinations. That's a guard against server-side request forgery, and it meant the placeholder destination used before Discord was set up had to be a real public URL that goes nowhere.
• Field names are lowercase. Traefik writes RequestHost, OpenObserve stores requesthost. Every query and alert was written against the stored names after a few silent zero-result queries.
• Memory, measured. OpenObserve is the only real consumer. Vector, node-exporter and the collector's API proxy use 7 to 25 MB each.

What I left out, on purpose

• The external monitor's server agent. It would duplicate node-exporter, and avoiding duplicate tools is the whole point.
• OpenObserve's infrastructure view. It expects OpenTelemetry host data. node-exporter already covers the host.
• The built-in AI assistant. It needs an external model API key, for features I don't need yet.
• Saved log views as code. OpenObserve stores them as UI state, so they're the one thing created by hand.

What's next

• A dedicated ingest-only account for the collector, instead of an admin login
• Traces, once an app is instrumented with OpenTelemetry
• Real-user monitoring on public frontends: page speed and JavaScript errors from actual visitors
• An external monitor for every new site, added the day it goes live

One platform inside, one service outside, eighteen rules in Git and less memory than a browser tab. The best test so far was a real outage. It showed which alerts were missing, and those rules are in the repository now.