Skip to content

Outage runbook (integrated)

Follow top to bottom. Each step tells you what layer failed and points to deep FAQ entries.

Decision tree

flowchart TD
  A[Something broken] --> B{Whole house no internet?}
  B -->|Yes| C[NET-001 PWR-* UDM/NBN]
  B -->|No| D{Works on LAN IP 192.168.2.203?}
  D -->|No| E[NAS power pool apps NAS-* DKR-*]
  D -->|Yes| F{Only public URL fails?}
  F -->|Yes| G[TUN-* CFL-* cloudflared]
  F -->|No| H{403 or login loop?}
  H -->|403 from overseas| I[CFL-* block no aus]
  H -->|Login| J[AUTH-* auth stack]
  H -->|Other| K[Service FAQ from matrix]

Step 1 — Whole house offline?

Test: Phone on Wi‑Fi → open google.com. PC on ethernet → same.

Result Meaning Do this
Nothing works WAN or router NET-001 → When internet is broken → power-cycle NBN then UDM
Only one device Client issue NET-002
Wi‑Fi bad, ethernet OK AP / signal NET-007, UNI-*

Check status: status.saxobroko.com only covers public Saxo_Broko sites — not homelab tunnel apps.

Step 2 — LAN vs public

Pick one broken URL from service matrix.

Test How Interpretation
LAN Same device on home Wi‑Fi → https://192.168.2.203 or app LAN port If LAN fails, problem is NAS/network, not Cloudflare
Public Mobile data (in Australia) → https://stream.saxobroko.com etc. If LAN works but public fails → tunnel/DNS/WAF

LAN works, public fails

  1. status.saxobroko.com — public sites only
  2. TrueNAS → Apps → cloudflared running? → TUN-* · restart cloudflared
  3. Cloudflare dashboard (Vaultwarden login) → Zero Trust → Tunnels → connected?
  4. WAF event log → false block? → CFL-*

Both fail

Skip Cloudflare — fix NAS first.

Step 3 — NAS down

Pool DEGRADED or FAULTED

Stop. Do not replace disks blindly. NAS-* · server-when-pool-degraded

Check Where FAQ
Power / lights Physical NAS PWR-*
Ping 192.168.2.203 PC terminal NET-003
Pool ONLINE TrueNAS → Storage NAS-*, STG-*
Disk space TrueNAS dashboard STG-*
Docker apps TrueNAS → Apps DKR-*

Recovery order after power outage: PWR-* → UDM → NAS (wait 3 min) → verify pool → start cloudflared → test one public URL.

Step 4 — One app down, NAS healthy

App First action Deep FAQ
Jellyfin Restart app in TrueNAS JFY-*, DKR-*
*arr / qBit Check disk space + queue ARR-*, QBT-*
Authentik Check app logs; affects all SSO AUTH-*
Vaultwarden Fix before anything else password-related VLT-*
Homepage Tile red but app up = probe config DSH-*

Step 5 — Auth / 403 / Access

Symptom FAQ Doc
403 outside AU CFL-* Expected for homelab
403 in AU CFL-*, TUN-* WAF or app down
Cloudflare Access loop AUTH-* Auth stack
Authentik 502 AUTH-*, DKR-* Authentik container

Step 6 — Escalation

Situation Who / what
Pool degraded, SMART FAIL Technical help — read-smart-errors
Data loss feared BAK-* before any wipe
Saxon unavailable LIF-* · Ryan inherits tech per will
Money / mortgage urgent FIN-* · Finance guide