GitHub - ninoxAI/nightwatch: Open-source, local-first, read-only AI SRE: clusters alert storms, investigates root cause over your live systems, proposes human-gated fixes.

The open-source, read-only AI SRE.
ninoxAI turns alert storms into incidents, investigates root cause over your live systems, and proposes human-approved fixes — without ever touching production.

Quickstart · AI SRE · Demo lab · Docs · Discord

Your monitoring tells you something broke. It pages you at 3am with fifty alerts for one outage and leaves the hard part to you:

What broke, why did it break, and what should we do next?

ninoxAI is a thin, local-first, monitoring-agnostic AI SRE layer that answers that question. It sits above Checkmk, Prometheus, Icinga2, Zabbix, webhooks, Docker, Kubernetes, AWS, Grafana, GitHub, Git and plain VMs, and:

• 🌊 Turns alert floods into incidents — one incident per outage, "confirmed by N tools", instead of one page per symptom.
• 🔇 Finds the noisy checks — flapping, over-sensitive, never-actioned — with evidence.
• 🤖 Investigates root cause — a tool-calling AI agent reads your live systems and forms a root-cause hypothesis.
• 🧰 Proposes classified fixes — copy-pasteable, ranked by risk and blast radius, for a human to gate.

🔒 Read-only by design

ninoxAI observes, reasons, and recommends — it never executes anything. No commands run, no alerts acked, no thresholds changed, no write-back to production. Every fix is a copyable artifact a human approves. Gated, governed remediation is on the roadmap; unconditional auto-execute is not.

⚡ Quickstart

Try it in 60 seconds — no LLM, no API keys, fully offline:

cp .env.example .env          # set NINOXAI_SECRET_KEY (one-liner is in the file)
docker compose up --build     # → http://127.0.0.1:8765

# No live monitoring? Watch it triage synthetic alert noise:
docker compose exec ninoxai ninoxai generate-mocks
docker compose exec ninoxai ninoxai import data/mock_alerts.json
docker compose exec ninoxai ninoxai reprocess
# → /recommendations now shows reasoned threshold + flapping fixes

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\Activate.ps1
pip install -e ".[dev,embeddings]"

python -m ninoxai generate-mocks
python -m ninoxai import data/mock_alerts.json
python -m ninoxai reprocess
python -m ninoxai serve            # → http://127.0.0.1:8765

Then light up the AI SRE: point ninoxAI at a tool-calling LLM (Anthropic / OpenAI / Mistral / a local Ollama) and connect your systems — either directly or via a ninox runner (below). The full end-to-end scenario — real monitoring tools and live investigator capabilities (Docker/Kubernetes/host/AWS/Grafana/GitHub) against a genuinely failing workload — lives in lab/.

🔭 How it works

ingest → normalize → cluster → score noise → recommend → dashboard
                                                  ↓
                       agentic, read-only root-cause investigator

Cross-tool correlation: the same fault fires in every tool. The Incidents view groups clusters that share (host, severity, time-window) into one incident — "confirmed by N tools" — read-only, no merge.

🤖 The AI SRE investigator

ninoxAI's standout capability. A tool-calling LLM drives a typed allowlist of read-only capabilities (a ReAct loop on native function-calling — reason → act → observe), builds a root-cause hypothesis from live evidence, and proposes classified fixes a human approves.

• Every action is classified read_only · reversible · irreversible + a scope (blast radius). Unknown coerces to irreversible — never silently auto.
• Pre-grounded: the agent starts with a compact brief of your environment, so it diagnoses instead of rediscovering.
• Hardened: untrusted logs/diffs are injection-shielded, secrets are one-way scrubbed, and a grounding gate caps confidence when claims aren't backed by evidence.

Run it live-streaming in the agent console (/agent) or from the CLI. → Investigator internals

🦉 Distributed ninoxes — the agent's eyes, anywhere

The agent can investigate systems it can't reach directly. A ninox is a thin, outbound-only runner that lives inside one environment (cluster, VPC, on-prem segment), holds that environment's credentials locally, and dials home to the brain — no inbound firewall hole. It advertises a read-only capability surface the brain calls as if local.

   ┌────────────────────┐                         ┌────────────────────┐
   │    ninoxAI brain    │  ◀── outbound only ───  │     ninox runner    │
   │  dashboard · API    │     (the ninox dials    │  inside k8s/Docker/ │
   │  incidents · RCA    │      home; no inbound    │  AWS/on-prem/VM     │
   │  AI SRE investigator│      firewall hole)      │  credentials stay   │
   └────────────────────┘  ◀── read-only evidence  │  local              │
                                                     └────────────────────┘

Capabilities self-select by environment — one binary, the right tools for the box it lands on. Connected ninoxes show up in the Parliament of Owls (/parliament). → Deployment & on-prem

🔌 Connectors

All adapters are read-only — no ack, no downtime, no write-back. Configured in the UI (/connections), credentials Fernet-encrypted.

Want to teach the AI SRE to read your stack (Jira, Sentry, Postgres…)? Point it at any MCP server, write a Python capability plugin, or expose tools via the runner protocol — every external tool runs through the same safety shell (namespaced, injection-scanned, classification-coerced). → Extending capabilities

🧠 LLM providers

Default is template — fully offline: no LLM, no network, no API keys, no tracking. It works out of the box for summaries/recommendations but deliberately can't drive the agent (that needs tool-calling). Pick a remote per role — a cheap model for high-volume summaries, a strong one for the rare investigation:

Redaction + secret-scrubbing run before every remote call — hostnames, IPs, UUIDs, emails, paths become deterministic placeholders, restored only in proposed commands; credentials are one-way scrubbed and never returned. → Technical architecture

🛠️ Development

Full CLI reference, test setup, and lint rules live in docs/development.md.

🤝 Contributing

Every contributor is an Owl. 🦉 Pull requests, connector adapters, capability providers, and bug reports are all welcome — see CONTRIBUTING.md.

Community: Join the parliament on Discord.

📜 License

ninoxAI is fully open source under the Apache License 2.0 — free to use, self-host, fork, and build on, in open or closed projects alike.

The owl observes; the human decides. 🦉