GitHub - astraverse-io/KubeAstra: AI-powered Kubernetes troubleshooting via chat or MCP

GitHub

25 min read Original article β†—

CI License: Apache 2.0 Python 3.11+ Next.js 16 MCP compatible PRs welcome

πŸ“¬ Subscribe for release updates β€” new versions, no spam

Your clusters are talking. This assistant helps you listen.

An AI-powered Kubernetes troubleshooting assistant that lets teams investigate, diagnose, and resolve cluster issues through natural language β€” as a signed macOS app (brew install --cask kubeastra), a chat-based web UI deployed for a team, or directly inside your IDE (Cursor / Claude Desktop / VS Code via MCP).

Combines live kubectl access with pluggable LLM providers (Gemini, Claude, GPT, or a fully local Ollama model) for root-cause analysis that turns cryptic Kubernetes failures into clear answers and actionable fix commands.

See it in action

Watch the 90-second demo

β–Ά Watch the 90-second demo β€” Kubeastra walking through 7 real Kubernetes failures (CrashLoopBackOff, OOMKilled, ImagePullBackOff, stuck PVC, unschedulable pod, namespace-wide health, runbook generation).

Want to reproduce it locally? make demo spins up a kind cluster pre-seeded with seven broken workloads. See demo/README.md.


Why this exists

Every DevOps engineer has been here: a pod is crashlooping at 2 AM, and you're mentally chaining together kubectl get, kubectl describe, kubectl logs, cross-referencing events, checking resource limits, and Googling error messages β€” all while half asleep.

This tool handles that investigation loop for you:

  • Ask in plain English β€” "Why is payment-service crashing in production?"
  • Get root-cause analysis β€” not just logs, but AI-synthesized explanations of what's wrong and why
  • Receive fix commands β€” ready to run, with safety confirmations for write operations
  • Generate runbooks β€” so your team doesn't debug the same issue twice
  • Stay on your own infra β€” run entirely locally with Ollama, no data leaves your cluster

Key Features

πŸ”— Connect Any Cluster in Seconds

Four ways to connect β€” pick what fits your setup:

Mode How it works Best for
Auto-detect Reads your local ~/.kube/config and lists available contexts Local dev, minikube, kind, Docker Desktop
Kubeconfig upload Paste or upload a kubeconfig file, pick a context Remote clusters, CI-generated configs
SSH Enter host/user/password β€” kubectl runs on the remote node over SSH Air-gapped clusters, bare-metal kubeadm
In-cluster Mounts the ServiceAccount token automatically When deployed inside the cluster via Helm

Switch between clusters without restarting. Each session tracks its own connection.

πŸ” 52 Built-in Kubernetes Tools

Live cluster tools (~30) β€” pod/deployment/service inspection, event streams, multi-namespace discovery, rollout status, kubeconfig context switching, log retrieval with previous-container support, resource-graph topology, deployment-level investigation, namespace-wide health analysis, and safe write operations (delete, scale, restart, patch β€” all gated behind dry_run + confirmation_token).

AI analysis tools (~9) β€” error analysis with RAG-backed similarity search, curated fix playbooks for 11 error categories, AI-generated runbooks, cluster health reports, post-incident summarization, k8sgpt-style pattern matching, RAG knowledge-base search (kb_search).

Helm tools (5) β€” release listing, revision diffing, release inspection, availability check, guided investigation.

Alerts + observability (2) β€” Alertmanager fan-in (get_recent_alerts), Prometheus range/instant queries (prom_query).

GitOps / deployment-repo (3) β€” fetch files, list paths, and grep the deployment repo directly from investigation.

Remediation plans (3) β€” propose_remediation_plan, get_plan, execute_plan_step (multi-step fixes with per-step confirmation).

πŸ€– Agentic ReAct Investigation

Unlike single-shot "ask β†’ answer" tools, Kubeastra runs a multi-step ReAct loop β€” reasoning through complex failures autonomously:

You: Why is checkout-service down?

Agent reasoning:
  βœ“ find_workload β€” searching across all namespaces
  βœ“ investigate_pod β€” found CrashLoopBackOff in checkout-svc-7d4f9b
  βœ“ get_pods β€” checking Redis dependency β†’ ConnectionRefused
  βœ“ describe_pod β€” Redis pod Pending: unbound PVC

Root cause: PersistentVolumeClaim redis-data is unbound,
preventing Redis from starting, which cascades to checkout-service.

Each reasoning step is visible in real-time via the Investigation Trail β€” no black box. The agent answers listing questions in a single step and complex debugging in 2-3 steps, with a 90-second wall-clock safety timeout.

πŸ”§ One-Click Fix Execution

When the AI identifies a fix, you get a Review & Execute button with the exact commands:

  • Only write operations are suggested (delete pod, rollout restart, scale, patch) β€” never diagnostic commands you've already seen
  • Slide-to-confirm safety gate before any command runs
  • Button disappears after execution β€” no accidental re-runs
  • When no safe automated fix exists (e.g., "update your Helm values"), the card shows Manual Steps Required with numbered instructions instead

πŸ”€ Propose Fixes as Pull Requests

When your cluster is driven by GitOps (Argo / Flux), applying a fix straight to the live cluster fights your source of truth. Connect a GitHub repo and KubeAstra opens the fix as a pull request instead β€” the same change, reviewed in Git before anything reaches the cluster:

  • You review a real diff, not a promise β€” KubeAstra fetches the repo, finds the exact line for the change (replicas, an image tag, a resource limit, an env value), edits only that line with comments and formatting preserved, and shows you the one-line diff before anything is pushed
  • A human always merges β€” it opens the PR, labels it, and writes the diagnosis + evidence into the description; it can never merge the PR itself (enforced by a CI test that fails the build if a merge call is ever added)
  • Refuses rather than guesses β€” if it can't find exactly one place to make the change, it stops and says so instead of inventing a diff
  • Deterministic β€” no LLM in the write path β€” the edit is a surgical text-span replacement, so the diff can't hallucinate; opening a PR is recorded as its own audit event, never as a cluster mutation (a PR changes nothing yet)
  • Plain YAML + Kustomize repos on GitHub, via a fine-grained token (no GitHub App to install)

Off by default; enable with GITOPS_ENABLED=true and connect a repo in Settings.

πŸ‘₯ Collaborative Sessions

  • Shareable URLs β€” click Share to copy a session link (/chat/:sessionId). Anyone with the URL sees the full investigation history β€” including the root-cause card, fix commands, and evidence.
  • Investigation timeline β€” every ReAct step (tool call, thought, observation) renders as a real-time timeline, not simulated placeholders.
  • Session not found β€” invalid or expired shared links show a clear message instead of a blank page.
  • One-click post-mortems β€” generate a structured post-mortem (summary, timeline, root cause, impact, resolution, action items) from any investigation session via the API.

πŸ—ΊοΈ Visual Debugging Canvas

The resource graph is an interactive investigation surface, not just a topology diagram:

  • Health-aware nodes β€” pods, services, deployments, and ingresses colored by health status with pulsing red glow for degraded resources
  • Click-to-inspect β€” click any node to see full metadata in a detail panel
  • Hover tooltips β€” quick metadata preview (phase, restarts, IP, ports, replicas)
  • Edge labels β€” see relationships at a glance: "routes β†’", "selects β†’", "manages β†’"
  • MiniMap + zoom/pan β€” navigate large cluster topologies with ease

πŸ’¬ Two Ways to Use It

Web UI IDE / MCP Integration
Chat-based Next.js interface for team-wide troubleshooting Direct integration into Cursor, Claude Desktop, or any MCP client
Connect any cluster (auto-detect, kubeconfig upload, SSH) Debug without leaving your editor
Shareable session URLs with persistent chat history (SQLite) 52 tools available via stdio or HTTP MCP transport
Visual resource graph with click-to-inspect Same ReAct agent powers both surfaces

🚨 Alert-Driven Auto-Investigation

Point Alertmanager at KubeAstra and it auto-triages incoming alerts:

  • Webhook receiver β€” POST /api/v1/alerts/webhook accepts standard Alertmanager payloads
  • Auto-investigation β€” each firing alert kicks off a scoped ReAct investigation (by namespace / workload from labels)
  • Playbook-first routing β€” hits deterministic runbooks before consulting the LLM (faster, cheaper, more predictable)
  • Ranked notification β€” findings post back to Slack / PagerDuty (or any webhook) with root cause + suggested fix
  • Concurrency caps + LLM offload β€” burst-safe: alert storms don't exhaust the model quota

πŸ“š RAG Runbook Cache β€” Cached / Grounded / Cold

Every investigation runs through a 3-tier retrieval router before touching the model:

Tier Trigger Behavior
Cached Verified runbook match β‰₯ 0.92 similarity Return the runbook answer verbatim β€” sub-second, near-zero cost
Grounded Any relevant chunk β‰₯ 0.70 similarity Feed retrieved docs into the LLM prompt as context
Cold No relevant match Full ReAct investigation from scratch

Backed by Qdrant (self-hosted, no external SaaS) with a nightly ingestion CronJob for internal docs + configurable Git sources (URL-allowlisted). Every answer shows its citations β€” no black-box RAG.

🧠 Multi-Step Remediation Plans

Complex fixes get proposed as atomic multi-step plans instead of one-shot commands:

  • LLM proposes plan β†’ user reviews steps β†’ each step needs its own confirmation token
  • Dry-run first β€” every destructive step exposes its --dry-run=server output before real execution
  • Atomic step state β€” steps are pending / running / done / failed with concurrency-safe CAS transitions
  • Fail-safe β€” if step 2 fails, step 3 doesn't fire; users see the failure and decide next action
  • Cancellable β€” abandon a plan mid-execution without leaving orphaned state

πŸ’Ύ Per-User Conversation Memory

The assistant remembers your prior investigation context across turns:

  • Auto-captured β€” last 24h of namespace / pod / cluster mentions inform the next prompt
  • Prompt preamble β€” memory is stitched into the system message, not the user message (invisible but effective)
  • Bounded β€” 10 items per category, 5 rendered per turn, 24h max age, tenant-scoped
  • Redacted β€” secrets pattern-matched and stripped before persist

βœ‚οΈ Tool Result Summarization

Massive kubectl logs outputs get compressed intelligently:

  • Heuristic first β€” strip ANSI, dedupe, tag errors/warnings, keep head + tail + relevant context
  • LLM polish (optional) β€” condense multi-hundred-line describes into 3-line summaries
  • Preserved evidence β€” every summary keeps the raw underlying data as an expandable "show all" section

πŸ‘ Feedback β†’ Runbook Promotion

Thumbs-up on any answer promotes it into the cached-runbook tier:

  • Great answers become verified runbooks other users hit via the "cached" tier
  • Thumbs-down quarantines low-quality captures so they don't feed grounding
  • Auto-capture classifier (strict-JSON Gemini prompt) decides which answers qualify for consideration
  • Redaction runs before persist β€” secrets never enter the KB

πŸ” Remote Diagnostics (Air-Gapped Clusters)

For clusters where you can't ship the assistant into the cluster:

  • SSH + Ansible β€” runs a hardened, scoped diagnostic playbook against a bastion + target nodes
  • Read-only by default β€” no writes without confirmation tokens; no shell access, only structured facts
  • Egress-controlled β€” Helm ConfigMap + NetworkPolicy pin which nodes/subnets the runner can reach
  • Auditable β€” every remote command emitted to the audit log

πŸ”Œ Pluggable LLM Providers

Pick your LLM β€” Google Gemini (default, free tier available), Anthropic Claude (Opus / Sonnet / Haiku), OpenAI (GPT-4o, or any OpenAI-compatible endpoint like Azure OpenAI / vLLM / LiteLLM), or Ollama (fully local β€” your cluster data never leaves your network). Set LLM_PROVIDER to gemini, anthropic, openai, or ollama and provide the matching API key.

Running fully local with Ollama

Investigation memory embeds past investigations so new ones can recall them. That needs an embedding model, which is not the chat model β€” so pull it as well:

ollama pull llama3.1            # chat
ollama pull nomic-embed-text    # embeddings

Then point both at the local daemon:

LLM_PROVIDER=ollama
EMBEDDINGS_MODE=ollama

Without EMBEDDINGS_MODE=ollama, embeddings fall back to whatever the mode is set to β€” which for a hosted provider means investigation text leaves your machine even though chat does not. Setting it is what makes "nothing leaves the network" true end to end.

nomic-embed-text is the default; override it with EMBEDDINGS_MODEL if you prefer another. Whichever you choose has to be pulled β€” Ollama does not fetch it on first use, and an unpulled model surfaces as an embeddings error rather than as a missing-model message.

Note the two separate URLs: OLLAMA_BASE_URL points the chat provider at a daemon, OLLAMA_URL points embeddings at one. Set both if your Ollama is not on localhost.

πŸ›‘οΈ Safety First

  • Read-only by default β€” all kubectl commands are validated before execution
  • Explicit confirmation required for write operations (delete, scale, restart, patch) via slide-to-confirm
  • Full audit logging of every command executed
  • RBAC-aware β€” respects your existing Kubernetes permissions
  • Input validation β€” namespace/name/label-selector safety checks prevent injection
  • Session isolation β€” temp kubeconfig files are scoped per session with 0600 permissions, sanitized session IDs prevent path traversal, cryptographic session tokens prevent URL guessing
  • Command allowlist β€” the execute endpoint only accepts specific kubectl write prefixes; everything else is rejected

πŸš€ Deploy Anywhere

  • Local dev β€” docker-compose one-liner
  • Kind demo cluster β€” make demo spins up a broken cluster so you can see the tool work in 60 seconds
  • Production Helm chart β€” deploy into the same clusters it monitors
  • SSH multi-cluster β€” query any remote kubeadm cluster without copying kubeconfigs

Which feature is right for me?

Match your situation to the feature that solves it:

If you… The feature Enable with
Just want an AI copilot for kubectl diagnosis Agentic ReAct + 52 tools Defaults β€” no flags needed
Get paged by Alertmanager at 2 AM Alert-Driven Auto-Investigation ALERTMANAGER_WEBHOOK_ENABLED=true + set ALERT_WEBHOOK_TOKEN
Investigate the same failure modes repeatedly RAG Runbook Cache (Cached tier) RAG_ROUTER_ENABLED=true + πŸ‘ on great answers
Read 500-line describe and 10K-line log outputs Tool Result Summarization ENABLE_LOG_SUMMARIZATION=true
Perform multi-step fixes with strong safety gates Multi-Step Remediation Plans ENABLE_RECOVERY_OPERATIONS=true + REQUIRE_DESTRUCTIVE_CONFIRMATION=true (default)
Drive your cluster with GitOps and want fixes reviewed in Git, not applied live Propose Fixes as Pull Requests GITOPS_ENABLED=true + connect a GitHub repo in Settings
Have to repeat "the payments namespace" every prompt Per-User Conversation Memory On by default β€” persists 24 h per session
Want cluster investigation without shipping into the cluster Remote Diagnostics (SSH + Ansible) Configure remoteDiag block in Helm values
Have a team wiki that everyone should query first Grounded RAG Populate RAG_INGESTION_SOURCES and let the nightly CronJob index them
Need to see the AI's evidence before trusting an answer Synthesis Critic + Citations On by default whenever RAG is on
Run air-gapped and can't call Gemini Ollama provider LLM_PROVIDER=ollama β€” all inference stays local

Quick Start

Option 1: Install the macOS app

brew tap astraverse-io/tap
brew install --cask kubeastra

Or download the DMG from the latest desktop release.

Signed and notarized by Apple, so it opens without a Gatekeeper warning. Apple silicon and macOS 11+ β€” there is no Intel build. It uses the kubectl and kubeconfig already on your machine; nothing is installed into any cluster, and the app listens only on loopback.

Verify the download yourself if you like:

spctl -a -t open --context context:primary-signature KubeAstra_*.dmg

accepted / source=Notarized Developer ID is the answer you want.

On Linux and Windows, use kubeastra open (Option 3) β€” the same app, run from a source checkout. A Windows installer needs a code-signing certificate and is not built yet.

Option 2: Try the demo (60 seconds, no cluster needed)

Prerequisites: Docker Desktop, kind, kubectl

git clone https://github.com/astraverse-io/KubeAstra.git
cd KubeAstra
make demo

Spins up a local kind cluster with pre-broken workloads (CrashLoop, OOM, ImagePull, stuck PVC) and launches the web UI.

Open http://localhost:3300 and ask "what's broken in the demo namespace?".

The demo generates its own kubeconfig automatically β€” it does not touch your host's current kubectl context. See demo/README.md for full prerequisites and troubleshooting.

Option 3: Run it as a local app (no Docker)

Prerequisites: Python 3.11+, Node 20+, and a kubeconfig you already use.

git clone https://github.com/astraverse-io/KubeAstra.git
cd KubeAstra
npm run build:desktop --prefix ui/frontend   # once
pip install -e cli
kubeastra open

Starts a loopback-only backend, serves the UI from that same origin, and opens your browser. It reads the kubeconfig already on this machine β€” nothing is installed into any cluster, and nothing listens on an external interface.

Add --no-browser to start it and print the URL instead.

Option 4: Run locally against your own cluster

Prerequisites: a running Kubernetes cluster with kubectl access, and a Google Gemini API key (free tier) or Ollama running locally.

# 1. Configure the backend
cp ui/backend/.env.example ui/backend/.env
#    β†’ set GEMINI_API_KEY (or LLM_PROVIDER=ollama) in .env

# 2. Start via docker-compose (kubeconfig mounted read-only)
cd ui
docker compose up --build

# 3. Open http://localhost:3300

Option 5: Use via MCP (Cursor / Claude Desktop)

cd mcp
./setup.sh        # creates venv, installs deps, writes MCP config entry

Edit mcp/.env:

GEMINI_API_KEY=your-key-here          # or LLM_PROVIDER=ollama
ALLOWED_NAMESPACES=prod,staging,default

Restart your IDE β€” all 52 tools appear as MCP tools.

Option 6: Use the CLI

For terminal-first workflows: a thin HTTP + SSE client for the backend, published as a standalone Python package.

pipx install kubeastra                # or: cd cli && pip install -e .

# assuming the backend from Option 3 is running on localhost:8000
kubeastra ask "why is checkout-service crashlooping in production?"
kubeastra investigate --pod api-gateway --ns production
kubeastra doctor                      # health-check CLI + backend + kubeconfig

Config lives at ~/.config/kubeastra/config.toml. Point at a remote backend with:

kubeastra config set backend-url https://kubeastra.mycompany.com

Full command reference in cli/README.md.

Option 7: Deploy to Kubernetes via Helm

The chart pulls public images from ghcr.io/astraverse-io/kubeastra-{backend,frontend}, so there is nothing to build. It defaults to latest; pin 0.2.0 for a fixed version, or main to track every merge.

Baseline install β€” chat UI + backend, no advanced features:

helm upgrade --install kubeastra helm/kubeastra \
  --namespace kubeastra --create-namespace \
  --set secrets.geminiApiKey="YOUR_KEY" \
  --set secrets.kubeconfig="$(cat ~/.kube/config | base64 | tr -d '\n')"

Opt into every Phase 1 feature (RAG runbook cache, Qdrant, ingestion CronJob, Alertmanager webhook, remediation plans) with the bundled overlay:

helm upgrade --install kubeastra helm/kubeastra \
  --namespace kubeastra --create-namespace \
  -f helm/kubeastra/values-production.yaml \
  --set secrets.geminiApiKey="YOUR_KEY" \
  --set secrets.kubeconfig="$(cat ~/.kube/config | base64 | tr -d '\n')"

Never commit secrets. Put geminiApiKey, kubeconfig, alertWebhookToken, and qdrantApiKey in a gitignored values-secrets.yaml and pass it via a second -f flag β€” see docs/BEST_FEATURES_QUICKSTART.md for the full pattern.


How It Works

Every request flows through a fixed routing order β€” cheapest, most predictable paths first, LLM reasoning last:

  1. Connect your cluster β€” auto-detect your local kubeconfig, upload one, or enter SSH credentials. The connection is scoped to your session.
  2. Ask a question β€” "Why are pods in checkout-service not starting?"
  3. Memory injection β€” the last 24 h of your investigation context (namespaces, pods, cluster) is stitched into the prompt preamble so the assistant doesn't ask you to repeat yourself.
  4. Playbook-first check β€” if the question matches a deterministic runbook pattern (e.g. OOMKilled recovery), the runbook answers directly. No LLM call. Sub-second.
  5. RAG router β€” if no playbook matches, embed the query and search Qdrant across runbook, devops_doc, and session_memory collections:
    • Cached (verified runbook, similarity β‰₯ 0.92) β†’ return verbatim, sub-second, near-zero cost
    • Grounded (any doc, similarity β‰₯ 0.70) β†’ feed retrieved chunks into the LLM prompt as context
    • Cold (no relevant match) β†’ full ReAct from scratch
  6. ReAct investigation β€” the LLM reasons step-by-step: picks a tool β†’ executes it β†’ observes the result β†’ decides the next action. Runs up to 6 iterations with a 90-second wall-clock timeout. Auto-discovery via find_workload handles missing namespaces; large clusters use text-format parsing instead of JSON.
  7. Tool result summarization β€” 500-line describes and 10K-line log dumps get compressed via heuristic (ANSI strip, dedupe, error tagging, head+tail+context) with optional LLM polish before entering the LLM's context.
  8. Synthesis critic β€” a hallucination auditor cross-checks the final answer against the raw tool evidence. Claims not backed by evidence are flagged or removed.
  9. AI synthesis β€” returns a severity-rated root-cause card with metrics, evidence, and either one-click fix commands or manual steps. Destructive ops require a dry-run first, then a single-use confirmation token.
  10. Persistence β€” every message, tool call, and result saved to SQLite so you can pick up where you left off. Session ID lets teammates load the full investigation.
  11. Auto-capture + promotion β€” a strict-JSON classifier decides which answers qualify for the cached-runbook tier. Thumbs-up promotes; thumbs-down quarantines. Secrets are redacted before persist.
  12. Alertmanager fan-in (optional) β€” POST to /api/v1/alerts/webhook and every firing alert kicks off its own scoped investigation via steps 3–10 above, posting the finding back to Slack / PagerDuty.

Example Interactions

Quick listing β€” answered in one tool call, ~5 seconds:

You: what pods are in the jenkins namespace?
Astra: Here are the pods in the jenkins namespace.
       β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
       β”‚ Name                    β”‚ Status   β”‚ Ready β”‚ Restarts β”‚
       β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
       β”‚ jenkins-0               β”‚ Running  β”‚ 2/2   β”‚ 1        β”‚
       β”‚ avatar-agent-1s7k7      β”‚ Pending  β”‚ 0/0   β”‚ 0        β”‚
       β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Deep investigation β€” multi-step ReAct, ~15 seconds:

You: why is mongo arbiter pod in crashloop?

Investigation Trail: 3/3 tools
  βœ“ kubectl β†’ pod status retrieved
  βœ“ events  β†’ events scanned
  βœ“ ai      β†’ analysis complete

β”Œβ”€ CrashLoopBackOff ─────────────────────── CRITICAL ─┐
β”‚ mongodb-arbiter-0 Β· infrastructure                   β”‚
β”‚                                                      β”‚
β”‚ The MongoDB arbiter pod is failing to start because  β”‚
β”‚ the designated primary host (mongodb-0) is not       β”‚
β”‚ available. The arbiter's setup process times out.    β”‚
β”‚                                                      β”‚
β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”        β”‚
β”‚ β”‚   360    β”‚ β”‚ CrashLoopBackOff β”‚ β”‚ False β”‚        β”‚
β”‚ β”‚ RESTARTS β”‚ β”‚      STATUS      β”‚ β”‚ READY β”‚        β”‚
β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”˜        β”‚
β”‚                                                      β”‚
β”‚ [Review & Execute Fix]                               β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Fix command: kubectl delete pod mongodb-arbiter-0 -n infrastructure

Cached runbook hit β€” sub-second, near-zero cost:

You: how do I recover from OOMKilled after a memory limit bump?

KB Router: cached (similarity 0.94 β†’ runbook#412 "OOMKilled recovery")

Answer: Increase the memory limit on the Deployment and redeploy β€” the
        pod will be recreated with the new limit. If the workload is
        still OOMing after the bump, capture heap or profile before
        increasing further.

Citations: docs/runbooks/oom-recovery.md

Alert-driven auto-investigation β€” Alertmanager β†’ root cause in under a minute:

[Alertmanager]  KubePodCrashLooping  β†’  POST /api/v1/alerts/webhook
                    ↓
KubeAstra opens an investigation session for {namespace, workload}:
  βœ“ playbook-first hit: "CrashLoop w/ ImagePullBackOff" pattern matches
  βœ“ investigate_pod β†’ confirms ErrImagePull on payments-api:v42
  βœ“ search_deployment_repo β†’ v42 tag doesn't exist in registry
  βœ“ synthesis critic β†’ high-confidence finding

[Slack]  Root cause: image tag `v42` was never pushed. Rollback to
         `v41` or push the missing tag. (fix command staged for review)

Multi-step remediation plan β€” dry-run gate + per-step confirmation:

You: unstick the stuck Redis PVC blocking checkout-service

Plan proposed (3 steps):
  1. scale_deployment redis --replicas 0     [dry_run OK]
  2. delete_pvc redis-data --wait            [needs confirmation]
  3. scale_deployment redis --replicas 1     [chained to step 2]

You: confirm step 1
KubeAstra: βœ“ scaled redis to 0/0 (ready in 4s)

You: confirm step 2
KubeAstra: βœ“ deleted redis-data PVC (finalizer removed cleanly)

You: confirm step 3
KubeAstra: βœ“ redis 1/1 running, checkout-service reconnected

Memory reuse across turns β€” the assistant carries context forward:

You: what's broken in the payments namespace?
Astra: 3 pods CrashLoopBackOff in payments β€” checkout-api, ledger,
       webhook-worker. Root cause: shared Redis dependency Pending.

You: fix it                                    ← no need to re-specify
Astra: [memory recalled: payments ns Β· redis PVC issue]
       Proposing plan to unstick Redis PVC (see previous investigation).

Configuration

All settings are read from environment variables (or .env):

Variable Default Description
LLM_PROVIDER gemini gemini, anthropic, openai, or ollama
GEMINI_API_KEY β€” Required when LLM_PROVIDER=gemini. Get one free
GEMINI_MODEL gemini-2.5-flash Gemini model to use
ANTHROPIC_API_KEY β€” Required when LLM_PROVIDER=anthropic. Get one at console.anthropic.com
ANTHROPIC_MODEL claude-opus-4-8 Claude model to use (claude-opus-4-8, claude-sonnet-5, claude-haiku-4-5-20251001, ...)
OPENAI_API_KEY β€” Required when LLM_PROVIDER=openai
OPENAI_MODEL gpt-4o OpenAI model to use
OPENAI_BASE_URL https://api.openai.com/v1 Override for OpenAI-compatible endpoints (Azure OpenAI, vLLM, LiteLLM, ...)
OLLAMA_BASE_URL http://localhost:11434 Ollama server for chat
OLLAMA_MODEL llama3.1 Ollama model name (must be pulled first)
EMBEDDINGS_MODE local local (sentence-transformers, in-process), api (a provider's embeddings endpoint), or ollama
EMBEDDINGS_PROVIDER β€” voyage, openai, or gemini. Only read when EMBEDDINGS_MODE=api
EMBEDDINGS_API_KEY β€” Key for EMBEDDINGS_PROVIDER. Without one, memory degrades to keyword-only rather than failing
EMBEDDINGS_MODEL provider default Override the embedding model. nomic-embed-text for Ollama
EMBEDDINGS_TIMEOUT_SECONDS 30 Per-request timeout for embeddings
OLLAMA_URL http://localhost:11434 Ollama server for embeddings β€” a separate setting from OLLAMA_BASE_URL
ALLOWED_NAMESPACES * Comma-separated list, or * for all
KUBECTL_TIMEOUT_SECONDS 15 Per-command timeout
MAX_LOG_TAIL_LINES 200 Max log lines per request
ENABLE_RECOVERY_OPERATIONS false Enables delete_pod, rollout_restart, scale_deployment, apply_patch
REQUIRE_DESTRUCTIVE_CONFIRMATION true When enabled, destructive tools require dry_run first then a confirmation_token
QDRANT_URL http://localhost:6333 Vector DB for RAG (use :memory: for tests)
QDRANT_API_KEY β€” Optional β€” required when Qdrant runs with authentication enabled
RAG_ROUTER_ENABLED false Turns on the cached/grounded/cold retrieval router
RAG_ROUTER_CACHED_THRESHOLD 0.92 Similarity β‰₯ this on a verified runbook returns the cached answer
RAG_ROUTER_GROUNDED_THRESHOLD 0.70 Similarity β‰₯ this on any doc returns a grounded (RAG-fed) answer
ENABLE_LOG_SUMMARIZATION false Compress large kubectl logs / describe output via heuristic + LLM polish
SESSION_CAPTURE_ENABLED false Auto-classify answers for promotion to the runbook cache
ALERTMANAGER_WEBHOOK_ENABLED false Accept Alertmanager webhooks on /api/v1/alerts/webhook. The endpoint 404s until this is true
ALERT_WEBHOOK_TOKEN β€” Bearer token required on incoming Alertmanager webhook requests. Leave unset and the webhook is open to anything that can reach the backend
PROMETHEUS_URL β€” Prometheus endpoint for prom_query (leave unset to disable)

Alert ingestion is two gates, not one. /api/v1/alerts/webhook is exempt from interactive session auth, because Alertmanager has no user session. ALERTMANAGER_WEBHOOK_ENABLED controls whether the route exists at all; ALERT_WEBHOOK_TOKEN controls who may call it. Turning the first on without the second publishes a route that starts LLM-backed cluster investigations β€” the backend logs a warning on every unauthenticated call, but set the token.

Upgrading: before this flag was implemented the endpoint was always on, regardless of what the environment said. If alert-driven investigation is already part of your setup, set ALERTMANAGER_WEBHOOK_ENABLED=true when you upgrade or ingestion will stop.


Repository Layout

kubeastra/
β”œβ”€β”€ ui/
β”‚   β”œβ”€β”€ frontend/                # Next.js chat UI (Astra Intent Light theme)
β”‚   β”‚   β”œβ”€β”€ app/chat/            # Chat page + /chat/:sessionId share routes
β”‚   β”‚   └── components/          # ClusterConnect, ResourceGraph, InvestigationTrail,
β”‚   β”‚                            #   CostBreakdownOverlay, YamlProposer, ApprovalOverlay, etc.
β”‚   β”œβ”€β”€ backend/                 # FastAPI app + SQLite persistence
β”‚   β”‚   β”œβ”€β”€ routers/
β”‚   β”‚   β”‚   β”œβ”€β”€ chat.py          # Chat flow, tool dispatch, fix execution
β”‚   β”‚   β”‚   β”œβ”€β”€ cluster.py       # Cluster connection management (4 modes)
β”‚   β”‚   β”‚   β”œβ”€β”€ sessions.py      # History, SSH targets, post-mortem API
β”‚   β”‚   β”‚   β”œβ”€β”€ feedback.py      # Thumbs-up/down β†’ runbook promotion
β”‚   β”‚   β”‚   └── alerts_router.py # Alertmanager webhook + investigation fanout
β”‚   β”‚   β”œβ”€β”€ react.py             # ReAct loop orchestrator
β”‚   β”‚   β”œβ”€β”€ memory.py            # Per-user conversation memory
β”‚   β”‚   └── db.py                # SQLite with cluster_connections, user_memory tables
β”‚   └── docker-compose.yml
β”œβ”€β”€ mcp/
β”‚   β”œβ”€β”€ mcp_server/              # MCP server (stdio + HTTP transports)
β”‚   β”œβ”€β”€ http_mcp/                # HTTP MCP transport service
β”‚   β”œβ”€β”€ k8s/                     # kubectl wrappers, SSH runner, validators
β”‚   β”œβ”€β”€ ai_tools/                # Error analysis, fix playbooks, runbooks
β”‚   β”œβ”€β”€ alerts/                  # Alertmanager domain, orchestrator, notifications, playbooks
β”‚   β”œβ”€β”€ playbooks/               # Deterministic runbook engine (playbook-first routing)
β”‚   β”œβ”€β”€ services/
β”‚   β”‚   β”œβ”€β”€ llm/                 # Gemini + Ollama providers
β”‚   β”‚   β”œβ”€β”€ rag/                 # Chunking, ingestion, router, capture, promotion
β”‚   β”‚   β”œβ”€β”€ summarizer/          # Log / event / describe summarization
β”‚   β”‚   β”œβ”€β”€ plans.py             # Multi-step remediation plans
β”‚   β”‚   β”œβ”€β”€ confirmation.py      # Dry-run + single-use confirmation tokens
β”‚   β”‚   β”œβ”€β”€ synthesis_critic.py  # Hallucination auditor
β”‚   β”‚   β”œβ”€β”€ tool_envelope.py     # Structured tool-response format
β”‚   β”‚   β”œβ”€β”€ error_parser.py      # kubectl error classifier
β”‚   β”‚   β”œβ”€β”€ prometheus.py        # Prometheus query client
β”‚   β”‚   β”œβ”€β”€ embeddings.py        # sentence-transformers wrapper
β”‚   β”‚   └── vector_db.py         # Qdrant client
β”‚   β”œβ”€β”€ tool_registry.py         # Single source of truth: 52 tools
β”‚   └── config/settings.py
β”œβ”€β”€ cli/                         # `kubeastra` CLI (PyPI) β€” ask, investigate,
β”‚                                #   connect, doctor, config, and `open`
β”œβ”€β”€ desktop/                     # Local-app packaging (macOS, in progress)
β”œβ”€β”€ helm/kubeastra/              # Helm chart β€” backend, frontend, Qdrant StatefulSet,
β”‚                                #   RAG ingestion CronJob, NetworkPolicies
β”œβ”€β”€ evals/                       # DeepEval baselines + eval runner
β”œβ”€β”€ demo/                        # Kind + broken workloads for `make demo`
└── docs/                        # Public documentation

Roadmap

  • Gemini + Ollama (local) LLM support
  • Demo mode with kind cluster
  • Approval flow for write operations
  • Deployment-level investigation (investigate_workload)
  • Namespace-wide health analysis (analyze_namespace)
  • Agentic ReAct investigation loop (multi-step tool calling)
  • Shareable session URLs + investigation timeline
  • Auto-generated post-mortems from investigation sessions
  • Visual debugging canvas (interactive resource graph with health glow, click-to-inspect, tooltips, MiniMap)
  • Multi-modal cluster connection (auto-detect, kubeconfig upload, SSH, in-cluster)
  • One-click fix execution with safety guards and slide-to-confirm
  • Manual steps fallback when no automated fix is available
  • Large cluster support (text-format parsing for all-namespaces queries)
  • Session security hardening (path traversal prevention, cryptographic session IDs, command allowlists)
  • Team playbook engine β€” deterministic runbook execution ahead of the LLM
  • Alert-driven auto-investigation β€” Alertmanager webhook + concurrency-safe fan-out
  • Multi-step remediation plans β€” atomic steps with per-step confirmation tokens
  • RAG runbook cache β€” cached / grounded / cold retrieval router backed by Qdrant
  • Tool result summarization β€” heuristic + optional LLM polish for large logs / describes
  • Per-user conversation memory β€” 24h scoped context injected into the prompt preamble
  • Feedback β†’ runbook promotion β€” thumbs-up ships great answers into the cached tier
  • Prometheus integration β€” prom_query for range/instant queries during investigations
  • Remote diagnostics β€” SSH + Ansible playbook for air-gapped clusters
  • Eval harness β€” deterministic offline + nightly LIVE regression tests for LLM decisions
  • OpenAI + Anthropic Claude adapters (plus any OpenAI-compatible endpoint β€” Azure OpenAI, vLLM, LiteLLM)
  • Loki / Tempo observability integrations
  • "What changed?" view β€” recent deployments, ConfigMap/Secret mutations
  • Real-time collaborative sessions (WebSocket sync + presence indicators)
  • Slack bot integration (alert β†’ investigation β†’ findings in channel)
  • PagerDuty / OpsGenie native adapters (Alertmanager works today; these would skip the webhook hop)
  • CNCF Sandbox submission

Contributing

Contributions are welcome β€” especially the items at the top of the roadmap. See CONTRIBUTING.md for local setup, project layout, and how to add a new tool, and CODE_OF_CONDUCT.md for community guidelines.

Looking for a starter task? Check the good first issue label.


Security

Found a vulnerability? Please don't open a public issue. Email security@astraverse.dev or use private reporting. SECURITY.md covers what's in scope, what isn't, and how long we'll take to answer.


License

Apache 2.0 β€” see LICENSE for details.