AI agents use CLIs differently than people do. Learn how we built gcx, a CLI designed for AI agents to work with Grafana, and the engineering decisions that make agentic workflows more reliable and efficient.
Press enter or click to view image in full size
At an internal offsite, a room of engineers pointed their AI agents at our previous-generation Grafana CLI and let them work. We watched the agents retry commands that had already failed, confidently reach for flags that didn’t exist instead of reading the help, and flood their own context windows with output — burning tokens to badly rediscover things the tool could have simply told them.
Agents use a CLI differently than you or I do, but they’re now a core part of how users access and operate Grafana Cloud, and the tools they call must account for that. The result is gcx, the CLI designed for AI agents to access self-managed Grafana instances and Grafana OSS stacks as first-class users.
Each of the failure patterns mentioned above, including relentless retries and runaway costs, needed fixing. This post walks through the steps we took.
Helping an agent actually navigate Grafana Cloud
You and I might explore a CLI through trial and error. You type a command and read the help docs, digging deeper as needed. Agents can do that too, but every round trip burns time and tokens. When an agent can’t find what it needs quickly, it starts guessing.
So we designed gcx to describe its outputs compactly, in one shot. gcx help-tree emits the entire command hierarchy as a compressed tree that can be injected directly into an agent's context window. This output is abridged:
$ gcx help-tree --depth 1{"name":"gcx","short":"Control plane for Grafana Cloud operations","children":[
{"name":"slo","short":"Manage Grafana SLO definitions and reports",
"skill":"slo-manage,slo-investigate","availability":"grafana-cloud-only"},
{"name":"api","short":"Make direct HTTP requests to the Grafana API",
"hint":"Prefer gcx slo, gcx metrics query, gcx logs query. Reserve gcx api
for endpoints without a dedicated command."},
...
That tree contains metadata that is rough for a human to read, but great for an agent to act on.
The tree also steers. The raw gcx api escape hatch carries an llm_hint annotation that agents see in gcx help-tree output: “Prefer gcx slo, gcx metrics query, gcx logs query. Reserve gcx api for endpoints without a dedicated command.”
Before that hint existed, we repeatedly watched agents asked to find a dashboard respond like this:
The
gcxCLI doesn't appear to have a dashboard search subcommand, so I'll query the Grafana HTTP API directly:gcx api "/api/search?type=dash-db&query=checkout"
The command existed: gcx dashboards search wraps exactly that endpoint. The agent confidently declared it missing and reached for the raw API, which felt familiar from the Grafana HTTP API docs it was trained on. In doing so, it bypassed everything the CLI would have handled for it.
Availability annotations do similar work. An agent pointed at an OSS instance can see that a grafana-cloud-only command won't work, without wasting a failed attempt to find out.
Where help-tree gives the compressed hierarchy, gcx commands lists every command in a single machine-readable document. It includes a token-cost annotation (small/medium/large) that tells the agent which commands return bounded reads and which might flood its context before it runs them. The inventory also carries hints, required scopes and roles, and the resource types each command touches.
Because stale metadata would send agents down confident dead ends, gcx commands --validate checks that inventory against live Grafana API discovery.
Discovery also extends to payloads, since agents write manifests as well as read them. gcx resources schemas dashboards returns the JSON schema. gcx resources examples provides a valid starting manifest. gcx resources validate checks generated resources server-side before anything gets pushed. The agent never has to guess what a valid dashboard looks like; it can ask for the schema and verify what it builds.
All of this help is optimized for completeness per token rather than readability, and it carries the judgment calls a senior engineer would normally provide: what’s cheap, what’s expensive, and what to prefer.
Errors that double as recovery plans
When a human hits an error, they read it, sigh, and figure out what to do differently. For an agent, the error message is the recovery plan. A vague error makes it more likely that the agent will take a wrong turn.
In agent mode, every gcx failure emits a structured error object on stdout with concrete recovery suggestions.
{"error": {
"summary": "Authentication failed querying datasources",
"exitCode": 3,
"details": "request failed with status 401: Unauthorized",
"suggestions": [
"Review your Grafana credentials: gcx config view",
"Re-authenticate if needed: gcx login"
]
}}Several deliberate choices sit behind that sample.
Suggestions are runnable commands. A human-oriented tool might tell you to check your configuration; gcx says gcx config view and hands the agent a literal command to execute next. Agents act on imperatives.
Summaries come from a fixed vocabulary. It includes Resource not found and Authentication failed. Exit codes follow a stable taxonomy from 0 to 6: success, general, usage, auth, partial failure, cancelled, and version-incompatible. An agent can branch on the failure class without parsing English. Each code maps to an obvious remedy path, making it clear where the error lies. Adding more codes would dilute their meaning, so we keep the set small.
Partial failure is its own outcome. Batch operations return both the successful items and the error in one envelope, under exit code 4. If an agent pushes 40 dashboards and loses two, it can see exactly which two failed in a single parse instead of rerunning everything.
Documentation links come with instructions. When an error contains a documentation link, gcx adds an explicit suggestion to “fetch the documentation at [url].” We learned that agents act on suggestions but often ignore a bare docsLink field. Every URL points to the Markdown rendering of our docs, so a fetching agent receives clean text rather than a page full of HTML chrome.
We keep refining this loop based on real failures, and gcx helps us collect them. An opt-in failure log appends every failed agent-mode invocation as a redacted JSONL record. Flag values are stripped, while the error class and exit code remain. That gives us a corpus of real agent failures to mine for the next round of hints.
One recent example involved gcx's built-in --jq flag, which we'll cover below. Agents sometimes wrote expressions based on the wrong output shape, then flailed. They would rerun the command without --jq just to inspect the schema. Now the error gives them everything at once:
{"error": {
"summary": "Invalid command usage",
"exitCode": 2,
"details": "jq: expected an object but got: array\n\nThe command's output is
an array of 18 objects.\nElement fields: description, name",
"suggestions": [
"Iterate array elements with .[], e.g. --jq '.[].name'",
"Run the command with --json list to enumerate all available field paths"
]
}}The error identifies the shape, counts the elements, and lists the fields. It even hands over a corrected expression: the agent’s next attempt is --jq '.[].name', taken directly from the suggestion.
For humans, an error is a dead end that needs researching. For agents, it’s a fork in a program. You’re writing the branch.
Output that respects the context window
An agent’s context window is a scarce, shared resource. Your CLI output competes against the agent’s own reasoning, its prompts, other skills, and MCP tool definitions for space. Verbosity is a correctness problem as much as a style problem. Flood the context and the agent forgets why it ran the command.
It starts with a dedicated agent output codec. gcx detects when it's running inside an agent harness — it recognizes Claude Code, Cursor, GitHub Copilot, Amazon Q, and OpenCode through their environment variables, with no configuration required — and switches its default output to compact, non-escaped JSON with forced pipe semantics. No tables, colors, truncation, or box-drawing characters leak into a parser.
Detection is deterministic and can be overridden. GCX_AGENT_MODE=1/0 or an explicit --agent flag pins the behavior for CI and testing.
Large results spill to disk instead of filling the context. When a response exceeds about 100 KB, gcx writes the full payload to a temporary file and prints a small envelope instead. The envelope says where the file lives and how many items it contains, shows a three-item preview, and tells the agent how to read the rest.
A dashboard search with 400 results costs the agent a few hundred tokens rather than 100,000. The full data is still one read away when it’s actually needed. gcx agent prune cleans up old spill files.
Transformation happens in-process, too. Agents often need to reshape data by grouping, filtering, or counting. Without built-in support, they’ll pipe the output into an external jq process or, worse, generate a throwaway Python script. gcxembeds a jq engine, gojq, so nothing needs to shell out.
$ gcx slo definitions list --jq '[.[] | select(.objectives[0].value < 0.995)] | length'3
For the question, “How many SLOs have a target below 99.5%?”, the answer crosses the context window as a single token. The agent doesn’t need to hold a JSON document in its head while counting.
Before asking for fields, the agent can also ask which fields exist:
$ gcx datasources list --json ?name
type
uid
url
...
$ gcx datasources list --json name,type
--json ?, also available as --json list, inspects the field paths in a sample object. It requires no extra documentation lookup. The agent can discover the schema before selecting from it instead of hallucinating field names and learning through repeated failures.
gcx also nudges. When an agent requests raw JSON without using --jq or --json, a one-time hint on stderr points it toward the in-process options.
The stream discipline matters. Errors are results, so they remain in-band on stdout. Advisory diagnostics are structured as {"class":"hint",...} and stay on stderr, leaving the result stream clean.
Even that hint had to justify its cost. An earlier version printed it into piped output on every call, and our own investigation benchmark scores dipped until we traced the regression to that one stray line polluting every parsed result. The hint now appears once, stays structured, and remains outside the result stream. When your audience pays by the token, help text has to earn its place too.
Sometimes the best output is delegation. Cross-signal root-cause work can take a general-purpose agent dozens of metric, log, and trace queries. gcx assistant instead hands the entire investigation to Grafana Assistant, the domain expert inside the platform.
Assistant Investigations sends specialized sub-agents across metrics, logs, traces, and profiles in parallel, right next to the data. It returns findings, hypotheses, and remediation recommendations rather than raw query results. In one real incident, it reached the root cause in eight minutes, 20 minutes ahead of the on-call team.
When using Assistant Investigations, your agent runs one command and gets a conclusion. The dozens of intermediate results never leak into its context window. The generalist delegates to the specialist, and both are better for it.
Human CLI design asks, “Is this output readable?” Agent CLI design asks, “Is this output worth its tokens?” It then gives the caller ways to pay only for what it needs.
Guardrails for a caller with no fear
An agent can still confidently delete the wrong thing, so the safety lives in the credential: gcx logs in through browser OAuth, and during login a human grants the session read-only scopes that are enforced server-side. The agent can never widen them on its own. From there, let it roam: in a read-only session, the worst it can do is get a request denied.
Contexts that are allowed to write get two more layers of protection.
The first is a client-side fail-safe on --dry-run. A preview is only safe when it truly is a preview. A customer report taught us this lesson: a dry run had applied its edits. The backend API accepted the dry-run parameter, then ignored it anyway.
Rather than documenting the exception, gcx now keeps an allowlist of resources with verified server-side dry-run support. It refuses to send mutating dry-run requests for anything else and reports those operations as skipped. A guardrail that only mostly works is worse than none, because agents will rely on it heavily.
The second is idempotent writes. gcx resources push is create-or-update. When an agent inevitably retries a push it half-believes failed, the second attempt converges on the same state instead of returning an error or creating a duplicate.
An “are you sure?” prompt protects a human at the moment of the mistake, but an agent will satisfy any prompt you put in front of it. The protection has to live somewhere the agent can’t reach. Capabilities are granted by a human during login. Previews are guaranteed to remain previews, and writes converge instead of compounding.
Skills are a crutch, and we treat them like one
gcx ships with a bundle of portable agent skills: Markdown playbooks for multi-step workflows such as debugging with Grafana Cloud or managing SLOs. They're embedded in the binary, installable into any compatible harness with gcx agent skills install, and held to code standards — every gcx invocation in every skill is validated against the real command tree in CI, so a playbook can't silently drift away from the CLI it describes. In our benchmarks, a skill that triggers at the right moment improves task scores and reduces tokens per task, because the playbook replaces expensive flailing with useful guidance.
And yet every skill is a crutch. When an agent needs a playbook to operate your CLI correctly, the CLI has failed to explain something. Maybe a hint is missing, an error leads nowhere, or the help text confused the agent. Every skill, built internally or externally, points to a tool that can still be improved, or serves as a stopgap until frontier models get better.
Two moments made this clear.
The first was the offsite from the top of this post. Most of the engineers in that room didn’t have the skills installed, so their agents fell back on trial and error, burning tokens to relearn what the playbooks already knew. The immediate conclusion is that skills matter. The more useful conclusion is that the tool had failed to give agents the context they needed to use it effectively. Several affordances discussed earlier — the hints, the nudges, the shape-aware errors — came from watching that flailing and asking what the tool should have said.
The second moment showed us the alternative. One of our engineers onboarded to gcx the way a new customer would, by letting an agent drive. The agent hallucinated its way through login. The durable fix wasn't a skill. It was a few clearer lines in gcx login --help.
That moved the knowledge into the binary. It became versioned alongside the code and appeared at the exact moment it was needed. It worked in every harness and required nothing extra to install or trigger.
That is now our standing loop. Whenever an evaluation shows that a skill materially helps, we ask what the CLI failed to say. Then we say it in the hint, error, schema, or help text. The skill shrinks and the tool grows.
Knowledge for agents belongs in the tool itself, delivered at the moment it’s needed. Put it anywhere else and the agent falls back on what it remembers from training, which it may be hallucinating, or on looking it up online, which costs tokens and time.
One lesson, and a long bet
Every shift in this post comes down to the same thing: stop assuming the caller shares your context. A human shows up with patience and a healthy fear of breaking things. They also bring a mental model from years of operations work. An agent has none of that, so key information has to live in the help and the clarity in the output.
gcx works as well for humans as any traditional CLI, because agent mode only changes the defaults. Designing for agents first showed us how much we'd been asking human operators to simply know.
There’s also a long bet here. Agents hallucinate gcx flags because kubectl is baked into their weights and gcx isn't yet. But LLMs train on public docs, including posts like this one. As gcx docs and examples build up in the training data, agents will show up already knowing the tool. The knowledge moves from playbook, to binary, to weights.
Weights go stale too, so the hints and errors stay on to fix whatever the models get wrong. Feeding the harness and feeding the training data are the same job.
gcx is the Grafana Cloud CLI. Install it, point it at your stack with gcx login, and try it. And if you're driving it from Claude Code, Cursor, or another agent harness, it already knows.