I’ve been building an agent to maintain my apps. It’s not as easy as it looks.

Press enter or click to view image in full size

I’ve been building this agent that watches my apps and keeps them working. It reviews changes before they ship, keeps an eye on production, and does the boring maintenance that never gets done (like checking mobile layouts). It currently runs against 3 of my projects, including one that’s public.

Let me be up front: my apps are pretty much AI-written. Claude does the coding and I tell it what to build and check the results (ideally without looking at the code). I get Claude to TDD and CI/CD to keep things on track.

Having an agent whose whole job is to be paranoid about the running app is key. Means I can do other stuff.

Still early and rough, but I thought I’d share my experience.

1. It catches hidden broken stuff.

Here’s a few examples of what the agent has caught and done:

A malformed URL on a public route threw an uncaught exception and returned a bare 500. Worse, the error was thrown before the tracking code ran, so it was invisible to the error-history. (#98 / #99)
On Cloudflare, a config variable can silently overwrite a secret of the same name on the next deploy. No error while the app kept serving 200s. That mistake took my https://aixi.football app down for ~11 hours. The agent figured it out and added a gate that fails the build so it won’t happen again. (#103)
CI passed every check, but nothing validated that the app would deploy. (#102)

Nothing dramatic here. Well, an 11 hour outage would be pretty dramatic if I had loads of users. But these fixes are the boring, invisible stuff a feature-focused dev (me) can miss.

2. The cost.

I’m not paying per token. I’m on flat subscriptions: a $200 Claude Max plan and a $144 GLM one. But the agents eat fast: left unchecked, they burned about a third of a weekly plan in an afternoon across the three apps. Part of that was GLM was making huge numbers of errors when calling gh. But the fact I needed a second Max plan just for the agents tells you everything you need to know about their hunger for tokens. They are voracious.

Since then, I’ve had to do quite a bit of engineering to keep usage in line. But I still have to keep the agent on a leash (see below).

A few things I’ve done:

Created scripts that called gh with the correct arguments, so I didn't waste so many tokens on failed calls.
Stop calling GLM 5.2 during peak hours (unless it’s a production error). z.ai charges 2x during peak hours and errors constantly (~65%). And with my big context, it was burning a lot of tokens whether or not I got a result. (And yes, that doesn’t seem right to me either. I’m looking at you, z.ai.)
Make wakeups deterministic (no LLM) and only spin up the full context if there’s something to do.
On top of pausing during peak hours, I created a governor that throttles the amount of subscription that can be used in a given timeframe. When I set ratio=0.5 it means that less than 50% of the plan can be used up or the agent doesn't run.

3. It makes mistakes.

The worst mistake was upgrading Astro from version 5 to 6. This was the root cause of the 11 hour outage. Well, me trying to make the upgrade work was the root cause.

And to be clear, I don’t let it merge PRs. I review and merge by hand. But still, I miss stuff.

You can watch the agent do it’s thing.

Want to see the agent in action? It maintains https://released.blabberate.com, one of my public repos, so you can just go and read what it’s actually doing: PRs it opens, what it fixes, what it leaves alone. See github.com/lukaso/released/pulls.

Is the agent worth it?

It’s a big question. The agent is doing a lot, and I’d say it’s all valuable. But then again, some of it I just wouldn’t have bothered doing in the first place if I had to do it by hand. But what a feeling it was when the agent detected and then fixed a production error. I felt giddy.

On balance, I think it is, even while I’m still heavily optimizing.

So it’s early days.

I’m mostly posting because I’m curious who else is running agents against their own live stuff.

And no, I’m not ready to open source it yet. A bunch more tweaking to do before it feels right to foist on the world.