How to Build an AI Software Factory: Agents That Open, Review, and Merge PRs
firecrawl.devI'm using fable and opus about 25%75/% respectively. I have a 3 step process - discuss a topic and create a review, convert the review to a task file and update the design docs, then finally implement the task file in code. Normally around 300-1000 loc implemented for a change, more for a new UI page or whatever.
And honestly it's incredibly good. - 50% of the time it'll generate code that I accept wholesale.
- 25%: it maybe makes a choice I don't like, or the implementation isn't the approach I liked, so I tweak it.
-10%: it needlessly creates semi-duplicate function logic, so it's a lot more code and execution paths become hard to conceptualise
-10%: it mirrors business logic it shouldn't, and subtly changes it. Which needs a huge amount of intervention to fix.
- 5% it goes completely off the rails. Eg decided that business/application logic should be in the database.
Prompt and check this honestly works very well (still waiting on tests with more than a handful of paying customers however!) but a FULLY automated agent pipeline? I'm not convinced. That last 15% of cases will make a degenerate codebase real fast.
Also without a CC subscription, paying per token, it would be horrifically expensive for my approach.
I am using a very similar approach and seeing about the same. It works well and fast with some attention to details.
I also do not understand the obsession with “software factories”. Like “JIRA to product” nonsense. Is like we can hand wave the details that matter and somehow the end product would still be what we want.
Not even talking about the degenerative codebase you mention, which is also an issue I am seeing after some time in any project.
Can I ask what you're building? Genuine question, you gave a detailed setup of your agents and I wonder about the context
I'm also curious where the discuss part comes from, because I guess you don't just let them start on a basic prompt without any context
> agents work in public channels, never DMs.
Recently I’ve been looking into why LLMs tend to write in some off putting ways. This one apparently is for LLMs to get better results.
It puts the cases to avoid closer to the main requirement so is easier for a LLM to follow without losing track with more ambiguous language.
Is very interesting because is harder to read by a human but it gets better results for an LLM.
This seems like a useful write up of the concepts involved here, but I just hate reading Claude’s writing style so much.
This is content marketing, at least give it a review pass before you hit publish… if your marketing is unpolished AI slop, I have to assume your product is too.
> every company that made this work built the gates before the fleet
Honestly I’m just glad they built the gates before the fleet! That’s not a step change, it’s a paradigm shift!
Yeah, it's bad. But at least whoever is behind it found some interesting references
We've had software factories for decades. They're usually called compilers, linkers, toolchains, etc.
Who tf or what tf wrote this. It’s like I’m talking to opus 5 again, terrible writing. I hate the product already
> Generation scales with spend, review does not, and that asymmetry is the whole design problem.
I see you guys have a blog post factory as well.
People are slowly loosing the brain muscles for writing (also thinking hard). Just like someone who is in shape and lets themselves go, it gets harder and hard to wake up and go for the run the longer you wait.
At this point people genuinely do not have the willpower to write, and instantly give into the lazy shortcuts llms provide. Just like how GPS apps degraded smart people's ability to navigate around their own towns (I've seen this happen to many over the years), LLMs will totally degrade people's ability to write (and think for themselves).
We're slowly auctioning off our sovereignty in exchange for quick shortcuts to every tech company. Not good.
Just wait, soon you will hear rumors of how tech CEOs don't let their children use AI in school (while they demand everyone else's children do), similar to how Zuck et al didn't let his kids use social media.
"We're slowly auctioning off our sovereignty in exchange for quick shortcuts to every tech company."
Slowly?
god i hate this style of sentence. claude uses it far too much. my teammates have been writing PRDs that are filled to the brim with this sentence structure. absolutely maddening.