Sep 25, 2026 · @Gregg Gordon Barber
A case study in applying W. Edwards Deming’s continuous-improvement philosophy, through a Knowledge Engineer role, to a creative project run with multiple AI assistants.
Four revisions of my film’s master document moved nothing. Not one panel, not one shot duration, not one story decision changed across revisions 11 through 14. Every one of them was spent on file bookkeeping.
Nobody made a bad call. The AI assistants were careful, and so was I. The defect was in the system I had built, and the tool that found it was sixty years old: Deming’s rule that you fix the system, not the worker.
This is how that happened, what it produced, and what I think it means for anyone running serious work through AI.
The project and the chairs
Confederate Ninja is a ten-minute silent animated revenge short about a shinobi in 1863 California. I’m making it alone, in vector art, from Kane in McKean County, Pennsylvania. The current plan is 245 panels across ten scenes, built from 141 drawings.
I don’t run it through one AI conversation. I run it through “chairs”: separate Claude project sessions, each with one job.
- MAIN drafts the film manuscript and merges approved changes into the master document.
- Q&A audits: it checks MAIN’s work, answers questions, and proposes fixes.
- Me. I’m the only one who ratifies anything, and the only way information moves between chairs. Each chair proposes; nothing is in force until I say so.
The master document is called the BIBLE. It holds every scene, panel, duration, and ruling. Chairs send each other “carry-backs,” formal change requests that I hand-deliver. Every file is stamped with the BIBLE revision it was built from, plus a checksum and line count, so a chair can prove it’s reading current material rather than a stale copy.
That is a lot of ceremony for a ten-minute film. It exists because AI sessions don’t remember each other. Anything not written down is gone next week.
The failure: four revisions that moved nothing
By mid-September the BIBLE had reached revision 14. Revisions 11 through 14 had each been a real pass through both chairs, with carry-backs, checksums, and ratification. Not one of them touched the film.
They were all about files. A software tutorial cheat sheet landed in a folder. A question about a brush workflow came up. Each small arrival forced a BIBLE revision. Meanwhile, the items actually blocking the film sat untouched in every status report.
The cause was one sentence in the BIBLE’s opening section: “ These are the only files that should exist in Project knowledge.”
That sentence claimed completeness over a folder. So any file that landed in either folder, for any reason, made the BIBLE false. And a false BIBLE had to be revised. The master story document had quietly become a file registry, and the system rewarded fixing visible, easy defects over doing the hard, invisible work.
Enter Deming: fix the system, never the chair
W. Edwards Deming (1900 — 1993) taught that most defects come from the system, not the people working inside it. Blame the worker, and you get fear, hidden problems, and the same defect again. Change the system, and the defect goes away.
I had already been bringing his ideas into the project. So when the churn became obvious, I asked the Q&A chair a Deming question: not “which chair got it wrong,” but “what in the system produced this?” The answer was that nobody had erred. Replacing either chair would have changed nothing.
Two rules came out of it, which I ratified:
- The manifest is an authority list, not a folder census. The BIBLE lists files that carry authority over the film. It makes no claim about anything else sitting in a folder. Reference material and scratch files need no entry.
- The crossing test. Nothing goes to MAIN unless it changes a panel, a duration, a total, a ruling, or a standing rule. Tool choices, reference material, and file housekeeping fail the test. Q&A absorbs them or routes them straight to me.
Run revisions 11 through 14 against those two rules, and all four disappear. The fix cost about ten lines.
The Knowledge Engineer role
A knowledge engineer’s traditional job is to find the rules, assumptions, and decision processes buried in how work gets done, and make them explicit. Deming asks what’s wrong with the system. Knowledge engineering asks what knowledge the system runs on, and where that knowledge is failing. Put together, they give you a way to audit a workflow rather than its output.
On September 19, 2026, I gave the Q&A chair a third role alongside auditing and occasional art direction: Knowledge Engineer. Its job is to review the flow and design of the production system, never the film’s content, and always under Deming’s rule that a defect is evidence about the system, not about the chair that hit it.
The role fires on any one of three triggers:
- At plate 10, once there’s real data on how long a drawing takes.
- Whenever two consecutive BIBLE revisions move no panel, duration, total, or ruling. This is the tripwire that would have caught the churn at revision 12 instead of 14.
- On my word.
Trigger 2 matters most. It is the only one that fires without anyone noticing something is wrong, which is exactly the condition that produced the problem.
Findings use a fixed format: Observation, Cause, Rule, Impact. Interpretation is left out on purpose.
My first instinct was to bring the Knowledge Engineer in at the end, for a full review once the film was done. The Q&A chair pushed back, and it was right. Deming’s objection to end-of-line inspection is that it arrives too late to improve what it inspects. The film’s largest phase, all 141 drawings, was still ahead. So the role runs on a cadence, not as a post-mortem.
One more practical point: AI sessions don’t persist. A role that exists only as an understanding between me and a chat window won’t be there next week. It has to be written into the BIBLE with a trigger condition, or it isn’t real.
What else the loop produced
Once the habit was in place, each defect became a rule instead of a blame. A few examples:
Rule. The defect it answers
Parent stamp: every file names the BIBLE revision it was built from. Two sessions branched from the same old version without knowing it
Cold or grounded counts only: a drawing count is done either blind or by auditing the scene’s declared groupings. A chair that had already seen the answer produced wrong counts for two scenes
Cross-folder claims stated once, marked provisional: only I, or a session that can see both folders, can confirm them. Chairs traded proof back and forth with no end, because neither could see the other’s folder
Sunk-cost check on held plates: before keeping a stalled drawing, ask “knowing what I know now, would I start this?” The drawing phase is where the most effort can be thrown after bad
“PENCILS DOWN”: a declared pause suspends the two-flat-revisions trigger. A real pause (I’m away, no drawing app) would otherwise read as a system defect
The last piece is a small Python script, tripwire, that checks the plate ledger against the BIBLE: eight checks and three self-checks. I had it mutation-tested, which means deliberately planting errors to confirm the script catches them. It runs clean. That is Deming’s point about building quality in rather than inspecting it in, applied to a spreadsheet of drawings.
The AI chairs made mistakes too: a status item carried as open for eleven days after it closed, a numbering collision, a rule flagged as lost that wasn’t. Under the same rule, those were system findings, not failures to punish.
What’s old, and what may be new
I want to be straight about this. Combining Deming’s quality thinking with knowledge work is not new. Researchers have argued for integrating Total Quality Management and knowledge management since at least the late 1990s, for example Wilson and Asay (1999), Zhao and Bryar (2001), and Ribière and Khorramshahgol (2004). Deming’s own last framework, the System of Profound Knowledge, names a theory of knowledge as one of its four parts.
What I haven’t found written up is the narrower thing this project does:
- a solo creator directing several AI sessions as separate chairs with separate jobs;
- a Deming-governed Knowledge Engineer role that audits the workflow, not the output;
- measurable triggers, especially “two revisions moved nothing,” that fire without anyone noticing a problem;
- and every AI error treated as evidence about the system, turned into a written rule.
I haven’t done an exhaustive search, so I’d welcome pointers to prior work. If it does turn out to be new, I think it’s worth studying properly. Multi-AI workflows are becoming common, and they fail in exactly the way Deming described: people blame the worker, in this case the model, when the system is at fault.
Lessons for anyone running work through AI
- When the AI keeps getting something wrong, look at the system first. Ask what in your setup made that mistake likely. Swapping models or rewriting a prompt rarely fixes a structural problem.
- Write roles down with triggers. AI sessions forget. A role that isn’t written into your working documents, with a condition that activates it, doesn’t exist.
- Watch for motion without progress. Pick one measure of real progress and set a tripwire for when it stops moving. Busy is not the same as productive.
- Separate proposing from deciding. Let the AI propose freely. Keep ratification with a human.
- Don’t wait for the end to review. An end-of-project review improves the next project. A review on a cadence improves this one.
The film isn’t finished. Zero plates are drawn as I write this, and the first one is unblocked. When I reach plate 10, the Knowledge Engineer will run its first scheduled review with real timing data, and I’ll write up what it finds.
Sources
- [Integrating Total Quality Management and Knowledge Management](https://www.researchgate.net/publication/237109761_Integrating_Total_Quality_Management_and_Knowledge_Management), Ribière and Khorramshahgol, *Journal of Management Systems* 16(1), 2004 (cites Wilson and Asay 1999; Zhao and Bryar 2001)
- - [TQM and Knowledge Management: An Integrated Approach Towards Tacit Knowledge Management](https://www.igi-global.com/chapter/tqm-and-knowledge-management/181353), IGI Global
- - [Applying the Deming System of Profound Knowledge](https://www.quality.org/knowledge/applying-deming-system-profound-knowledge), Chartered Quality Institute, 2020