Constraints Breed Discipline - Blain Smith

21 min read Original article ↗

[Main] [Essays] [Articles] [Projects] [Now] [Resume] [Talks]

Walk into any old stone church and look up. The arches, buttresses, and ribbed vaults that hold the roof against gravity for a thousand years were not chosen for their beauty. They were chosen because stone is heavy and weak in tension, and the people who built those walls had to obey what stone could do. The beauty came later, almost as a side effect. The masons did not have the option to ignore the material. They could not summon a new substance into existence by writing a manifesto about it. The constraint of the stone forced them to think harder, and the thinking is what produced cathedrals.

Software has very few constraints like that, and the ones it does have can be dissolved by writing more software. A bridge engineer cannot decide that load-bearing walls are inconvenient and abolish them by Friday. A pharmacist cannot ship a new compound on Tuesday because the deadline is Wednesday. We can. Every boundary we encounter is implemented in code, and code can be rewritten by whoever has commit access. That power is what makes software so productive, and it is also what makes it so easy to ruin.

The most innovative industries in human history have done their innovation inside constraints, not by escaping from them. Software has spent four decades doing the opposite, and the costs of that choice are now legible in the daily experience of using the systems we have built. The argument here is not against innovation. It is for the discipline that makes innovation possible in the first place.

+++

Christopher Alexander spent his career studying how good buildings get built, and his answer was that the building has to fit its context. He called the process of design the activity of inventing physical things that respond to function. The 1964 book where he laid this out, Notes on the Synthesis of Form, defines design as the work of finding fit between a form and the demands placed on it. An adaptive process will be successful only if it proceeds piecemeal instead of all at once, he wrote, and he pointed to the traditional builders of unselfconscious cultures as the people who got this right. They could not afford to redesign the whole village every generation. They made one change at a time, kept what worked, and discarded what did not. The constraint of slow change is what produced coherent forms.

Alexander is describing a feedback loop. Real constraints force the designer to confront misfits, places where the form fails to meet its context, and to address those misfits one at a time. The designer cannot pretend the misfit does not exist; they have to either change the form or change their understanding of the context. Either way, the work gets sharper.

When the designer has the power to remove the constraint instead of addressing the misfit, the misfit disappears not because the form got better but because the demand was rewritten. The feedback loop that produced cathedrals stops working. This is what software does to itself, and what most modern programming culture treats as a feature rather than a problem.

+++

Henry Petroski spent his career writing about engineering failure, and his central claim is that engineering is a discipline organized around the anticipation of disaster. "Engineering design has as its first and foremost objective the obviation of failure." Bridges, dams, airframes, pressure vessels, the whole catalog of civil engineering exists because someone, once, calculated what would happen if the structure carried more load than it was built for, and then designed it not to. The factor of safety is intended to allow for the bridge built of the weakest imaginable batch of steel to stand up under the heaviest imaginable truck going over the largest imaginable pothole.

Petroski's larger point is that the discipline of civil engineering has been built by studying the failures, not the successes. The colossal disasters that do occur are ultimately failures of design, but the lessons learned from those disasters can do more to advance engineering knowledge than all the successful machines and structures in the world. The Tacoma Narrows bridge twisted itself apart in 1940 because the designers had pushed the slenderness ratio beyond what the existing aerodynamic knowledge could account for. The Kansas City Hyatt Regency walkway collapsed in 1981 because a shop drawing change doubled the load on a single connection. Each disaster left a written record, a code change, a new constraint in the way bridges and buildings get built from then on. The constraint is the lesson.

Civil engineers do not operate under arbitrary constraints. They operate under the accumulated memory of every collapse and fatigue crack the field has recorded. The building code is a graveyard's worth of failure compressed into rules, and anyone who wants to skip the rules is asking to relearn the lessons that killed people. Petroski is careful to say that engineering is a human endeavor, that the goal is not to give engineers license to experiment with abandon but to recognize that human nature wants to go beyond the past and that this impulse has to be checked by the discipline of what has already failed.

None of this stopped innovation. Suspension bridges got longer and concrete got stronger inside the same tradition of failure analysis that produces building codes. The Crystal Palace, the Brooklyn Bridge, the Burj Khalifa were built by people who treated the codes as the ground under their feet, not as an obstacle. The codes told them where the floor was so they could push the ceiling without falling through.

+++

The same pattern holds in automotive manufacturing, which spent the second half of the twentieth century learning a discipline that software has largely ignored. W. Edwards Deming spent decades teaching American manufacturers that quality was not something you inspected for at the end of the line. Quality was a property of the system that produced the work, and the way you achieved it was by reducing variation, surfacing defects early, and treating every failure as information about the process rather than as a problem to be patched over.

The American auto industry initially rejected this. Deming went to Japan in the 1950s, where Toyota was rebuilding itself after the war, and the ideas took root. Taiichi Ohno turned those principles into what became the Toyota Production System, with its famous andon cord that any worker on the line could pull to halt production when they saw a defect. The cord imposes a constraint that nothing in physics requires: production may not continue while a known defect moves down the line. Stopping the line is expensive in the moment. Over the lifetime of the car it is cheap, because the defect never reaches the customer and the cause is identified while the evidence is still fresh.

Toyota chose to install that constraint because they had figured out that without it, defects compound. Lean manufacturing is not a method that exists independently of the cord and the dozens of other constraints like it. The constraints are what the method is. Remove them and you have a faster way to build broken cars.

Automotive engineering is also bounded by regulation that the rest of the world recognizes as legitimate. Crash standards, emissions limits, fuel economy requirements, seat belt mandates, airbag deployment specifications: none of these were chosen by the manufacturers, none of them can be removed by the manufacturers, and all of them cost money to comply with. They have also driven the innovation. Catalytic converters, fuel injection, crumple zones, anti-lock brakes, traction control, stability control, and modern restraint systems did not exist before the regulations that demanded them. The constraints pulled the field forward rather than freezing it in 1975.

The lesson for software is that the regulated industries are not slower because of their constraints. They are more disciplined because of them, and the discipline is what makes the innovation stick.

+++

Software started its life with hard constraints. Memory was measured in kilobytes, CPU cycles were expensive, and network bandwidth was a fraction of what a single image weighs today. The early programmers worked the way the medieval masons worked, inside the limits of what the material would allow. Dijkstra's algorithms, Hoare's logic, Ritchie's C, all of it bears the marks of an environment where you could not waste anything because there was not anything to waste.

Then the constraints started to dissolve, and not by accident. Moore's law gave us cheap transistors, and bandwidth, storage, and RAM followed. Each generation of hardware removed a constraint that the previous generation had taken for granted. The same process happened at the software layer. Garbage-collected languages removed the burden of manual memory management. Higher-level frameworks meant you no longer had to write your own data structures. Cloud computing meant you no longer had to provision a server before you could use one.

None of this was bad on its own. The removed constraints made it possible to build kinds of software that were unthinkable in 1985. The problem was that the discipline that had grown up around those constraints did not get carried forward. The mental habits that came from working inside hard limits, the care about waste, the suspicion of unnecessary complexity, the willingness to think before reaching for another abstraction, those habits did not survive the transition. The constraints went away, and most of the discipline went with them.

The shift from bare metal to virtual machines was the first big inflection point. A server you had to order, rack, and cable was a server you thought hard about before adding another one. A server that came up with a button click was a server nobody had to think about at all. The constraint that forced architects to consolidate workloads, plan capacity, and design for density went away, and what replaced it was sprawl. Most cloud environments today are full of resources that nobody can fully account for, running code that nobody fully owns, costing money that nobody can quite explain. The sprawl is not a side effect of virtualization itself but of the lost constraint that had been making people careful in the first place.

+++

Infrastructure as Code arrived as the proposed solution. The argument was that clicking buttons in a web console was error-prone and unreviewable, and that infrastructure should be defined in version-controlled text files that could be reasoned about, audited, and replayed. This was correct. It was, in fact, an attempt to put a constraint back into a process that had lost one. If you cannot change production without a pull request, then the discipline of the pull request gets applied to production. That is a real improvement.

The tools that implemented this idea, Terraform and CloudFormation and the rest, used configuration formats that looked like markup but behaved like programs. YAML, in particular, became the lingua franca. YAML is whitespace-sensitive, weakly typed, easy to write, and almost impossible to validate without running it. The format itself imposes almost no constraint on what you can express. Every team built its own conventions, every project grew its own custom modules, and the same configuration would behave differently depending on which version of which provider was installed at the time it ran.

The discipline that the pull request was supposed to enforce got diluted by the lack of constraint in the tooling. Reviewers could not tell, looking at a hundred-line YAML diff, what it would actually do when applied. The abstractions multiplied. Each new tool added a layer that promised to make the layer below it easier to manage, and each layer added new failure modes that nobody had time to learn. Anyone with an editor and credentials could make a change that took down production, and the audit trail would show only that a YAML file got updated.

IaC itself was not the mistake. The mistake was removing a constraint, the friction of clicking buttons, without installing a replacement constraint that was equally strong. The pull request was meant to be that replacement, but the pull request only works when the reviewer can actually understand what they are approving. When the configuration language is permissive enough that a one-character indentation error can change the meaning of a deployment, the reviewer becomes a rubber stamp. The constraint exists in name but not in force.

In 2017, an Amazon engineer running a maintenance command in the wrong environment took down a large portion of the S3 service in us-east-1 for several hours. The command itself was valid and the infrastructure code was valid. Nothing prevented a single keystroke from destroying the storage backend of a substantial chunk of the internet, because the tooling treated all environments as the same kind of thing. No andon cord. No building code clause forbidding a single point of failure in a system of that scale.

This is not a story about Amazon being careless. The same pattern has played out at every major cloud provider and at most of their customers. The underlying problem is that the IaC ecosystem inherited the speed advantages of the cloud without inheriting any of the discipline that the regulated industries developed over a century of failure.

+++

The next iteration of the same pattern is now arriving in the form of language models that can write code, write infrastructure, write documentation, and submit pull requests on behalf of humans who may or may not understand what is being submitted. The pitch is identical to every prior wave: the constraint that was slowing you down was the constraint of having to write the thing yourself, and now you do not have to. An hour of work becomes a minute. A team of five ships with two.

It is too early to know what the long-term effects will be. The empirical evidence about whether code generated by language models is more or less reliable than code written by humans is genuinely thin, and the studies that do exist disagree with each other depending on what they measure and how. Anyone making confident claims in either direction right now is reading from priors, not from data. What can be said, without going beyond what is observable, is that the pattern matches the IaC pattern, and the IaC pattern matches the cloud pattern before it.

A constraint is being removed. The constraint, in this case, is the requirement that someone who understands the code be the one who writes the code. The tooling makes it possible for a person who does not fully understand a system to produce changes to that system that look plausible enough to pass review. The reviewer, in turn, may also be using the same tooling, and may also not fully understand the change. The pull request becomes a transaction between two parties neither of whom has the full context, mediated by a model that has no context at all, only patterns.

The discipline that should compensate for this has not been installed. Nothing in the tooling functions like an andon cord, the moment when somebody on the line says stop, this is wrong, we need to figure out why. Nothing accumulates the way a building code does, the long record of what has failed before and what must not be done again. The lessons exist in scattered post-mortems on company blogs but in no form the code-generation tools can be made to respect. The requirement that you understand your own work is, for the moment, optional.

This is the slippery slope, not because the tools themselves are bad, but because the discipline has not caught up to them, and because every prior wave of this pattern has produced a stretch of years where the costs were absorbed by users rather than by the engineers shipping the work.

+++

None of this is an argument against innovation. The argument cuts the other way. The industries that innovate most reliably, civil engineering, automotive, aerospace, pharmaceuticals, do so because of their constraints and not despite them. Suspension bridge design accelerated after Tacoma Narrows, because the new constraints forced engineers to understand the aerodynamics they had previously been guessing at. Automotive safety bloomed once the federal crash standards were imposed, because the manufacturers had to find ways to hit performance targets that previously did not exist.

The same is true in pharmaceuticals. The FDA approval process is slow, expensive, and full of constraints that no drug company would choose to impose on itself. It is also the reason that the drugs that reach the market in regulated countries are, on average, safer than the drugs that reach the market where the process is weaker. The constraint of having to prove that your compound does not harm people is what produces the discipline of doing the experiments that show it does not. Without the constraint, the experiments do not happen. With the constraint, the experiments not only happen but become the basis for an entire scientific tradition.

The pattern in all of these fields is the same. A real constraint, honestly respected, forces the people working inside it to think more carefully. The thinking produces both better work and a deeper understanding of why the work is the way it is. Over time, that understanding accumulates into the craft of the discipline. Remove the constraint without replacing it with something equally honest, and the craft erodes, because the daily practice that used to demand it no longer does.

Software has been removing constraints for forty years. Each removal has been locally justified. The cumulative effect has been a field that ships faster than it understands what it is shipping, and that periodically rediscovers through painful outages the lessons that the older engineering disciplines wrote down a century ago.

+++

What would it look like to install constraints back into software, the way that the older disciplines did?

It would not look like a return to writing assembly by hand. The hard constraints that have genuinely been lifted, around memory, compute, storage, are gone for good reasons and there is no value in pretending otherwise. The constraints that need to come back are the ones that were never about hardware in the first place. They are about the relationship between the engineer and the work.

A rule that says you may not merge a change you do not understand is a rule about discipline, not about technology. A practice that requires a review to actually read the diff before approving production changes is a practice of discipline. The requirement that a service ships with a runbook, a rollback plan, and a documented owner is the same kind of requirement. None of these demand a new tool. All of them demand the willingness to say no, to slow down, to refuse to ship something that has not been thought through.

This is harder than it sounds because the economic incentives in most software companies push the other way. The metrics that get tracked are velocity, throughput, lines of code, features shipped, tickets closed. The metrics that do not get tracked are the ones that the older disciplines learn to track: the number of unowned services, the number of unreviewed configuration changes, the number of post-mortems whose action items never got implemented, the number of dependencies whose maintainers no one in the company has ever spoken to. The unmeasured part of the work is where the discipline lives, and software has built a culture that systematically refuses to measure it.

Alexander would call this a problem of fit. The form of modern software development has been adapted to a context that rewards speed over correctness, and the misfits, the outages, the breaches, the half-broken systems that nobody fully owns, are addressed not by changing the form but by adding more layers of tooling that promise to manage the consequences. Each new layer reduces the immediate pain of the misfit while preserving the underlying mismatch between what the work is and what it should be. The piecemeal adaptation that Alexander said was the only honest way to design gets replaced by a series of patches that protect the developer from the consequences of their own work.

For Petroski, the diagnosis would be the absence of any culture of failure analysis. Post-mortems get written, but they rarely produce code changes, and the lessons learned in one company rarely make it to another. Software has no equivalent of the National Transportation Safety Board, no authoritative body that investigates failures, publishes findings, and issues recommendations the rest of the industry is expected to respect. The failures are private, the lessons are private, and the next team makes the same mistakes because they have no way to learn from the last one.

Deming's diagnosis would land on the system itself. The defects are not the fault of individual workers. They are the fault of a system that does not give those workers the time, the tools, or the authority to stop the line when they see something wrong. The andon cord is missing not because nobody has thought of it but because pulling it would slow down the metrics that the system rewards.

+++

The case for constraints is not nostalgic. It is not an argument that software was better in 1985. It was not. The case is that the disciplines that have produced durable, trustworthy work, civil engineering, automotive manufacturing, pharmaceuticals, aviation, all share a structural property that software has been steadily abandoning. They treat constraints as the ground their work stands on, and innovation as the work of finding what is possible inside that ground rather than escaping from it.

The building code is not in the way of the bridge designer. It is the floor that lets the designer push the ceiling. A federal crash standard is what makes a crumple zone a meaningful innovation rather than a marketing claim, and an FDA protocol is what makes a drug trial's result trustworthy. The constraints are not the obstacle. They are the precondition for the work counting.

Software has spent four decades building tools that promise to remove the floor so that the engineer can run faster. The cloud removed the floor of physical infrastructure. IaC removed the floor of manual operations. Language models are now in the process of removing the floor of having to understand what you ship. Each removal is sold as progress, and each one ships before the discipline that should accompany it has been worked out.

The work of putting the floors back is not glamorous and does not get written about in product launches. It looks like writing runbooks that nobody reads until the night they save the company, refusing to merge a pull request until the author can explain it in their own words, spending an afternoon on a post-mortem instead of on the next ticket, and building tools that make the right thing easy and the wrong thing visible rather than making all things equally fast.

Innovation does not come from the absence of constraints. It comes from the discipline of working honestly inside them. The masons did not invent the flying buttress because somebody handed them lighter stone. They invented it because the stone was heavy and they had to figure out how to build a soaring nave anyway. The constraint set the problem, and the answer was the cathedral.

Software can build cathedrals too. It has, in the past, and it will again when the field learns what the older disciplines learned a long time ago: that the constraints are the ground good work stands on.

+++

References

[1]: Alexander, C. (1964). Notes on the Synthesis of Form. Harvard University Press, Cambridge, MA. ISBN 0-674-62751-2.

[2]: Petroski, H. (1985). To Engineer Is Human: The Role of Failure in Successful Design. St. Martin's Press, New York. ISBN 0-679-73416-3.

[3]: Deming, W. E. (1986). Out of the Crisis. Massachusetts Institute of Technology, Center for Advanced Engineering Study, Cambridge, MA. ISBN 0-262-54115-7.

[4]: Ohno, T. (1988). Toyota Production System: Beyond Large-Scale Production. Productivity Press, Cambridge, MA. ISBN 0-915299-14-3.

[5]: Ammann, O. H., von Kármán, T., & Woodruff, G. B. (1941). The Failure of the Tacoma Narrows Bridge: A Report to the Honorable John M. Carmody, Administrator, Federal Works Agency. Washington, DC. https://hdl.handle.net/2027/mdp.39015021064251

[6]: Marshall, R. D., Pfrang, E. O., Leyendecker, E. V., Woodward, K. A., Reed, R. P., Kasen, M. B., & Shives, T. R. (1982). Investigation of the Kansas City Hyatt Regency Walkways Collapse (NBS BSS 143). National Bureau of Standards, Gaithersburg, MD. https://doi.org/10.6028/NBS.BSS.143

[7]: Amazon Web Services (2017). Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region. February 28, 2017. https://aws.amazon.com/message/41926/