Settings

Theme

Was software just always too complex for humans?

anil.recoil.org

21 points by nathcd · 14 comments

Reader

12 threads
analog31

Take this from a rank amateur "scientific" programmer. I write functions with maybe a half dozen lines, then I piece them together. Sometimes my programs work, sometimes they don't, and I fix them. I've done it this way since I first learned Pascal 40+ years ago. I'm not allowed to touch production code, so don't panic.

When I ask AI to write code with similar functionality, it's about 10x more code. Likewise when I look at the code for well respected open source projects. It's good code in terms following accepted practices, but hard to read because there's too much of it to see where the actual "working" code is.

A lot of the code is "defensive," i.e., type and bounds checking, trapping error conditions, etc. When I've looked at the unit tests, most of them test the defensive code.

I tell my engineering friends: "Engineers like to over-complicate things." Maybe the AI has picked up on this and is serving it back to us.

yyyk

No, and users of this site know better. Humans built massive reliable software system. Inasmuch as anything was 'too complex', it was the choice to rely on underspecified interfaces, ancient tools and legacy protocols for various reasons (often boiling down to economics of time and resources). But it would have been humanly possible to rebuild the entire software stack using more reliable methods (that the article suggests for LLMs, but there's nothing yet making these methods inaccessible for mere humans), and these reliable methods would have solved nearly all bugs. It would just be a lot less economic and take more time (both directly and as opportunity cost).

maxrev17

It’s just how quickly you can deploy the resource right? Models are not bad but you can deploy tonnes of energy quickly. With people you can’t deploy as quickly. Maybe?

hypfer

Answering the Headline: Not really, no.

Things are the way they are not because they would be difficult but because various systemic incentives and goals fought out each other, leading to the outcomes we then saw.

This balance these various forces fought out relied on a bunch of undocumented assumptions about reality. Every time one of those breaks, you get disruptions. LLMs broke quite a few of them, leading to a lot of CVEs lately.

dgoodell

I think it was a business decision to make it too complex for humans. I don’t think fundamentals were prioritized as much as features, resulting in a fragile house of cards.

There’s a reason it doesn’t look like most engineering. If it was a bridge it would have collapsed long ago and killed people.

amelius

The always changing requirements are what make it too complex for humans.

randallsquared

Specification has its place, but even with "perfect" pieces of software, the interaction of them, together with their interaction with the physical hardware, mean that you still can't know everything that can happen in the whole system. You can specify a unix tool (do one thing and do it well...), but you can't formally prove all of an operating system plus userland without very strictly limiting what outcomes are possible, and even then novel or buggy hardware or other physical environment variables could produce unexpected results: bugs. The formal specifications themselves, to cover all this, will have to become so complex that no human can understand them, which just means that some potential exploits will be explicitly allowed by the spec, and an evaluator at a given level of intelligence (human or other) may not understand the bug or even that it exists.

For any system designed by some intelligent agent IA, there's going to be holes/bugs/exploits that can be easily seen by some agent IA*100. This has never been a serious problem in anything before, because humans don't span that range of intelligence, and collaboration doesn't provide such insights.

ssrc

Maybe for some software systems it was. For most systems, the problem was just professional immaturity caused by the eternal September state the profession is in since the beginning.

0c3ca83

Betteridges law of headlines strikes again.

yepyoukno

Many handle it just fine.

ex-aws-dude

I mean if you zoom out the whole lesson we keep learning over and over with AI/ML is “we are not smart enough to design this function”

AIShill101

Yea I agree with the article wholly. This new era of AI assisted or driven coding is going to help us extend our capability way beyond anything humans could've built.

  • onion2k

    This new era of AI assisted or driven coding is going to help us extend our capability way beyond anything humans could've built.

    For individuals, sure. I'm fairly sure we're all building things that we couldn't build ourselves if only because we don't have the time needed. For large scale projects like building an OS from scratch or a database engine that's a lot less likely to be true, simply because there are dozens of examples of humans doing that level of complicated work without AI already (the OSs and DB engines we have are very literal evidence of it.) They just need to work collectively. Every part of those projects is understood people a person even if no person understands the whole thing.

    So the problem with doing the same sort of thing with AI becomes a question of whether someone can build the equivalent sort of project with AI by describing something more complex than they understand themselves. And even if they just ignore the complex details and let AI do that bit, how can they validate that the AI has actually done what they asked if they can't understand it? It might have taken a shortcut that works until it doesn't. I hope no one builds an aviation or a life support system by doing that.

    Ultimately, I suspect we want to be able to understand the system even if it limits what we build. A system that fails in an edge case carries significant risk which people probably don't want to be liable for.

  • dingaling911

    At the moment, oversight is still heavily required.

    Before, we used to write code and machines converted it to binary.

    Now we will write designs and machines will convert it to code (and then to binary.)

    There is a possibility of models taking over design for somewhat run-of-the-mill applications with minor architectural tweaks. This is the area that "vibe coding" is currently dominating. How many vibe coded apps, written by others, are you using? None. So that tells you something.

    At best, models are currently at VB6 level: domain experts can do a lot of good. But they could've done it with VB6 as well. And MS Access. And Excel.

    I am not sure how we would give the models an idea of taste and beauty because that is often what differentiates good/bad from great designs.

Keyboard Shortcuts

j
Next item
k
Previous item
o / Enter
Open selected item
?
Show this help
Esc
Close modal / clear selection