The narrative around the major models today seems amazing on the face of it. Consider this article from Anthropic describing how far Claude has come and how much Anthropic code agents write now: When AI builds itself \ Anthropic
If you are new to software and systems engineering or if you have only a superficial knowledge of it, then you may have missed the most important line in that article. So, I'm going to point it out to you.
This is it: “Good code” means two things: it works, and it is written in a manner that allows another engineer to understand it and build upon it.
Why is that line the most important? Because, that definition is, by far, the lowest bar I've ever seen an experienced software or system engineer set for "good code."
There is so much more to engineering software than that.
We care, for example, about total cost of ownership. So, we learn from work on technical debt, originated with Ward Cunningham, that quick fixes create future maintenance costs, that system complexity increases engineering effort, and that architectural debt often dominates long-term ownership costs.
From Kent Beck, we learned how to avoid tangling our architectures, when he told us to "Make the change easy, then make the easy change."
Many of our industry's luminaries warned us off of complexity, including Fred Brooks, John Gall, Sandi Metz, and more.
Others have taught us that it isn't about the code itself. For example, Rob Pike taught us how important it is to get the data models right and Melvin Conway taught us about the impact of human communication on system design.
These are but a few examples of the maxims every engineer needs to know, and understand, to build cost-effective, quality software and systems that meet functional and non-functional requirements.
And this is where the model of AI agents building independently falls down.
For engineers, we don't think about these specific rules every time we write code. We develop the "muscle memory" over time. We are introduced to our industry's body of knowledge through education and mentorship, early in our careers. Through repetition, we apply these principles, only rarely needing to think about it, by the time we are mid-career. By the time we have been writing code for 20 years or more, quality designs and code are our default.
For a large language model to achieve that same quality of output, though, it would need to consider this entire body of knowledge in every decision it makes. It doesn't have "muscle memory" like we do. There are no shortcuts for LLMs. And so, the economics of quality code from LLMs just doesn't add up.
To make LLMs cost less than human programmers, you cannot design them to do as much as human programmers can do. You have to find another shortcut. You have to lower the bar for what you expect it to produce.
And so, we see model providers lowering that bar and expecting us not to notice.
[link] [comments]