dysphoric.dev

AI Code Generation Is Just Legacy Code on Demand

First of all

I do not believe that the way people talk about AI is productive at all.
There seem to be two prevalent opinions which are entirely incompatible with each other: One side is arguing that AI will change everything, become more important than the internet, and make everyone obsolete, while the other is arguing that it is entirely useless, evil, and that a single query to an LLM uses up more energy than flying transcontinentally.
I disagree with both of these extremes. I find it important to note that AI in its current iteration is nothing new. It is mainly statical analysis, which we used to call Big Data a few years back, and probably something else before that. It simply got combined with a huge amount of GPUs suddenly finding themselves without use after the Blockchain hype died down and executives scrambling to find something that would get investor money flowing again after banks stopped giving away free money when the war in Ukraine started.
Data analysis has strengths and weaknesses, and I am not going to argue that the current iteration of chatbots is anything less than amazing. While I can smell a ChatGPT generated message from a mile away, I do think it is pretty cool what it can output when you don't take it too seriously. It can feel like a real human's writing in certain situations, but if you've been on twitter during the 2010s, you probably saw "eBooks bots", which were Markov chains trained on people's posts that could generate complete nonsense which sounded like a post made by them.
They are so good at that that I started blocking them on sight at some point, because I was spending way too much time trying to figure out what these posts where trying to tell me before realizing that they were made by a Markov chain. AI Chatbots are better at making up stuff that actually makes sense, but you can not, and due to the fact that they are just statistical models without any actual understanding of what they are saying, will never be able to trust that what they say is actually true. That does not make them useless. They can be great for changing the tone of a message, finding key points in long text, and if they were trained on a good, custom, dataset could revolutionize in IDE code completion. Everything they put out must be fact checked and verified, but doing so can be easier than writing a summary yourself.

So, now you know where I stand on this whole thing. I don't think it's the devil incarnate, but I also understand that it is not the complete revolution everyone is trying to sell to combat their falling stock prices. They will, with time and polish, become a tool among many in the toolkit of people who work with computers, and I think that can be pretty neat.
I do find it difficult, however, to explain to people why fully AI generated code or "vibe coding" is a terrible idea without sounding elitist. I feel that I have found a way to do so today, which is why I am writing this article.

Setting the scene

I am currently learning Kotlin for a work project. It is the first time in too long that I am learning a completely new stack, and I feel like it has gotten substantially more difficult since I last did it.
The reason for that is not that I got too old to learn new stuff -I am hopefully decades away from that if it ever happens. What is really making it difficult is the fact that Google has gotten much worse in recent years.
Most results seem to be AI generated SEO spam, websites that only exist to scrape StackOverflow and GitHub issues while removing them from their context, and complete nonsense.
Once I find an article that seems to be written by an actual human, there also is sadly a very high chance that it will break with all of the best practices and conventions I still have fresh in mind from learning them just last week, while not explaining why they do deviate.

Realizing once again that the vast majority of code samples you can find on the internet are not actually functioning code that should be copy pasted into people's projects made me think of AI, and how this is exactly what it was trained on.
Turning bad input into good output is not within the realm of statistical analysis. It is alchemy, and as far as I know we still haven't figured out how to do that.

But what does that have to do with legacy code? To explore that, I first need to provide a definition, since not everyone means the same thing when they say legacy.

What even is legacy code?

Years ago I watched a talk that was somewhat eye opening to me. I sadly forgot who it was from or what the title was, but there was one piece of it that really changed my way of thinking: The speaker argued that legacy code is any piece of code whose author is no longer around to defend or explain it.
I personally would add that it is also code whose author does not care to defend or explain it, since I have fixed a lot of my own legacy code that I wrote earlier in my career.
I believe that most people write code with good intentions, and that they often have reasons for breaking with existing patterns. As long as those reasons are documented well, or explained in another medium, the code is not legacy and can be easily worked with. As soon as that context goes missing, any break with predefined patterns and rules turns from being a quirk to being a roadblock to further development.
If you want an example of this, think of that often shared code comment:

/*
* Dear Maintainer
*
* Once you are done trying to 'optimize' this routine,
* and you have realized what a terrible mistake that was,
* please increment the following counter as a warning
* to the next guy.
total_hours_wasted_here = 73
*
*/

I assume that the code in question does its job reasonably well, and that the weirdness of it that people want to optimize away has its reason for being there, otherwise there would be no need for the comment. The only issue is that the original author did not document their reasoning for the weirdness and they seem to no longer be around to ask, leading to hours and hours of wasted time.

So, we now have established a shared definition of what legacy code is: Quirky code without explanation.
I don't think any major project can exist without accumulating at least some of this. People tend to think that their thought process must be obvious to others, so documentation is not needed when it is, sometimes there simply is not enough time to do a job well, and the cleanup after the deadline keeps being procrastinated because other projects and deadlines are upcoming, or and other host of reasons contribute to it. We can, and should, minimize it, but it will always be there in some way.
Therefore any code that an AI could be trained on will contain these quirks. No matter if it is someone's CV enhancing medium post that mainly regurgitates half understood docs, or if it is a huge project used by billions of people, there will always be weirdness and jank.

So where does AI come into this?

You can probably already see where I am going with this, but I am going to spell it out regardless.
AI generated code is exactly what I defined as legacy in the previous article. The original author is not around to explain why the code is weird, because there is no original author. There is a statistical model that can only output things that are likely to exist near each other. It may sound like it understands, but it does not.
If you get lucky, the model might've been trained on similar looking quirky code that came with an explanation, so it might be able to come up with something that sounds plausible, but there is no, and there can never be a, guarantee that this explanation actually makes sense.

AI code generation makes it very easy to create code that might even look like it does what one wants it to on the surface, but as soon as it falls apart, it will leave the developer on their own to figure out what went wrong.
In the worst case they have vibe coded themselves into some well paid position where they now have to fix their work with a deadline approaching and no idea how the technology they use actually works, leaving the rest of their team with a huge chunk of extra work to clean up the mess after said employee has been let go.

Anyone who programs professionally and has made it beyond the junior level will also know that most of our time is not spent writing code. In a typical week, that is maybe 10% of my work time. (-and it is the most fun part of my job, so why would I even want it automated away?)
The rest of it is spent reviewing, understanding, and figuring out how to work with existing code.
An AI cannot do any of that because it lacks the understanding required to do it.
It will respond confidently if you ask it to, but it will often be confidently wrong without a way to explain itself or understand where its misunderstanding lies, because there is no misunderstanding. There is only a statistical likeliness for these words to be used in this context and order, with some post processing to prevent grammatical or spelling errors and slurs.

How did we even get here?

Here in Berlin it is rare that I go out to a café or to dinner without somehow ending up overhearing someone tell their coworkers/ potential investors/ dates (I still feel bad for that woman, she seemed miserable) how amazing AI technology is and how much they've been able to do with it.
These people are never professional programmers.
They usually are business people who want to build a company to get rich off selling, and I will be the first to admit, that they are probably not having a particularly fun time interacting with programmers. One person even directly said as much in a conversation I overheard.
We can be a difficult bunch, and we keep saying things like "this is impossible", or "this will cost thousands of euros to implement".
ChatGPT does not do that. If you talk to it like a business person, it will talk back to you like a business person.
One of the first things I asked it to do was solve the halting problem, and it put out some okay looking python code to do so. Even if that specific issue has been fixed since, it still does not understand what is impossible so it can not say no to those things without a hard coded blocker preventing it from answering certain queries.

There is an understandable need by would-be founders for something that will turn their ideas into an actual product without requiring months work, difficult conversations, and a lot of money.
People selling AI tooling have understood that, and they are capitalizing on it. They want their potential customers to believe that it only takes a 15$ a month subscription to get their idea to a state where they can sell it to VCs.
The scary thing is not even that they are selling a lie here. The scary thing is that it might even be partially true. The founder might get to a point where they have a flashy demo to show VCs, get a high evaluation, and then run away with the money as the people who are left are trying to pick up the pieces and turn it into something that actually works without leaking customer data through IDOR, SQL injections and the like.
Anyone who tries to build up an actually sustainable business that ends up profitable, will likely not get much out of having AI generate code. They might believe that they do, because the vendors are very good at selling that, but in the end it will likely backfire catastrophically and make it so that any runway they have will need to be used up on redoing everything they have so far. If they plan on doing that, they would be better off just having a designer build an interactive wireframe with Figma or something similar, and then hire devs to turn it into something that can actually be used.

Conclusion

So, what to make of all of this?
I think that the main message should be that a lot of us needs to be a lot more careful now that the Pandora's box of semi competent code generation has been opened. Hiring managers will need to learn to not let themselves be blinded by a huge amount of green GitHub squares and a perfect FizzBuzz solution, and early stage investors need to get technically competent people to interview founders about the actual technical stack of the thing that they might buy.