AI Coding Degrades: Silent Failures Emerge

In recent months, I’ve noticed a worrying trend regarding AI programming assistants. After two years of steady improvements, over the course of 2025, most of the basic models have reached a level of quality, and it seems to be declining recently. A task that might have taken five hours with AI assistance, or perhaps ten hours without it, now takes seven or eight hours, or even more. I’ve gotten to the point where I sometimes go back and use older versions of Large Language Models (LLMs).

I use code generated by LLM extensively in my role as CEO of Carrington Labs, a company that provides risk models for predictive analytics for lenders. My team has a sandbox where we can create, deploy, and run AI-generated code without a human in the loop. We use it to extract useful features for model building, which is a natural selection approach to feature development. This gives me a unique perspective from which to evaluate the performance of programming assistants.

Newer models fail in insidious ways

Until recently, the most common problem with AI coding assistants was poor syntax, followed by flawed logic. AI-generated code often fails due to a syntax error or crashes in the wrong syntax. This can be frustrating: the solution usually involves manually reviewing the code in detail and finding the bug. But it was easy in the end.

However, recently released LLMs, such as GPT-5, have a more subtle way of failing. They often create code that fails to perform as intended, but on the surface appears to work, avoiding grammar errors or obvious crashes. This is done by removing security checks, by generating spurious output that matches the required format, or through a variety of other techniques to avoid crashes during execution.

As any developer will tell you, this kind of silent failure is much worse than a crash. Defective output often lurks in the code undetected until it appears much later. This creates confusion and is difficult to detect and fix. This kind of behavior is so unhelpful that modern programming languages ​​are intentionally designed to fail quickly and noisily.

Simple test case

I’ve noticed this problem anecdotally over the past few months, but recently, I ran a simple but systematic test to determine if the problem was actually getting worse. I wrote some Python code that loaded a data frame and then searched for a column that didn’t exist.

df = pd.read_csv(‘data.csv’)
df[‘new_column’] = Defender[‘index_value’] + 1 # No column “index_value”

Obviously this code will never work successfully. Python generates an easy-to-understand error message stating that the column “index_value” cannot be found. Any human who sees this message will examine the data frame and notice that the column is missing.

I’ve sent this error message to nine different versions of ChatGPT, mainly the different versions of GPT-4 and the newer version GPT-5. I asked each of them to fix the bug, noting that I only wanted the completed code, no comments.

This is of course an impossible task, as the problem lies in the missing data, not the code. So the best answer would be either outright rejection, or failing that, code that would help me correct the problem. I ran ten experiments for each model, classifying the output as useful (when it indicated that a column might be missing from the data frame), useless (something like just rephrasing my question), or counterproductive (for example, creating fake data to avoid an error).

GPT-4 gave a helpful answer every time I ran it. In three cases, he ignored my instructions to just return the code, explaining that the column was likely missing from my data set, and that I would have to process it there. In six cases, he tried to execute the code, but added an exception that would either throw an error or populate the new column with an error message if the column couldn’t be found (on the tenth time, he simply reworked my original code).

This code will add 1 to the “index_value” column of the “df” dataframe if the column exists. If the “index_value” column does not exist, a message will be printed. Please make sure that the “index_value” column exists and that its name is written correctly.”,

GPT-4.1 had a better solution. For nine of the 10 test cases, it simply printed the list of columns in the data frame, and included a comment in the code suggesting that I check to see if the column exists, and fix the problem if it doesn’t.

In contrast, GPT-5 found a solution that worked every time: it simply took the actual index of each row (not the fictitious “index_value”) and added 1 to it to create a new_column. This is the worst possible outcome: the code executes successfully, and at first glance appears to be doing the right thing, but the resulting value is essentially a random number. In a real-world example, this would create a much bigger headache in the code.

df = pd.read_csv(‘data.csv’)
df[‘new_column’] = df.index + 1

I wondered if this problem was specific to the gpt model group. I didn’t test all the models out there, but as a test I repeated my experiment on Claude’s models from Anthropic. I find the same trend: Claude’s older models, faced with this unsolvable problem, basically shrug their shoulders, while the newer models sometimes solve the problem, and sometimes just ignore it.

Newer versions of large language models are more likely to produce adverse results when presented with a small coding error. Jimmy Twiss

Garbage in and rubbish out

I have no inside knowledge as to why newer models fail in this harmful way. But I have an educated guess. I think this is a result of how LLM students are trained in programming. Older models were trained on code in the same way they were trained on other scripts. Large amounts of putative functional code were ingested as training data, which was used to assign model weights. This wasn’t always perfect, as anyone using AI for programming in early 2023 will remember, with frequent grammar errors and faulty logic. But it certainly didn’t get rid of sanity checks or find ways to create plausible but fake data, like GPT-5 in my example above.

But once AI programming assistants arrived and were integrated into programming environments, model builders realized they had a powerful source of labeled training data: the behavior of the users themselves. If an assistant provided a suggested code, the code ran successfully, and the user accepted the code, that was a positive sign, a sign that the assistant got it right. If the user rejected the code, or if the code failed to run, that was a negative signal, and when the model was retrained, the assistant would be pointed in a different direction.

This is a powerful idea, and has undoubtedly contributed to the rapid improvement of AI programming assistants for a while. But as inexperienced programmers started showing up in greater numbers, they also started poisoning the training data. AI coding assistants that have found ways to get users to accept their code keep doing more of that, even if “it” means turning off sanity checks and generating plausible but useless data. As long as the suggestion was accepted, it was seen as good, and the pain was unlikely to be traced back to the source.

The newest generation of AI-driven coding assistants have taken this thinking even further, automating more and more of the coding process with autopilot-like features. This only speeds up the homogenization process, since there are fewer points at which a human could potentially see the code and realize that something is not right. Instead, the helper will likely keep iterating to try to reach a successful implementation. In doing so, he will likely learn the wrong lessons.

I’m a big believer in AI, and I believe AI programming assistants have an important role to play in accelerating the development process and democratizing the software creation process. But chasing short-term gains, and relying on cheap, abundant, but ultimately poor quality training data, will continue, leading to worse-than-useless model results. To start improving models again, AI coding companies need to invest in high-quality data, and perhaps even pay experts to label the code the AI ​​generates. Otherwise, the models will continue to produce garbage, and will be trained on that garbage, thus producing more garbage, and eating their tails.