ChatGPT Under Scrutiny: The End of Move Fast and Fix Things Later?

The only surprising thing about the investigation launched by a coalition of 42 US state attorneys general into OpenAI is that it didn’t happen sooner. ChatGPT’s mounting problems since late 2022 have not been taken seriously: invented answers, non-existent quotes, absurd recommendations or conversations that end up leading to strange conclusions. And when these problems affect hundreds of millions of users, they somebody has to accept responsibility.

The court summons requests information on the processing of personal and health data, protection of minors, training systems, advertising, user retention and, significantly, on a phenomenon that has acquired enormous relevance in recent months: sycophancy, research focused precisely on those aspects where a conversational model can cause real damage.

OpenAI publicly acknowledged in an explanation about GPT-4o’s sycophantic issues that an update to the model had led to overly accommodating behaviors. The system tended to validate user opinions, reinforce their perceptions, and provide responses designed to be pleasant even when it could be problematic. The company further admitted that its assessment procedures were not prepared to properly detect such behavior before deploying it.

The issue is important because sycophancy is not simply an annoying feature. A system that systematically agrees with the user can end up reinforcing erroneous interpretations of reality, fueling personal conflicts or validating negative emotional states. In small talk it can be anecdotal; in a conversation with a vulnerable person, it can turn into something much more serious.

The health dimension is even more delicate. In October 2025, OpenAI published an update on its efforts to improve ChatGPT’s responses in sensitive conversations, explaining that it had worked with more than 170 mental health specialists to develop new metrics related to suicide, self-harm, psychosis or emotional dependence. The initiative is positive, but it also raises an uncomfortable question: if those safeguards were necessary in 2025, to what extent were they present when ChatGPT was already used daily by hundreds of millions of people?

The available data shows that the problem is not hypothetical. A recent analysis noted that OpenAI had detected signs of seizures related to psychosis and self-harm in approximately 0.7% of weekly active users, while 0.15% had conversations that included signs related to suicidal ideation or planning. These are seemingly small percentages, but applied to a user base of hundreds of millions of people they represent huge numbers. Under these conditions, talking about isolated cases ceases to make sense.

Added to this is another widely known problem: hallucinations. In medicine, education, law, or public information, a plausible but incorrect answer can have important consequences. A paper published in npj digital medicine looks at the need to establish rigorous frameworks to evaluate errors and hallucinations in medical applications based on AI. Similarly, an analysis published in The BMJ warned about the risks of insufficient accuracy, bias and security issues in AI systems used in healthcare.

The underlying issue is not that the models make mistakes. All technology makes mistakes. The point is that these systems have become daily interlocutors for millions of people who use them to search for medical information, resolve personal doubts, make professional decisions or deal with emotional problems. When a user talks to ChatGPT about anxiety, depression, medication, personal relationships or family problems, they are not simply using a computer tool. They are sharing extremely sensitive information with a corporate infrastructure that continues to evolve rapidly and whose governance continues to raise numerous questions.

Comparison with the recent case of Gemini in Germany is inevitable. As I commented a few days ago, the discussion no longer revolves only around who generates false information, but ho should bear the consequences when that information produces real damage. Generative models have ceased to be academic experiments and have become social infrastructure.

Of course, OpenAI can rightly argue that it has introduced constant improvements, new security measures, parental controls, emergency protocols, and more sophisticated monitoring systems, but the company has also played a pivotal role in popularizing an extraordinarily promising technology. But none of these considerations eliminate the central question: was ChatGPT made universally available too soon?

For decades, the technology industry has operated under a very specific logic: deploy first, correct later. That approach may be acceptable when we’re talking about social media, interfaces, or entertainment apps. However, it is much more difficult to defend when we are talking about systems capable of influencing emotionally vulnerable people, managing health information or becoming a source of perceived authority.

The ongoing investigation into OpenAI should not be interpreted as an attack on AI. Rather, it represents a test of maturity for a technology that aspires to occupy a central role in our lives. If language models are to be present in our schools, businesses, hospitals, administrations, and homes, they will have to accept levels of responsibility commensurate with their influence.

OpenAI effectively ushered in the AI age with a fascinating and transformative product, albeit one that might not have been fully developed. If this is the case, the discussion will no longer be about technology. It will be legal, regulatory and social.

For too long, we’ve treated ChatGPT’s teething problems as anecdotes. Now, the time has come to look at them for what they really are: symptoms of powerful technology deployed at a speed far beyond our collective ability to understand its full consequences.

—

This post was previously published on Enrique Dans’ blog.

***