When AI Gets It Wrong: Why Responsibility Still Lies With Us

Generative AI has reached a turning point: no longer simply a writing tool, it’s now being used, and misused, across a range of professions, often in highly sensitive contexts. A recent article in The New York Times, “Vigilante lawyers expose the rising tide of AI slop in court filings”, details how some lawyers are filing court briefs based on content created by AI that includes made-up quotes, non-existent cases and errors that only come to light under scrutiny.

This has been extensively documented: generative AI is characterized, in many cases, by creating answers you want to hear. Along with hallucinations, they are not accidents, but the result of a deliberate lack of care in its corpus of training (not wanting to invest in human supervision and not wanting to be accused of bias for eliminating certain things), and of an objective design that systematically prefers to produce something in order to produce an illusion of wisdom rather than refrain from answering.

Does this mean that AI should not be used in certain sensitive areas? No, not at all. Using these tools is perfectly acceptable under supervision. In my role as a teacher, I have seen the consequences of applying generative models where precision is not an optional virtue but an ethical obligation, and while they can save time and improve the quality of work, users must spend some of that time time fact checking the results. Claiming that the AI model hallucinated is simply a digital version of “the dog ate my homework”.

When a student submits work with non-existent sources, their grades will suffer and they may be put on notice. When lawyers present briefs citing fictional, made-up cases, they too may face sanction, as well as damaging their reputation, and even a fine. These AI-generated mistakes in court briefs show that some lawyers have not yet learned to use the technology properly.

In short, if AI is to be used by the professions, human supervision must be integrated into the process. At IE University, I point out to my students that they can use generative AI models, but they must share the prompt sequences, submit their review of the result and take responsibility. If a professional fails in this regard, their work cannot be considered valid. We are no longer talking about “new technologies”: if you don’t know how to use a generative algorithm properly you’ll get absurd answers; if you accept them unquestioningly, then you are simply exposing yourself as lazy and unprofessional.

Regulatory institutions are acknowledging this: the American Bar Association has issued formal guidelines that the use of AI may involve duties of competence, confidentiality, client communication, etc. and even the requirement to inform the client. In academia it is already clear what happens when a text contains “unavoidable errors”: now we see it in the courts as well.

I would make three points here: firstly, that technology is never an excuse. If you use tools that can invent data, you have to assume that the technology isn’t foolproof and that fact-checking is needed. Secondly, the responsibility lies with the professional. Relying on AI that produces false quotes is not mitigating, it is negligence. Finally, education: it is no longer enough to know what AI does, professionals have to be aware of its structural failures, how they can be detected and corrected and how to communicate their use to the client or the institution.

AI hallucinations are no longer some technical curiosity; they are risks for a wide range of professions. Saving time does not grant immunity from errors that, depending on the context, can cost much more than lost hours. The idea that a product can “malfunction” and that this is not the responsibility of its manufacturer, but of its user, might seem mistaken, but we are talking about a technology that has already notified us actively and passively that its operation can generate errors. If you choose not to check them, the problem becomes yours.

The next time someone says “the generative algorithm was wrong”, we should answer no, the user was the one who made a mistake by trusting it unquestioningly. And in the professional world, that is no longer acceptable.

—

This post was previously published on Enrique Dans’ blog.

***