What 'AI Hallucination' Means: A Guide for Law Firms
By Stacey Tallitsch | September 9, 2026
You have probably heard the word "hallucination" attached to AI by now. Maybe it came up in a news story about lawyers who got in trouble. Maybe one of your kids used it at dinner. It is a strange word to hear about a computer program. Most people nod along without ever getting a clear answer about what it means.
Here is the plain version. An AI hallucination is when an AI tool tells you something false and says it with complete confidence, as if it were a fact. The tool is not lying on purpose. It does not know it is wrong. It produces an answer that sounds right, and sometimes that answer is simply made up.
If you run a law firm, this matters more to you than to almost anyone else. Your work depends on facts being exactly right. A wrong case name or a wrong deadline is not a small typo. So it is worth a few minutes to understand why this happens and what to do about it.
Why would a computer make things up
To understand hallucination, it helps to know one thing about how these tools work. A tool like ChatGPT is built on what is called a large language model, or LLM for short. That is a program trained to predict the next word in a sentence, based on patterns in an enormous amount of text. I wrote a plain-English explainer on what a large language model actually is if you want the longer version.
The key point is this. The tool is not looking anything up. It is not checking a library or a database of real court cases. It is guessing the next word, over and over, until it has built an answer that reads well.
You have seen a smaller version of this before. When your phone finishes your text message for you, it is doing the same basic thing. It predicts the next likely word. An AI tool does this at a much larger scale, across whole paragraphs and pages. Most of the time the guess is good. The trouble starts when a confident guess gets treated as a checked fact.
Most of the time, the best-sounding answer is also true. But not always. When the tool does not have a solid fact to draw on, it does not stop and say "I do not know." It fills the gap with something that fits the pattern. That filler can be completely invented, and it will look just as polished as the real thing.
OpenAI, the company behind ChatGPT, published research on this in 2025. Their researchers argue that the way these tools are trained and tested rewards confident guessing over admitting uncertainty. You can read their own write-up on why language models hallucinate. The short version is that the tool was built to always give you an answer, even when the honest response would be that it does not have one.
What this looks like in a law firm
The most famous example happened in a real court case. In 2023, two New York lawyers filed a brief that cited six court cases to support their argument. The cases looked real. They had names, judges, and direct quotes.
None of them existed. The lawyers had used ChatGPT to help with their research, and the tool had invented all six. When the other side could not find the cases, the judge could not either. Both lawyers were fined $5,000. The Association of Corporate Counsel has a clear breakdown of the lessons from that case if you want the full story.
This is the risk in one story. The tool did not refuse the request. It did not warn anyone. It handed over six cases that read like the genuine article, and two experienced attorneys trusted them.
It does not only show up in litigation. Picture a small firm handling a real estate closing or an estate plan. Ask the tool for the filing deadline on a form, or which statute governs a step, and it can hand you a confident answer that is wrong. The number looks right. The statute sounds real. That is exactly the trap.
Two things people get wrong about this
The first mistake is believing hallucinations only happen with strange or obscure questions. They do not. They happen most often with the specific details your firm relies on every day. A case citation. A statute number. A filing date. A dollar figure. These are the facts the tool is most likely to invent, because they are precise and it feels obligated to produce something.
The second mistake is assuming the newer tools have fixed this. They are better than they were a year ago. They hallucinate less. But "less" is not "never." No tool on the market today can promise you it will never state a false fact with a straight face. Anyone who tells you otherwise is selling something.
So what should you actually do
You do not need to be a technology expert to use these tools safely. You need to use them for the right jobs and check their work on the rest.
Think of the AI as a fast assistant who is sometimes wrong. You would never file that assistant's research without reading it first. The same rule applies here. Anything the tool gives you that is a fact you would sign your name to, you verify against the real source. A case, a number, a name, a rule. Look it up yourself before it leaves your office.
A simple habit protects you. Before you rely on anything the tool states as fact, ask it one question: where did that come from. If it cannot point you to a real source you can open and read for yourself, treat the claim as unconfirmed. Then check it the way you always have, with the actual case, statute, or record in front of you.
Some jobs are low risk. Rewording a paragraph you already wrote. Summarizing a document you handed it. Drafting a first version of a client email in plainer language. In those cases the tool is working with material you provided, so there is little for it to invent. That is the sweet spot for a busy firm. You get faster drafts, and the facts still come from you. The way you ask also shapes the result, which I covered in why the way you word a request changes the answer.
Other jobs are high risk. Asking the tool to recall facts, find cases, or state what a law says. That is where it invents. And if you have wider concerns about handing these tools sensitive client material, I wrote a related piece on whether AI learns from the information you type in.
When this matters
Hallucination matters most at the exact moment you are about to trust the tool with something you cannot afford to get wrong. For a law firm, that moment comes often. So the habit is simple. Use the tool freely for drafting and rewording. Verify every fact before it carries your name. None of this requires new software or a change to how you run the firm. It requires one rule, applied every time. Do that, and a made-up case never makes it into your brief.
-- Stacey | The Standalone
About the Author
Stacey Tallitsch runs The Standalone, an AI Implementation Diagnostic practice for small business owners. He has 30 years of experience in technology and has written 21 books on systems thinking and decision-making. More than 30,000 students have learned from his online courses.
- Stacey Tallitsch, The Standalone