The short version
- A general-purpose model generates plausible text, and a legal citation is a format it can imitate perfectly. Treat every case name, statute number and section reference as unverified until you have looked it up.
- Most law that affects ordinary people is state law. A chatbot will usually give the national-average answer to a state-law question without flagging that it has done so.
- There is no attorney-client privilege in a chat with a machine. The log is electronically stored information on someone else's servers and is discoverable like any other record.
- It is genuinely good at explaining terminology, summarising a long document, drafting a first pass and listing the clauses a document is missing — all tasks where you can check the answer against the document in front of you.
What the model is doing when you ask a legal question
It is producing the most plausible continuation of your text. That is the whole mechanism, and it explains both halves of the picture. Explaining what "indemnify" means is a task where plausible and correct are the same thing, because the explanation exists in millions of documents. Naming the controlling case in your state is a task where plausible and correct come apart, because a citation is a format — a name, a volume, a reporter, a page, a court, a year — and the format is trivially easy to imitate without the underlying decision existing.
Nothing in the output distinguishes the two. There is no confidence marker, no hedge that appears only on the invented answer. This is why the failures below are not edge cases that better prompting removes. They are the shape of the tool.
Fabricated citations, and what they have cost people
The canonical case is Mata v. Avianca. In 2023 a lawyer in the Southern District of New York filed a brief containing citations to decisions that did not exist — complete with case names, reporters, page numbers and quoted passages. When opposing counsel could not locate them, he asked ChatGPT whether the cases were real, and it said yes.
How Mata v. Avianca unravelled
March 2023
The brief is filed
Six of the decisions cited do not exist. They read exactly like decisions that do.
April 2023
Opposing counsel cannot find them
The cases are absent from every database. Copies are requested and cannot be produced.
June 2023
Sanctions
A $5,000 penalty, and an order to write to each real judge whose name had been attached to an invented opinion.
Judge Castel was careful to say that there is nothing inherently improper about using a reliable AI tool for assistance. The sanction was for filing without verification, and for standing behind the citations once challenged.
It has kept happening, and to people who knew the story. In July 2025 a federal court in Colorado fined two attorneys $3,000 each in Coomer v. Lindell after a brief was filed with nearly thirty defective citations, including a fabricated Tenth Circuit decision. A public tracker maintained by the legal researcher Damien Charlotin now catalogues more than a thousand court decisions worldwide addressing AI-fabricated material, most of them from the United States, and the majority filed by litigants representing themselves rather than by lawyers.
Jurisdiction blindness is the quieter failure
A fabricated case gets caught, eventually, because someone looks for it. A confidently national answer to a question of state law never gets caught, because there is nothing to look up — the rule the model states is a real rule, correctly described, and simply not the one that governs you.
Almost everything a person is likely to ask about is state law: leases, deposits, employment, small claims limits, notice periods, non-competes. Ask whether a non-compete is enforceable and you will get a reasonable summary of the general position, which in California is wrong at the root — section 16600 of the Business and Professions Code makes every contract restraining a lawful profession, trade or business void to that extent, with narrow statutory exceptions. Ask whether you can be dismissed without a reason and the at-will answer holds in every state except Montana, which replaced the doctrine with a statute requiring good cause once a probationary period ends.
The fix is not clever prompting. It is naming the state in the question, then treating the answer as a lead to verify rather than a conclusion — and noticing that the model rarely volunteers that the answer would change if you had named a different one.
There is no privilege, and the log is not yours
Attorney-client privilege attaches to confidential communications with a lawyer for the purpose of obtaining legal advice. A chatbot is not a lawyer, no privilege arises, and there is no separate AI privilege waiting to be recognised. What the conversation is instead is electronically stored information, held by a third party, discoverable on the same terms as email or Slack.
That is not theoretical. In the copyright litigation brought against OpenAI by the New York Times and other publishers, the Southern District of New York ordered the production of a sample of twenty million de-identified user conversations, and affirmed that order over objections about user privacy. The practical point for an individual is narrower and sharper: a chat in which you set out the facts of a dispute, including the parts that hurt you, is a written record of your own account, created before you had advice, sitting somewhere you do not control.
The workaround is the ordinary one. Ask the general question without the identifying facts. Save the facts for the conversation that is privileged.
Pasting in the counterparty's document may itself be the breach
This is the failure that costs money quickest, because it does not require the answer to be wrong at all.
Is pasting their draft into a chatbot a disclosure?
Arguments it is fine
- The tool is on a business tier with training switched off
- The document is already public, or has no confidential content
- The NDA permits disclosure to advisers and service providers
Why it often is not
- Most NDAs bar disclosure to any third party without consent
- Consumer tiers commonly retain inputs and use them for training
- The adviser carve-out usually names professionals bound by their own duty
Read the confidentiality clause before the document goes anywhere. It is two paragraphs and it is the only thing that settles this.
The same analysis applies to your own obligations to clients and staff. The American Bar Association's Formal Opinion 512, issued in July 2024, treats entering client information into a self-learning generative tool as raising a duty-of-confidentiality question that generally requires informed client consent — a standard written for lawyers, but a reasonable benchmark for anyone holding someone else's data under a non-disclosure agreement.
The honest case for using it
None of the above argues for not using the tool. It argues for using it on the tasks where you can check the answer, and the set of those tasks is larger than the sceptics allow.
What each side is actually for
A chatbot, well used
- Explaining a term you have never met
- Summarising forty pages before you read them
- Listing the clauses a draft is missing
- Rewriting a clause in plain language
Both, and the cheap one first
- A first draft of a routine agreement
- Preparing the questions for a meeting
- Spotting an internal contradiction
Only a lawyer
- Advising on your facts, in your state
- Anything already in dispute
- Judgement about what to concede
- Being accountable for being wrong
The common thread on the left is that the document is in front of you. When the model says a clause is missing, you can search the draft. When it explains a defined term, you can read the definition. The verification is free, which is exactly what is not true when it tells you what the Ninth Circuit held.
Ask about a document you have uploaded
Answers are drawn from the document in front of you and cite the clause they came from, so a claim about what your contract says can be checked in one click. The general-law caveats on this page still apply — jurisdiction is still yours to name.
How to use it without any of this happening
A working discipline
- Name the jurisdiction in the question. Then ask what would change in a neighbouring state — a model that cannot answer that has not localised its first answer either.
- Look up every case, statute and section number before it leaves your hands. If you cannot find it in a primary source, delete it rather than rephrasing it.
- Ask the model to quote the clause it is describing. Fabrication survives paraphrase and rarely survives a demand for the exact words.
- Keep names, figures and identifying detail out of a consumer tool. Describe the shape of the problem, not the parties.
- Check the retention and training settings on whatever account you are using, and check them again after each product update.
- Read the confidentiality clause before pasting a counterparty's document anywhere, and use the service agreement or NDA you signed as the test, not your instinct.
- Take the output to a lawyer as a draft with three specific questions attached. That is a cheaper hour than the same hour spent from a blank page — the point our guide to AI-drafted contracts makes at more length.
The distinction worth carrying
The useful line is not between important questions and unimportant ones. It is between questions you can check and questions you cannot. Everything the tool tells you about the document on your screen is verifiable in seconds, and on that ground it is excellent and getting better. Everything it tells you about the law outside that document — what a court held, what your state requires, what the deadline is — is an assertion you have no way to test from inside the conversation, and the tool gives you no signal about which assertions to test. Sorting your questions into those two piles before you ask them is most of the skill. The rest is remembering to read the contract itself rather than a summary of it.
Sources
General information, not legal advice. This guide explains how these documents and rules generally work. Law varies by jurisdiction and changes, and none of it is applied to your circumstances here. For anything consequential, consult a licensed attorney where you are.
Frequently asked
Can ChatGPT give legal advice?
It can produce text that reads like legal advice, and it is not a lawyer, is not licensed, owes you no duty and carries no insurance. Practically, it is reliable on explanation and unreliable on anything specific to a jurisdiction or to your facts. Use it to understand a document and to prepare questions; do not use its answer as the basis for a decision you cannot reverse.
Why does AI invent case names and citations?
Because a citation is a pattern — name, volume, reporter, page, court, year — and a language model generates the most plausible continuation of a pattern. Producing a citation that looks exactly right requires no underlying decision to exist. Retrieval-backed legal research tools reduce this by grounding answers in a real database, but published testing has still found meaningful error rates.
Are my ChatGPT conversations confidential in a lawsuit?
No privilege attaches, because privilege requires a communication with a lawyer for legal advice. Chat logs are electronically stored information held by a third party and can be reached in discovery or by subpoena like any other record. Courts have already ordered large-scale production of user conversations in litigation against an AI provider.
Is it safe to paste a contract into a chatbot?
It depends on the confidentiality clause and on the tool. Most non-disclosure obligations bar disclosure to any third party without consent, and an AI provider is a third party; consumer tiers frequently retain inputs for training. Check the clause and the account settings before, not after. Where the document is genuinely sensitive, use a tool that states plainly that your content is not used for training.
What is a chatbot genuinely good at in legal work?
Four things, all verifiable against the document in front of you: explaining unfamiliar terminology, summarising something long before you read it properly, drafting a first pass of a routine agreement, and listing the standard clauses a draft is missing. It is also good at turning a vague worry into three specific questions to put to a lawyer, which makes the billed hour shorter.