Why AI Invents Cites

Primary sources are linked where cited — Vaswani et al. and Mata v. Avianca. Current as of August 22, 2026.

The box looks like a search field. A few seconds later you get a paragraph that sounds like a careful associate. If you’ve spent a career on Westlaw, that’s the natural story: the machine went and found something.

It didn’t.

I already wrote what these tools are good for. This is how they work.

What a token is

When you type into the box, the software doesn’t keep your sentence as a sentence. It chops the text into tokens.

A token is a bit of text the model can handle. Sometimes a whole word. Sometimes a piece of a word. Sometimes punctuation. “Holding” might be one token. A longer word might be two or three. You don’t need the inventory. You need the unit.

Given the tokens already in front of it, the model produces a probability for every token it knows. Software picks the next one. That pick is appended. The model conditions on the new string and does it again, until a stop rule fires.

That loop is next-token prediction. The 2017 paper that put the modern version of this architecture into wide use is Vaswani et al., Attention Is All You Need. Section 3 describes a transformer decoder that is autoregressive: each step conditions on what came before. Section 3.4 is the softmax step that turns raw scores into next-token probabilities. Current large language models are mostly decoder-only descendants of that design. The brand on the box changes. The job does not.

A large language model is a next-token sampler conditioned on context. Token means the bit. Sampler means the thing that picks. Context means the tokens sitting in front of it on this call.

You look the case up. The machine keeps writing.

In English. The box does not keep your sentence as a sentence. It cuts the sentence into bits. Then it guesses the next bit, adds it, and guesses again.

Training

People hear that a model was “trained on the internet” and picture a warehouse. They picture a clerk who can walk the stacks and pull Smith.

Training is a different process. In pretraining, the model is shown enormous amounts of text and asked, over and over, to predict the next token. The loss is how wrong that prediction was. Weights move so the next guess is a little less wrong. Do that enough times and you get fluency. The model becomes very good at continuing a string in a way that looks like the text it saw.

The weights are parameters. You cannot walk them and pull a case.

After pretraining, vendors usually fine-tune. Some of that is supervised examples. Some of it is preference work, including RLHF and later preference-optimization methods. Those stages shape style and refusal. They teach the model to sound helpful, or to decline a certain class of request. They do not install a reporter the model can query later.

Inference is the live call. Training already happened. On this call the weights are fixed. The only new information is the context you put in the window. If a case appeared in the training text years ago, that does not mean the model has the case. It means some statistical residue of citation-shaped language may still be in the weights. Residue is not a reporter.

“The model knows Smith” can mean two things. It can emit a string that looks like a cite to Smith. That is cheap. Or a retriever pulled the opinion into the window. That is a different machine.

In English. Training is practice at guessing the next bit. After that, the model can sound like the text it saw. It still cannot walk to a shelf and pull a case.

The window

The context window is working memory.

Every call has a budget of tokens the model can condition on: your prompt, the system instructions, any files that were stuffed in, and the tokens it has already emitted on this turn. Outside that window, for that call, there is nothing.

What sits in the window is not raw. Software ranks it, filters it, decides who can see it, and orders it. That step is context assembly. Privilege starts there, not at the sampler.

“The model read the file” means tokens from the file were in the window. If the file was too long and the system truncated it, the unread part was not read. If last week’s chat rolled off the window, last week’s chat is gone. If a privileged memo sat in a side pane the product never tokenized, it never entered the sampler.

Privilege questions, “what did the tool see,” and “did we put the whole contract in” collapse to the same check. Which tokens were in the window on that call? Keep a record of that when the answer matters.

In English. On this call, the model can only see what fits in the box. What gets into the box is assembled first. If the file was too long, the rest was not read. If last week’s chat fell off, it is gone.

Fluency

Hallucination gets treated as a bug. For a next-token sampler, fluent continuation is the objective.

The model is trained to produce the token that would have come next in text like the text it saw. Legal writing is full of citation-shaped tokens: Id., F.3d, “the court held.” Those shapes are cheap. A plausible caption is cheaper than a real one, because a real one has to match a document that exists outside the window.

Mata v. Avianca, No. 22-cv-1461 (PKC) (S.D.N.Y. June 22, 2023), is the case everyone already knows. Steven Schwartz used ChatGPT. The tool fabricated cases. At the sanctions hearing he described “the false perception that this website could not possibly be fabricating cases.” His other line is the one I still hear in rooms: “ChatGPT is finding that case somewhere.” Judge Castel wrote that there is nothing inherently improper about using a reliable artificial intelligence tool. He also wrote that Rule 11 still imposes a gatekeeping role. The court imposed $5,000 and required letters.

Schwartz described a retrieval machine. He was using a sampler. Citation-shaped tokens came out. He treated them as holdings.

In English. The machine is good at sounding finished. A fake cite can look like a real one. Sounding like a holding is not the same as having one.

Sampling

At each step the model produces scores, often called logits, over the next token. Softmax turns those scores into probabilities. Software then samples.

Temperature is a knob that rescales the logits before that step. Raise it and the distribution flattens, and more of the less-likely tokens get a chance. Lower it and the distribution sharpens. The model sticks closer to its top pick. Some products hide the knob. Some set it for you. Naming the button “Precise” doesn’t turn a draw into a deposition.

Two runs can differ. Same prompt, same model, same temperature. Sampling is sampling.

“The AI said” is a thin record unless you can say which run, which context, which temperature, and which tool calls. If you cannot, you have a draw. That is why a mature system keeps a log. The log is the statement. The sampler is not.

When the output will be shown to a court, a regulator, or a board, treat it like a draft from an unnamed associate who will not be available for a declaration. You can use the draft. You can’t put the sampler on the stand.

In English. Each next bit is a pick. Run it twice and you can get two answers.

Retrieval

RAG means retrieval-augmented generation. A retriever searches an index (your policies, your contracts, a case library) and stuffs chunks through assembly into the window. The model then does the same job it always does: predict the next token, now conditioned on those chunks plus the prompt.

Retrieval puts actual text in working memory. That is the progress. If the index is stale, the window is stale. If the chunker split a hold notice in the wrong place, the model conditions on a fragment. If the retriever missed the amendment, the model never saw the amendment.

The model can still emit a fluent sentence the stuffed chunks do not support. The chunks are context. A prompt that asks for authority will still get citation-shaped tokens if that is the cheap continuation.

That is the mechanism. What a serious product does with it lives in the harness, and that layer has moved.

In English. Sometimes extra text is stuffed into the box. That extra text is just more to guess from. If you ask for a case, the cheap next bits still look like a case.

The harness

The model only writes the next bit. Everything around it is software someone else wrote.

That software is the harness. It holds the system instructions. It holds the list of tools and the shape each call has to take. It reads the output. When a string looks like a call, the harness runs code. The result comes back through assembly as more tokens in the window. Then the model predicts again. A stop rule ends the loop.

There are two gates. One is whether a tool may run. The other is whether the draft may leave. Those are different questions. The permission set and the human gate are the firm’s or the client’s, not the model’s.

Chat products hide most of that. Agent products turn more of it on. Same sampler. Different wrapper.

The model does not reach for a tool. It predicts a string that looks like a tool call. The harness runs it, or it refuses.

An agent product is a harness. The tools, the unattended runs, the logs, and where the results go: those answers are the product. The brand on the sampler is not.

In English. The model only writes the next bit. Other software decides whether that bit becomes a search, a send, or a stop. That other software is the harness.

Tools

A tool is a named thing the harness will actually do if those tokens come out. Search the web. Open a file. Query a docket. Run code. Draft and send. File.

Search is not the model looking something up. Search is software that hits an index. The hits go back through assembly into the window. The model keeps predicting.

A tool call does not make the output a holding. The retriever can return a real opinion. The next tokens can still misstate it. If “file” is on and no person sits on the gate, the harness can file the misstatement.

Send is the same problem with less time to catch it. Once the harness runs send, the email is gone. The sampler only emitted a shape.

Unsupervised send or file is an act. It is your act, or your company’s, because you wired the permission. The sampler predicted a token sequence that the wrapper treated as an instruction. That is enough for the ethics rules, for privilege waiver, and for the record.

A filing tool without a human stop can file. A send tool that can reach the other side can send. A vendor that trains on prompts can keep what you pasted. Those are harness choices. They are also the ethics, privilege, and record questions.

Which tools are on. Who can approve them. What from the result enters the window. What you keep of that loop.

In English. A tool is a thing the harness will actually do if those bits come out. Search, send, and file are acts. The model only guessed a shape. You wired the permission.

Evolution

The sampler’s job hasn’t changed. The wrapper has.

Retrieval that actually grounds is one change. A serious index, not a web scrape. Chunks you can open. The answer is supposed to come from those chunks.

Citation grounding is another. The cite is supposed to point at a retrieved passage, not at a shape the model likes. Some products refuse to emit a cite they can’t attach to a chunk. That’s a harness rule. The model still writes the sentence.

Verification loops are a third. Draft. Check the cites against the retrieved text. Flag the ones that don’t match. Run it again, or stop. A person still has to open the case. The loop makes the cheap fake harder.

Legal-specific systems are the rest of that stack. A case library, a matter file, a playbook, permissions by seat, and a log. The Westlaw-shaped products and the newer legal AI shops are this: a harness with an index and a compliance story, sitting on the same kind of sampler.

Enterprise systems add audit logs, retention, tenant isolation, and contracts that say the vendor will not train on your prompts. Those aren’t decorations. In discovery they are the record. In privilege they are the facts about who saw what.

This is progress. A 2023 paste into a consumer chat box is a different product than a 2026 matter workspace with retrieval, grounding, a verification pass, and a log. Mata is still the cartoon of the first. It is a worse description of the second.

The need for a person who opens the authority is smaller. It isn’t gone. Grounding can attach a cite to the wrong passage. A verification loop can pass a real case that doesn’t stand for the proposition. A log can show you exactly which wrong sentence shipped. The sampler still writes the next token. The harness still decides whether that token becomes a send or a filing.

Triage. First drafts. Pattern spotting over a pile you will still read. That is the upside. It is real, and it is thin on purpose. The useful work still goes through a person who can open the source.

In English. The guesser is the same. The wrapper got better. A cite can now be tied to a real chunk. A person still has to open the source.

Why this is the legal problem

The failures people actually get are still sampler-as-library failures. They look different depending on the wrapper.

An associate pastes a draft into a consumer box and asks it to add authority. A GC forwards a line that says the tool confirmed a case. A product team drops a privileged memo into a consumer window because the pane felt private. An ops lead turns on send so a queue clears overnight. Those are the generic-tool cases.

Mature enterprise systems look different. They have a tenant, a log, a retention rule, and often a retrieval index. The mechanism is the same. The record is better. The need to open the authority is not gone.

Privilege. Tokens in the window are tokens the vendor, the log, or a later reviewer may see. Assembly is what decided they got in. “We asked the tool” is a disclosure fact. Whether the communication stays privileged depends on who ran it, for what purpose, and what actually went in. Counsel-directed work on a logged enterprise tenant is a different fact pattern than a Saturday paste into a consumer product. Decide the channel before the first sensitive prompt, not after the subpoena.

Discovery. The useful evidence is the context, the tool calls, the harness permissions, and the output that actually shipped. “The AI said” will not survive a careful request. Logs will. So will the fact that two runs differed. A mature system already keeps that trail. If you are going to use a thinner product on live matters, decide now what you keep and what a hold would freeze.

Fake cites. Citation-shaped tokens are a cheap continuation. Mata is the cartoon version. The quieter version, including on grounded systems, is a real-looking quotation that is slightly wrong, or a real case standing for a proposition it does not stand for. Competence still requires a person to open the authority. Rule 11 still requires gatekeeping. Castel left the tool alone. He sanctioned the filing.

Unsupervised send and file. The leave gate is the question. The harness turns a predicted token into an act when the tool is on and that gate is open. An email that goes out, a form that gets filed, a message that reaches an employee: those already happened. Treat them as events. Put a human stop on anything that leaves the building or enters a docket.

Competence. The duty did not move. You do not need to become an engineer. You do need to know that the machine predicts bits, that assembly decides what this call can see, that fluency is the job it was trained to do, and that a tool is code the harness will run if those bits come out that way.

A next token is not a holding. Do not file it as one.