The Pronoun That Made Our Chatbot Lie

Last updated: August 2026

We had it working. A student of mine, someone who started with me on Python years ago and is now building retrieval systems, had spent the session wiring a chatbot to a document store. We asked it who one of the people in the demo dataset was. It came straight back with the right answer, pulled from the actual records: her role, her location, the date she started. We checked it against the raw data. Correct.

So he typed a follow-up. Four words, the kind anyone would type. What is her performance history.

It answered immediately and confidently. It gave us dates. It gave us a review cycle. It gave us a year that appears nowhere in the dataset, for a person whose record it had read correctly thirty seconds earlier.

Nothing had broken. That is the part worth sitting with. The retrieval was fine, the model was fine, the data was fine. If you have never built one of these, a retrieval chatbot is just a search step bolted to a language model, and both halves were doing their jobs correctly. The only thing missing was that the chatbot had no memory of the previous message, so the word "her" pointed at nothing at all. And a model handed a pronoun with no subject does not stop and ask. It fills the gap.

Watching it happen is the whole lesson

I could have told him about conversation history at the start of the session. I have explained it a hundred times and it lands as an abstraction every time. Instead we built the thing without it, asked a normal question, and watched it fabricate a year in front of us.

That is now how I teach almost everything. Trigger the failure, then fix it. The fix took two minutes: pass the previous turns along with each new question so the model can resolve what "her" refers to. But the two minutes only mattered because of the thirty seconds before them, where he saw a confident, well-formatted, completely invented answer to a question he had asked in good faith.

History is not a nice-to-have in a chatbot. It is load-bearing.

The reason this catches people is that large language models are stateless. Each request arrives with no knowledge of the one before it. What feels like a conversation is really the entire transcript being re-sent from the top, every single time, so the model can re-read it and continue. Anthropic's context window documentation says the same thing in more formal language. Nothing is remembered. Everything is resent.

Once you have that sentence properly in your head, a lot of otherwise baffling behaviour becomes obvious. Why a long chat gets worse the longer it goes. Why starting a fresh session fixes problems that no amount of rephrasing would. Why the assistant suddenly forgets a constraint you set forty messages ago.

The same mechanism, running the other way

A few weeks later the same student came to a session with a different problem, and it took me a while to notice it was the same problem wearing a different coat.

He had set two AI agents to debate each other on a task, left them running, and burned roughly a thousand dollars in a single day. Not on the most expensive model available. He wanted to know where the money had gone.

We worked it out from first principles rather than guessing at symptoms, and the answer was the pronoun problem in reverse. Every turn re-sends the whole transcript. With two agents debating, both transcripts grow, and they grow on both sides at once. The first message is short. By the twentieth exchange each message is carrying the entire history of the argument with it, and every one of those histories is being re-read and paid for. Models also do not stop on their own. They will happily keep a conversation going long after it has stopped being useful.

So: too little history and it lies to you. Too much history and it bankrupts you. One mechanism, two failure modes, sitting at opposite ends of the same dial.

There is a second trap layered under that one. Subscription products carry usage limits and automatic compaction, because the provider does not want you consuming far more than you paid for. Those limits feel annoying right up until you move to raw API access, where none of them exist. My student was on the API. Nothing was protecting him from himself, and nothing was designed to. This catches people at work too, because a company that hands you API access instead of a subscription has handed you the version with no brakes. It is also most of the reason the providers repriced their plans in the first place: the cheap tiers were never sized for people running agents in loops, which is how the twenty dollar plan stopped being the professional default.

What I tell every student now

Conversation history is a resource you are spending, not a feature you are given. Treat it that way and both failures become manageable.

On the too-little side, that means never assuming the tool knows what you are referring to. If you have started a fresh session, say the thing again. If you are building something that takes follow-up questions, pass the history explicitly and test it with a pronoun before you ship it, because a pronoun is the cheapest way to find out whether your memory is actually wired up.

On the too-much side, it means watching how full the context is and stopping deliberately. My own practice, which I now teach to everyone regardless of level, is to stop at around seventy percent, write a handoff file describing where things stand, and open a fresh session pointed at that file. You lose the transcript and keep the substance, which is the right trade. If you are running anything autonomous, put a stopper on it: drop the parts of the conversation that are settled, right-size the model to the task, and set a limit before you set it running rather than after the invoice arrives.

The deeper habit is the one I keep coming back to with every student, from the ones opening a terminal for the first time to the ones building retrieval pipelines. When the tool behaves strangely, the question is almost never what is wrong with the model. It is what does this thing actually know at the moment I asked. That question has a real answer, you can usually work it out in about a minute, and it explains far more of what goes wrong than any amount of prompt tweaking. It is also the habit that carries you when you move from calling models to building with them, and it is most of why applied machine learning does not need a PhD: the job is working out what the system actually knows and whether its output is honest, not deriving the mathematics underneath it. If you want to build that instinct with someone watching your screen, a free Discovery Call is where most of my students started.