Why Your AI Chatbot Forgets the Customer: Context Engineering, Explained for Business Owners

Short answer: An AI chatbot forgets your customer because of how its context is managed, not usually because the AI model is weak. Context engineering is the work of deciding exactly what information the model sees at each step of a conversation, and the agent harness is the software around the model that does that work. Two businesses can use the same model and get one chatbot that remembers the customer’s order and another that asks for their name three times. The difference is the build.

What is context engineering?

Context engineering is the practice of choosing which information fills an AI model’s limited working memory, called its context window, at every step of a task. Anthropic, which makes the Claude models, describes it as the successor to prompt engineering and says context “must be treated as a finite resource with diminishing marginal returns”, with the goal of finding “the smallest set of high-signal tokens that maximize the likelihood of your desired outcome” (Anthropic).

In business terms: a good employee answering a customer does not reread every email the company ever sent. They pull up this customer’s file, glance at the last conversation and answer. Context engineering makes an AI agent work the same way.

Why does an AI chatbot forget what the customer said earlier?

Usually because the system around the model either dropped that information or buried it under too much else. Some chatbots only send the model the last few messages, so anything older simply disappears. Others send everything, including long product catalogues and old conversations, and the detail that mattered gets lost in the noise. Anthropic notes that as context grows, “the model’s ability to accurately recall information from that context decreases” (Anthropic). Both failures look the same to your customer: the bot forgot them.

What is an agent harness, and why does it matter more than the model?

A harness is everything wrapped around the AI model: the instructions it starts with, the tools it can use, how it looks things up, how it stores notes and how it hands off to a person. The model does the thinking; the harness decides what the model knows and can do at each moment. The term has moved from research into products: Microsoft’s Copilot Studio documentation now distinguishes agents by the harness that powers them, each with its own memory behaviour (Microsoft Learn). Coding tools such as Claude Code are harnesses too, which is why the same model can feel very different from one tool to another.

Anthropic’s own research on long-running agents found the typical failures were harness failures: the agent tried to do too much at once and ran out of context mid-task, later sessions declared victory early, and features were marked done without real testing. The fixes were a progress file, a feature list and a clear record of the state of work, because “the key insight here was finding a way for agents to quickly understand the state of work” (Anthropic).

Does a bigger context window fix the forgetting?

No. A bigger window lets you send more, but sending more makes the model less reliable at finding what matters. Chroma’s research tested 18 language models and found that models “do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows”, even on simple tasks (Chroma). Researchers call this context rot. When a vendor answers a memory complaint with “the new model has a bigger context window”, they have not fixed the problem; they have made room for more noise.

How do well-built agents remember the right things?

They fetch what they need at the moment they need it, and keep the rest out. Anthropic lists the main techniques as compaction (summarising a long conversation and continuing from the summary), structured note-taking outside the context window, sub-agents that handle focused tasks and return short summaries, and just-in-time retrieval that loads data only when it is needed (Anthropic).

Technique	What it looks like in a customer chat
Just-in-time retrieval	Looks up this customer’s past orders by phone number when they message
Structured notes	Saves “wants a demo, budget confirmed, prefers evenings” to the CRM
Compaction	Summarises a long chat so the next reply still sees the key facts
Sub-agents	A separate step checks stock or slots and returns only the answer

What questions should you ask an AI vendor about context?

Ask how the agent decides what it sees, not which model it uses. A vendor who can answer these clearly has built a harness; a vendor who only names a model has probably wrapped a prompt around it.

• When a returning customer messages, what does the agent know about them, and where does it get it?
• What happens in a very long conversation: does the agent summarise, or start forgetting?
• What does the agent remember between conversations, and how is one customer’s data kept out of another’s?
• When a human takes over, do they see a summary or have to read the whole chat?

How do we handle context in the agents we build?

The same way a good employee does: look up, note down, hand over. We design our agents to look up the customer by their phone number or email when the conversation starts, keep their instructions short and specific to the job, and write a brief summary of every conversation to the CRM so the next conversation, human or AI, starts from what matters. Our homepage voice agent works this way: after each call it saves a short summary note, so a follow-up never starts from zero. Memory stays per customer, never shared, for the reasons in our post on agent permissions.

If your current chatbot keeps forgetting customers, the model is rarely the fix. See how we build on our chat agents page, or book a free 15-minute call and we will show you where the context is being lost.