Retrieval quality is the product

For any chatbot answering questions about your business, accuracy is dominated by what you put in the prompt, not by which model you use.

That means the engineering effort belongs in retrieval:

Chunking that respects document structure rather than splitting every 500 characters mid-sentence
Hybrid search: vector similarity plus keyword matching, because exact terms like product names and error codes are where pure vector search fails
Reranking the top candidates before they reach the model
Metadata filtering so a user only retrieves what they're allowed to see

Teams reach for a bigger model when the real problem is that retrieval returned the wrong three paragraphs.

It has to be able to say "I don't know"

A chatbot that answers everything confidently is worse than one that escalates, because users stop trusting all of its answers after the first confident wrong one.

Build in:

Grounding: answers cite the source documents they came from
A confidence threshold below which it declines and offers escalation
A clean handoff to a human with the conversation attached

The refusal path is a feature, not a failure mode, and it's the one most demos skip.

Scope for an MVP

One channel, web widget, Slack, or WhatsApp, not all three
One corpus, your docs or your help centre, not every system you own
Retrieval with citations
Escalation to a human
Conversation logging with feedback capture

Actions come later, carefully

A bot that can do things, issue refunds, change bookings, update records, is far more valuable and far riskier. When you add actions, each one needs a confirmation step, an idempotency key so retries don't double-execute, and a hard boundary on what it's permitted to touch.

Read-only first is the right sequencing for almost everyone.

You cannot skip evaluation

Collect fifty to a hundred real questions before building, with known-good answers. Without that set, prompt changes are guesswork and you have no way to tell whether an improvement regressed something else.

This is the step teams cut for time and regret after launch, when accuracy turns out to be 60% on real questions rather than the 90% the demo suggested.

What it costs to run (list prices read October 17, 2026)

A chatbot's bill has three parts: model tokens, the retrieval step, and storage. The model is often the smallest of them.

Item	List price
Small model tokens (OpenAI gpt-6-luna)	$0.10 input and $0.50 output per million tokens; $0.01 per million for cached input
Mid model tokens (OpenAI gpt-6.1-sol)	$2 input and $10 output per million tokens; $0.10 per million for cached input
File search tool call (OpenAI)	$2.50 per 1,000 calls
File search storage (OpenAI)	$0.10 per GB per day, first 1 GB free
Web search tool call (OpenAI)	$10 per 1,000 calls, plus search content tokens at model rates
Cache hit (Anthropic)	10 percent of the standard input price on most models

A worked example with stated assumptions: 2,000 conversations a month, 6 turns each, and every turn sends 1,200 input tokens (retrieved passages plus history) and returns 250 output tokens. That is 12,000 turns, 14.4 million input tokens and 3 million output tokens. On the small model the tokens cost $2.94 a month; on the mid model they cost $58.80. If each turn also makes one file search call, the retrieval calls add $30.00 a month, about ten times the small model's token cost. Count the retrieval step before you pick a model. Sources: OpenAI pricing and Anthropic pricing.

What better retrieval buys

Anthropic's September 19, 2024 write-up on contextual retrieval measured the top-20-chunk retrieval failure rate on its own test sets: 5.7 percent at baseline, 3.7 percent with contextual embeddings, 2.9 percent when contextual BM25 was added, and 1.9 percent after reranking, a 67 percent reduction. Those are Anthropic's results on its datasets, so measure your own corpus before you promise a number to anyone. The techniques line up with the list above: hybrid search of vectors and keywords, reranking, and added context for each chunk. Source: Introducing Contextual Retrieval.

Disclosure and security before launch

If people in the EU will use the chatbot, Article 50 of the EU AI Act requires telling them they are dealing with an AI system unless that is obvious from the context, and it has applied since 2 August 2026; EU AI Act for AI MVP founders lists the dates and duties. Prompt injection through retrieved content, one customer's data reaching another and runaway spending are covered by OWASP LLM Top 10 pre-launch tests for AI MVPs. Where users' messages go is set by the model vendor's terms, which OpenAI, Anthropic and Gemini API data terms compares, and a mobile chatbot also has to meet App Store and Google Play rules for AI apps.

FAQ

How do you stop an AI chatbot from making things up?

Ground answers in retrieved documents and require citations, set a confidence threshold below which it declines rather than guesses, and give it a clean escalation path to a human. You reduce hallucination rather than eliminate it: any vendor claiming zero is overselling.

Which model should an AI chatbot use?

Less important than teams assume. For a bot answering questions about your business, retrieval quality dominates accuracy: chunking, hybrid keyword-plus-vector search, and reranking move results more than switching models. Benchmark two or three against your own evaluation set and keep the model swappable.

Should the chatbot be able to take actions?

Not in the first version. Read-only retrieval first, then add actions individually: each with a confirmation step, an idempotency key so retries don't double-execute, and a hard boundary on what it can touch. Actions are where a chatbot becomes valuable and where it becomes risky.