How to Build an AI Chatbot MVP

Retrieval quality is the product

For any chatbot answering questions about your business, accuracy is dominated by what you put in the prompt, not by which model you use.

That means the engineering effort belongs in retrieval:

Chunking that respects document structure rather than splitting every 500 characters mid-sentence
Hybrid search — vector similarity plus keyword matching, because exact terms like product names and error codes are where pure vector search fails
Reranking the top candidates before they reach the model
Metadata filtering so a user only retrieves what they're allowed to see

Teams reach for a bigger model when the real problem is that retrieval returned the wrong three paragraphs.

It has to be able to say "I don't know"

A chatbot that answers everything confidently is worse than one that escalates, because users stop trusting all of its answers after the first confident wrong one.

Build in:

Grounding — answers cite the source documents they came from
A confidence threshold below which it declines and offers escalation
A clean handoff to a human with the conversation attached

The refusal path is a feature, not a failure mode, and it's the one most demos skip.

Scope for an MVP

One channel — web widget, Slack, or WhatsApp, not all three
One corpus — your docs or your help centre, not every system you own
Retrieval with citations
Escalation to a human
Conversation logging with feedback capture

Actions come later, carefully

A bot that can do things — issue refunds, change bookings, update records — is far more valuable and far riskier. When you add actions, each one needs a confirmation step, an idempotency key so retries don't double-execute, and a hard boundary on what it's permitted to touch.

Read-only first is the right sequencing for almost everyone.

You cannot skip evaluation

Collect fifty to a hundred real questions before building, with known-good answers. Without that set, prompt changes are guesswork and you have no way to tell whether an improvement regressed something else.

This is the step teams cut for time and regret after launch, when accuracy turns out to be 60% on real questions rather than the 90% the demo suggested.

FAQ

How do you stop an AI chatbot from making things up?

Ground answers in retrieved documents and require citations, set a confidence threshold below which it declines rather than guesses, and give it a clean escalation path to a human. You reduce hallucination rather than eliminate it — any vendor claiming zero is overselling.

Which model should an AI chatbot use?

Less important than teams assume. For a bot answering questions about your business, retrieval quality dominates accuracy — chunking, hybrid keyword-plus-vector search, and reranking move results more than switching models. Benchmark two or three against your own evaluation set and keep the model swappable.

Should the chatbot be able to take actions?

Not in the first version. Read-only retrieval first, then add actions individually — each with a confirmation step, an idempotency key so retries don't double-execute, and a hard boundary on what it can touch. Actions are where a chatbot becomes valuable and where it becomes risky.