AI Chatbot Buying Guide: 12 Questions to Ask

Chatbot demos are designed to show fluency. Buying decisions should be based on operations. A system that answers twenty sample questions beautifully can still fail on the twenty-first because the answer was not in the indexed content, the customer needed a human, or the conversation drifted into a request the bot was never authorised to handle. The questions below are the ones that expose those gaps before you sign a contract, and they are worth asking regardless of which platform presented the demo.

Start with knowledge sources. Which systems will the assistant read from, and how does content get in? A help centre, shared drive, policy wiki, and product database have very different ingestion needs. Ask whether the vendor supports your actual formats, including scanned PDFs and spreadsheets, and how often the index refreshes after a content change. A chatbot trained on last quarter's documentation is a liability, not an asset. Also ask who inside your team will own content quality once the project ends.

Then ask about grounding and citations. Does the assistant answer only from retrieved content, or may it use general model knowledge? Can it show the source of an answer, and what does it do when it finds nothing relevant? The acceptable answer is a clear not-found response plus an offer to escalate. Watch for systems that always produce an answer, because confident improvisation is the most expensive failure mode in a customer-facing assistant.

Escalation and handoff deserve a live test. Ask exactly how a conversation reaches a human, what context travels with it, and what happens outside office hours. The handoff should include the transcript, customer details, the question that triggered escalation, and any answers already given. If the customer must repeat everything to a person, the assistant has added friction rather than removing it. Confirm whether the human can take over mid-conversation without the customer restarting in a different channel.

Guardrails are a security topic, not a tone topic. Ask how prompt injection is handled, whether the assistant can be tricked into revealing its instructions or other customers' data, what personally identifiable information is stored, and how long conversations are retained. Confirm role-based access if the assistant serves staff as well as customers. A vendor that treats these as edge cases has not operated an assistant in a hostile environment, and hostile environments are the normal condition on the public internet.

Model choice and portability matter over a two-year horizon. Ask which models power the assistant, whether you can bring your own API key, and how switching providers works. Models are updated and retired on vendor schedules you do not control. A platform that abstracts the model is more resilient than one hard-wired to a single provider, provided the abstraction does not prevent you from using capabilities you need. Put the answer in writing, because migration costs are easy to underestimate later.

Analytics close the loop. Which metrics does the platform report: containment rate, escalation rate, resolution time, unanswered questions, customer satisfaction? Can you see the actual conversations, and can you export them? The list of questions the assistant could not answer is the most valuable output of the first month. Without it, the team cannot tell whether the assistant is improving or slowly drifting away from what customers actually ask. Ask how often the report is produced, who receives it, and whether the platform flags topics that are rising in frequency. A weekly review of that list, rather than a monthly glance at a dashboard, is what keeps the assistant aligned with the business.

Pricing models vary enough to change the business case. Common structures include a platform fee plus usage, per-seat pricing for human agents, per-resolution pricing, and token-based billing passed through from a model provider. Estimate cost at your realistic monthly conversation volume, including spikes, and ask what happens when you exceed the plan. Also ask about implementation fees, content migration, and whether support during onboarding is included or billed hourly. The cheapest headline price is rarely the cheapest total.

Ownership and exit terms deserve a careful read. Who owns the prompts, the indexed content, the conversation history, and the configuration? Can you export all of it in a usable format, and what happens to your data if you cancel? A vendor that cannot answer those questions plainly is asking you to accept lock-in you did not choose. The same applies to the underlying model provider's data policies; make sure you understand what is retained, for how long, and whether it is used for training.

Finally, ask what done looks like and how the first ninety days will run. A credible plan includes a pilot on one topic area, a defined escalation path, a weekly review of unanswered questions, and success measures agreed before launch. Buy the operations, not the demo. If the answers to these questions are specific and written down, you are dealing with a partner. If they are reassuring but vague, keep looking, because the cost of the wrong choice is measured in customer trust, not subscription fees.