How to Evaluate a RAG System Before It Talks to Customers

By the LaunchLen Web Solutions Team · Kitchener, Ontario · Last updated: October 26, 2026

A chatbot that answers from your own content can save your team time, but only if its answers are right. RAG evaluation is how you check that before customers see it: testing whether the system finds the right information and whether it turns that information into a correct, sourced and helpful reply.

This guide walks through a practical approach: why RAG needs its own testing, how to build a test set, how to check retrieval and answers separately, and how to keep evaluating after launch. If you’re new to the idea, start with our explainer on what RAG is.

Why RAG needs its own testing

Retrieval-augmented generation (RAG) works in two stages. First, the system searches your knowledge base for passages related to the question. Then an AI model writes an answer using those passages.

Either stage can go wrong:

• Retrieval can miss. The search pulls an outdated policy, the wrong product page or nothing useful at all.
• Generation can drift. The model has the right passage but adds details that aren’t in it, misreads it or answers a different question.

OWASP’s guidance on misinformation lists RAG as a way to improve reliability by retrieving verified information, and says it helps mitigate the risk of hallucinations. Helping isn’t the same as solving, so a RAG chatbot still needs testing. Because the two stages fail in different ways, evaluate them separately as well as together.

Build a test set from real customer questions

A test set is a list of questions with the answer you expect and the source that answer should come from. It’s one of the most useful things you can build before launch, because every later check depends on it.

Good places to find questions:

• Emails, contact-form messages and chat logs (with personal details removed).
• Questions your front-line staff hear on the phone.
• Your existing FAQ page and help articles.

For each question, record:

• The question, written the way customers actually ask it, including vague or misspelled versions.
• The expected answer, checked by someone who knows the business.
• The expected source, meaning the page or document that should support it.

Include harder cases too:

• Out-of-scope questions the chatbot should decline or hand to a person.
• Questions with no answer in your content, where the right response is “I don’t know” plus a handoff.
• Tricky or adversarial inputs, such as attempts to make the bot ignore its instructions.
• Restricted topics, if some documents should only be visible to certain users.

Check retrieval: did it find the right passage?

Retrieval accuracy is about whether the search step brings back the passages needed to answer. For each test question, compare what was retrieved with the expected source.

Questions to ask:

• Was the expected source retrieved at all? Missing it means the model can’t answer from it.
• Was it near the top? Relevant passages buried under irrelevant ones can confuse the answer.
• Was anything outdated or off-topic pulled in? Old versions of policies can be a culprit.
• Did access rules hold? OWASP warns that inadequate or misaligned access controls can lead the model to retrieve and disclose sensitive information. Test that restricted content stays restricted.

Evaluation tools describe these checks with names such as context recall (did it retrieve everything needed?) and context precision (were the relevant passages ranked above the irrelevant ones?). You don’t need special tools to start; a spreadsheet and careful review work.

When retrieval fails, the fix can be in the content or the search setup: clearer source documents, removing duplicates and old versions, better splitting of long documents, or adjusting how results are ranked.

Check answers: correct, sourced and on-tone?

Next, review the answers themselves. A simple rubric keeps reviewers consistent:

• Correct: does it match the expected answer?
• Faithful: is every claim supported by the retrieved passages? One common definition treats a response as faithful when all its claims can be supported by the retrieved context.
• Sourced: if the bot shows citations or links, do they point to the right page?
• Complete: does it answer the whole question, or only part?
• On-tone: is it clear, polite and in your brand voice?
• Safe: does it decline out-of-scope requests and hand off when it should?

Rate each answer as pass or needs work, and note why. Patterns in the notes tell you whether to fix the content, the instructions given to the model or the retrieval step. Our guide on how to reduce AI hallucinations covers fixes in more depth.

Some teams also use an AI model to grade answers automatically. That can speed up repeat testing, but the grader can be wrong too, so keep people reviewing a sample of its judgements.

Ongoing RAG evaluation after launch

Testing doesn’t stop at launch. Content changes, customers ask new things and models get updated.

• Re-run the test set after any change to content, instructions, settings or the underlying model.
• Review real conversations on a routine you set, looking for wrong answers, missed handoffs and new question types.
• Add new cases to the test set whenever a real conversation exposes a gap.
• Track feedback, such as thumbs up or down, and handoffs to staff.
• Keep a log of what the bot retrieved and answered, so issues can be traced.

Here’s an illustrative example: a clinic updates its cancellation policy. Before the change goes live, the team updates the source page, re-runs the related test questions and confirms the bot quotes the new policy, not the old one.

When we help with a RAG project, we can agree the scope, deliverables and reporting with you, including how testing and review can work.

FAQ

What is RAG evaluation?

RAG evaluation is the process of testing a retrieval-augmented generation system. It checks whether the system retrieves the right information and whether its answers are correct, supported by that information and appropriate for customers.

How do you test a RAG chatbot before launch?

Build a test set of real customer questions with expected answers and sources, including out-of-scope and unanswerable questions. Check retrieval and answers separately, fix what fails and re-test before going live.

Do I need special tools for RAG evaluation?

Not to start. A spreadsheet and careful human review can cover a first round of testing. Evaluation tools can help automate repeat checks, but people should still review results.

Let’s work together on your project

Planning a chatbot that answers from your own content? We’re LaunchLen Web Solutions Inc.

Based in Kitchener, serving Waterloo Region and businesses across Ontario.

Our AI Solutions page explains how a RAG chatbot works, in plain English, and the safeguards a build can include. Book a free consultation to get started.