Ask Your Data in Plain English: How We Build an AI Reporting Assistant

By the LaunchLen Web Solutions Team · Last updated: November 16, 2026
When a chatbot answers from your own documents, it doesn’t read the whole library each time. It searches for a few relevant passages and uses those to write its answer. How your documents are split into those passages, called chunks, has a big effect on whether the answer is right.
This post explains the main RAG chunking strategies, what each one is good at and how to test which one works for your content.
Retrieval-augmented generation, or RAG, combines a language model with a search step over your own information. The approach was described in the research paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, and it’s now a common way to ground chatbot answers in real documents.
Chunking is the step before search. Each document is split into smaller pieces, and each piece is indexed, often in a vector database, so it can be found by meaning. When someone asks a question, the system retrieves the most relevant chunks and passes them to the model.
Chunks matter for two reasons. Models can only read so much at once, as our post on the context window explains. And each chunk needs to make sense on its own, because the model only sees the pieces that were retrieved.
Microsoft’s documentation on chunking large documents for vector search is one vendor’s written example of these techniques. We cite it as a reference, not as a description of any particular stack.
The simplest approach splits text into pieces of roughly equal length. It’s quick to set up and works reasonably well for plain, flowing text.
The weakness is that a fixed split can cut a sentence, a list or an idea in half. Overlap helps: each chunk repeats a little of the text before it, so context carries across the boundary. Too much overlap, though, fills the index with near-duplicates.
There’s no single right size or overlap. Sizes vary by content, so test with your own documents and questions.
Most business documents already have structure: headings, numbered steps, lists and tables. Structure-aware chunking uses that structure to decide where to split.
This is one reason clean, well-organized source content matters so much. Our post on preparing content for a chatbot covers how to get documents ready.
Semantic chunking looks at meaning rather than length or layout. It groups sentences that belong together and starts a new chunk when the topic shifts.
It can produce more coherent chunks for long, loosely structured documents, such as meeting notes or long reports without clear headings. But it takes more processing to set up and update, and for documents that already have good headings, structure-aware chunking may do just as well. It’s worth testing against the simpler options before committing.
A chunk on its own can lose important context. Metadata is extra information attached to each chunk, such as:
Metadata can also help filter results, and it pairs well with hybrid search, which combines keyword and meaning-based search.
Where a passage ends up in the model’s input matters too. The research paper Lost in the Middle found that language models can make less use of information placed in the middle of long contexts. That’s another reason to retrieve a few focused chunks rather than many loosely related ones.
The only reliable way to choose a chunking strategy is to test it on your own content:
Our post on RAG evaluation explains how to set up this kind of testing. Good chunking helps ground answers, which the Government of Canada’s guide on the use of generative AI describes as having models build responses from only the information you provide and control. Grounding reduces errors but doesn’t remove them, so it works alongside the other methods in our post on how to reduce AI hallucinations.
No. The right size depends on your documents and the questions people ask. Test a few options with real questions and compare what gets retrieved.
Often, yes. Tables can lose meaning if split in the middle, and PDFs may need their text and layout extracted carefully before chunking.
Test with real questions and check which chunks were retrieved. If the right passage is missing or cut in half, chunking is a likely cause.
Planning a chatbot or assistant that answers from your own documents? We’re LaunchLen Web Solutions Inc.
Our AI Solutions page covers how we build AI chatbots and assistants that answer from your content. AI chatbots and assistants we build include full technical support and ongoing SaaS updates.
Serving local businesses across Canada and the US, delivered remotely.
Book a free consultation to get started.
Ready to take the first step towards unlocking opportunities, realizing goals, and embracing innovation? We're here and eager to connect.