RAG Chatbots Explained: How to Answer Questions from Your Company Documents

A general AI assistant may write fluently without knowing your latest policy, product details or internal procedures. A retrieval-augmented generation system—usually shortened to RAG—addresses this gap by finding relevant material from an approved knowledge base before producing an answer.

RAG does not make every response correct. Its value is that the answer can be grounded in documents your organisation controls, with source references that a user can inspect.

How RAG works in plain language

A typical RAG chatbot follows five stages:

1. A user asks a question.
2. The system converts the question into a representation suitable for search.
3. It retrieves relevant passages from an indexed document collection.
4. The language model receives the question and selected passages.
5. It prepares an answer, ideally with links or citations to the source.

Changing a document does not retrain the underlying model. Instead, the application fetches relevant context when a question arrives.

RAG versus fine-tuning

RAG and fine-tuning solve different problems.

Use RAG when answers depend on changing facts, policies, manuals, catalogues or records that must be traceable to a source. Update the knowledge base and re-index it when documents change.

Fine-tuning changes model behaviour by training on examples. It can help with style, output format or repeated task patterns, but it is not the simplest way to keep factual knowledge current.

Many business knowledge assistants should begin with retrieval because documents can be updated, access controlled and cited.

Choose a bounded knowledge problem

Good first use cases include:

• staff questions about an approved policy set;
• product or course information from a maintained catalogue;
• technical support from current manuals;
• finding clauses in standard operating procedures; and
• answering questions from a controlled project library.

Do not begin by indexing every file the organisation owns. Mixed permissions, duplicate versions and obsolete documents will create confusing and potentially unsafe answers.

Prepare documents before adding AI

Knowledge quality determines response quality. Assign an owner to each collection and remove drafts, expired versions and duplicates.

Record useful metadata such as title, owner, effective date, department, confidentiality level and version. Metadata helps the retrieval layer filter material before similarity search.

Documents with scanned images, complex tables or poor formatting may need special extraction and validation. Confirm that important headings, values and relationships survive the ingestion process.

Chunking and retrieval matter

Large documents are usually divided into smaller passages called chunks. Chunks that are too small lose context; chunks that are too large may contain several unrelated topics.

Test retrieval directly. For a known question, inspect which passages the system returns before judging the final answer. If the correct passage never reaches the model, prompt changes alone will not solve the problem.

Useful retrieval controls include:

• metadata filtering;
• a minimum relevance threshold;
• limits on the number of passages;
• reranking results; and
• refusing to answer when evidence is weak.

Make sources visible

A trustworthy assistant should show where important information came from. The citation must lead to the actual document or passage, not merely list a filename the user cannot access.

Tell users when the knowledge base was last updated and whom to contact if the source appears wrong. For consequential decisions, the document—not the generated summary—remains authoritative.

Protect document permissions

Retrieval must respect the user's access rights. A chatbot should not reveal a confidential passage simply because it exists in the same vector database.

Separate knowledge collections when necessary, filter by identity and role, and test whether a user can infer restricted information through indirect questions. Keep credentials protected and minimise logs containing sensitive prompts or answers.

Evaluate with real questions

Create a test set containing:

• straightforward questions with one clear source;
• questions requiring two passages;
• outdated terminology;
• ambiguous questions;
• questions outside the knowledge base;
• conflicting documents;
• requests for restricted information; and
• malicious instructions inside uploaded documents.

Score retrieval accuracy, answer correctness, citation quality, appropriate refusal and response time. Re-test after changing documents, embeddings, prompts or models.

Common RAG failure modes

The wrong document is retrieved

Improve metadata, chunking or ranking. Do not expect the model to recover facts it never received.

The answer overstates the source

Instruct the model to stay within evidence and quote or cite the supporting passage. Test whether uncertainty is expressed clearly.

Old and new policies conflict

Remove obsolete versions or apply effective-date rules. The knowledge library's ownership and version controls directly shape the assistant's behaviour.

The chatbot answers outside scope

Add a refusal policy, relevance threshold and escalation path. A helpful “I cannot verify that from the approved documents” is better than a confident invention.

Build a grounded chatbot workflow

The Build a RAG Chatbot with n8n course from Tertiary Courses Malaysia teaches participants to prepare documents, create retrieval workflows and connect grounded AI answers to a chatbot using n8n. The emphasis is on practical construction, testing and controlled use of knowledge.

The course is HRD Corp claimable. Employer claims remain subject to current HRD Corp requirements, available levy, supporting documents and approval. Visit the course page for current dates and delivery options.

Frequently asked questions

Draft research references

• Tertiary Courses Malaysia — Build a RAG Chatbot with n8n: https://www.tertiarycourses.com.my/build-rag-chatbot-with-n8n.html
• n8n Docs — RAG in n8n: https://docs.n8n.io/advanced-ai/rag-in-n8n/
• Microsoft Azure — Retrieval-augmented generation overview: https://learn.microsoft.com/en-us/azure/search/retrieval-augmented-generation-overview