Large language models are impressive generalists, but they know nothing about your internal policies, contracts, project archives or service procedures. Retrieval-augmented generation (RAG) closes that gap: instead of retraining a model, you retrieve the right passages from your own content and give them to the model at question time.
Why RAG instead of fine-tuning?
Fine-tuning changes how a model writes; it is a poor way to teach it facts that change every week. RAG keeps knowledge in your systems, where it can be updated, permissioned and audited.
- Fresh: update a document and the next answer reflects it.
- Traceable: every answer can cite the passages it used.
- Governed: users only retrieve content they are allowed to see.
- Cheaper: no training runs, and you can swap models later.
The RAG pipeline, step by step
- Ingest: collect documents from your CMS, file shares, wikis and ticketing systems, and extract clean text.
- Chunk: split text into passages small enough to be precise but large enough to keep context, typically a few hundred words with some overlap.
- Embed: convert each chunk into a vector with an embedding model, so similar meanings sit close together.
- Index: store vectors plus metadata (source, date, language, permissions) in a vector-capable search engine.
- Retrieve: embed the user's question, find the closest chunks, and filter by permissions.
- Generate: send the question and retrieved passages to the model with instructions to answer only from them and cite sources.
In library-agnostic pseudocode, the core loop is short:
chunks = split(document.text, size=800, overlap=120)
vectors = embed(chunks)
index.upsert(vectors, metadata={doc_id, title, updated, allowed_groups})
# at question time
hits = index.search(embed(question), top_k=8,
filter={allowed_groups: user.groups})
answer = llm(system="Answer only from the sources. Cite them.",
context=hits, question=question)
Five decisions that make or break quality
1. Chunking that respects structure
Split on headings and paragraphs, not arbitrary character counts. Keep the document title and section heading with each chunk so retrieved passages make sense on their own.
2. Hybrid search
Vectors are great at meaning but weak at exact terms such as policy numbers, product codes or names. Combining keyword (BM25) and vector search, then re-ranking the merged results, consistently beats either alone.
3. Permissions at retrieval time
Never retrieve first and filter later in the prompt. Store access groups as metadata and filter in the search query itself, so restricted text never reaches the model.
4. Grounded prompting
Instruct the model to say "I don't know" when the sources do not contain the answer, and to cite each claim. This single instruction removes most hallucinations users notice.
5. Freshness
Re-index on change events (publish, update, delete) rather than nightly batches where possible, and expire content that has been unpublished.
How to measure whether it works
Build a test set of 50 to 100 real questions with known good answers before launch, and track:
| Metric | Question it answers |
|---|---|
| Retrieval recall | Did the right passage appear in the top results? |
| Faithfulness | Is every statement supported by the retrieved sources? |
| Answer relevance | Does the answer actually address the question? |
| Citation accuracy | Do the cited sources say what the answer claims? |
Re-run the set whenever you change chunking, models or prompts. Small changes can have large effects.
Where to start
Pick one well-bounded knowledge domain with clear owners, such as HR policies or a service catalogue, and a group of users who ask the same questions repeatedly. Prove accuracy there, then expand. If your content lives in Drupal, the Drupal AI module can provide the search and vector index layer. Need help designing it? Talk to our team.