RAG: Building Context, Not Search

The real decision: RAG or a tool?

In the first version we sent every question down the same path: take the question, search the documents, give what you find to the model. “How many dollars were deposited today?” went down that path too, and retrieved the “deposit procedures” page. The answer was a nice paragraph with no number in it.

The third row is the most common type and the most instructive: RAG brings the rule, the tool brings the number, the code does the comparison. Asking the model “has this customer exceeded the limit” and expecting it to combine the rule and the number in its head opens the door to three mistakes at once.

Chunking: the document draws the line

Our first attempt split at 1,000 characters. The result: the start of a limit rule in one chunk, its exception in another. The model never saw the exception and explained the rule incompletely.

{
  "id": "policy-withdrawal-v8#daily-limit",
  "heading_path": "Withdrawal Policy > Limits > Daily limit",
  "text": "The daily withdrawal limit ... (whole section, 420 tokens)",
  "source": "policy/withdrawal.md",
  "version": 8,
  "updated": "2026-09-08",
  "access": ["support", "risk", "management"],
  "neighbour": ["policy-withdrawal-v8#exceptions"]
}

# the text is not embedded on its own:
#   to_embed = heading_path + "\n" + text
# separate fields in JSON; joined on the way to the vector.

• The heading path goes into the embedding. It sits in its own JSON field, but it is added to the front of the text before the vector is taken. That is the only way the model knows which section of which policy it is reading; a paragraph alone is a pile of sentences with no context.
• Target 300–800 tokens. Too short and the rule is broken up; too long and irrelevant text eats the window.
• Neighbour links. The “exceptions” section is a separate chunk but marked as a neighbour, so when the rule is retrieved its exception comes with it. That single setting ended most of our incomplete answers. Neighbours are not counted within the top 5; they are added on top, so the model receives between 5 and 8 chunks. In the recall@5 measurement, however, only the 5 that came from the ranking count — a chunk delivered through a neighbour link is not included, or the measure would inflate itself.
• Permission labels. Filtering happens during the search. Filtering after the answer is written means you already showed the data to the model.
• The version field. Not just for information: removing the old version when a new one arrives depends on it — see below.

Freshness: our 14 days

12 Jul   the nightly indexing job died quietly (after a dependency upgrade)
14 Jul   withdrawal policy v7 published  -> document current, index not
28 Jul   INCIDENT: the agent read a 14-day-old rule correctly
~2 weeks "the model is hallucinating" discussion
August   three-week round of fixes (router / neighbour chunks / webhook + alert)
 8 Sep   policy v8 shipped -> v7 chunks were still in the index; delete step added

The cause of the incident in the opening paragraph was simple: a nightly job updated the index, and that job had quietly died on 12 July — unrelated to the policy update on 14 July; it had stopped after a dependency upgrade. Nobody noticed, because the system kept answering. A broken RAG runs silently; an old answer does not look like an error message.

We made three changes:

1. The index updates when a document changes (a webhook); the nightly job stayed as a backup.
2. Every answer shows its source and date: “Source: Withdrawal Policy v8, updated 8 September 2026”. If the user sees an old date, they can push back.
3. A freshness alert: for each document we compute the gap between the last change at the source and the version in the index; if the maximum of those gaps goes over 24 hours, the alert fires. (Our first alert looked at “the oldest document in the index” and was useless: a document can legitimately be two years old without changing. What you measure is not age, it is lag.) Now the job cannot die quietly again.

Deleting the old version: the sibling of those 14 days

We did not notice the second half of freshness during the three-week round below; we only understood it three weeks after that round ended, when v8 of the policy shipped on 8 September. The chunk id carries the version: policy-withdrawal-v8#daily-limit. When v8 was published, new chunks were written — but did v7’s leave?

They did not. The result is worse than an old answer: the model sees two contradictory rules at once and decides for itself which one applies. And because both are “sourced”, the citation check does not catch it.

Writing to the index now has two steps: the new version’s chunks are written, then every older version with the same source field is deleted. The delete step is separate and idempotent; if it is interrupted, the next run finishes it. The same path handles a deleted source document — deleting a document means removing it from the index.

Measuring accuracy: two separate numbers

“Is our RAG working?” is not one question but two: did it retrieve the right chunk, and did it produce the right answer from that chunk? Measure them separately, because the fixes are different.

A word on why the unsourced claim row is measured before the filter: the code already strips unsourced sentences from the answer, so in the text the user sees, that rate is close to zero. But that does not mean the model stopped inventing — only that the invention no longer reaches the user. 14% and 2% show how often the model tries, and that is the number to watch; if it rises, retrieval has degraded. The filter is a safety net, not a measure.

The unnecessary refusals row is the bill for the unsourced claim row. One distinction matters here: recall@5 is 27/30, so in three questions the right chunk never reaches the top 5 — a refusal on those would be retrieval’s doing, not the threshold’s, and would in fact be correct behaviour. The 2 refusals we count do not overlap with those three: in both, the right chunk was in the top 5 and simply scored below the threshold. So these really are the threshold’s bill. We pay that price on purpose — better than a wrong answer — but you should not accept it without measuring it: a threshold is always a two-sided trade.

The answer accuracy row told us where the problem was: given the right chunk, answer accuracy was already 28/30. So our problem was not the model, it was retrieval. Changing the model would not have fixed this table; we only saw where to improve once we separated the two numbers.

A note on sample size: in a set of 30, one question is worth about 3.3 points, and the “I don’t know” set is only 10 questions. These numbers pointed us in the right direction, but we treat differences of 3–4 points as noise and I suggest you do the same. While the fixed set is small, do not claim improvements to the decimal point.

What raised recall@5?

• Reranking. Vector search brings 20 candidates, a small ranking model reorders them against the question, and the top 5 go to the model (5–8 chunks once neighbours are added). The largest single contribution: 21/30 → 25/30 (70% → 83%).
• Combining with keyword search (hybrid search). Vector search alone misses exact matches such as “IBAN”, “MT103” or “v8”. Combined with classic text search: 25/30 → 27/30 (83% → 90%).
• Putting the heading path into the embedding. Small and free: it tells apart which policy a word appeared in.

A problem specific to Turkish

The hybrid search point above matters more in Turkish than in English, because the keyword side misses easily due to suffixes: limit, limiti, limitinin, limitler are four different terms to a classic text search. (Turkish is agglutinative: words take a chain of suffixes rather than standing alone.)

• Stemming is mandatory. We turned on the search engine’s Turkish analyser so that queries and documents are reduced to the same stem. Before that, hybrid search added almost nothing — the keyword side simply was not finding anything.
• Embedding models are weaker in Turkish. The same model does not reach its English accuracy on Turkish text, so in our setup the keyword leg of hybrid search carries more weight than it would in an English deployment. When you pick a multilingual embedding model, measure it on your own set with your own language; the published general scores mislead.
• Do not translate terms. If the document says “chargeback”, keep it that way in the chunk; that is how users search for it too.

Teaching it to say “I don’t know”

The most dangerous answer is not a wrong one. It is an answer that is confident and unsourced. We set two rules:

# 1) Threshold: if the retrieved chunks are not similar enough,
#    never ask the model
if best_score < THRESHOLD:
    return MSG_NO_DOCUMENT   # "I have no document on this.
                             #  I can ask the policy team."

# 1b) If the permission filter removed every candidate, a SEPARATE message:
#     (see the permissions section above - we chose to split them)
if candidates_before_filter > 0 and candidates == 0:
    return MSG_NO_ACCESS     # "There is no document on this that you can see."

# 2) Mandatory citation: every claim attaches to a chunk.
#    after the answer is produced, code checks it:
for claim in answer.claims():
    if not claim.source_id:
        remove_from_answer(claim)   # unsourced sentences do not ship

The second rule sometimes shortens the answer, and that is a good thing. Under the text the user sees there is a source list; they can click through to the document itself. The nicest side effect we measured: users trust the system more now that they can verify it — trust comes from a source you can show, not from sounding certain.

From the field: three weeks, three fixes

We did not choose the order. Freshness came last because we understood the cause last: for the first two weeks we talked about the model “making things up” and looked in the wrong place. We also missed half of freshness in that round entirely: old versions staying in the index only surfaced three weeks after the round, when v8 shipped on 8 September.

The third row deserves an honest reading: our normal lag was never 14 days, it was at most a day with the nightly job. The 14 days were a failure caused by that job dying — the webhook shortened the lag, and the alert made the failure visible.

What to track

• recall@5. Weekly, on a fixed set of 30. If it drops, the document structure has changed.
• Index freshness. Per document: p95 and maximum of “last change at source − version in index”. The alert watches the maximum.
• Leftover versions. No chunk older than the source’s current version should remain; if the count is not zero, the delete step is not running.
• “No document” rate and unnecessary refusals. Read them together: if the first falls it is inventing, if the second rises it is useless.
• Unsourced claim rate (before the filter). After the filter it will be near zero; the one to watch is before.
• Wrong citation rate. By sampling, by hand. Even if you cannot automate it, know the number.
• Retrievals per chunk. Documents that are never retrieved are either unnecessary or badly split.
• Recall per permission profile. Recall measured with one profile is an upper bound; measure a narrow profile too.
• Permission filter accuracy. A chunk a user must not see should never enter the results; measure it with a test set.

Checklist

Conclusion

The agent read the old page correctly that day, and the answer was still wrong. For the next two weeks we talked about the model “making things up”, when the model had summarised what we gave it perfectly well. Once we separated the two numbers and measured retrieval on its own, it was clear in ten minutes: given the right chunk, answer accuracy was already 28/30, while the rate of retrieving the right chunk was 21/30.

Today the question is routed first, chunks are split at heading level, results are reranked, old versions are deleted from the index, below the threshold it says “I have no document”, and every answer carries the source name and date. The lag went from a day to two minutes — but the real gain is that we will know on the day it slips. What produced those 14 days was not slowness. It was silence.

The sentence to remember: in RAG the model is the last step; quality is decided in the steps before it. A system that summarises the wrong page well produces a wrong answer that looks trustworthy.