Sovereign RAG: the model matters less than your documents

The principle of RAG (Retrieval-Augmented Generation) is simple: instead of relying on what the model memorised during training, you fetch the relevant passages from your documents and hand them over at the moment of answering. The idea was formalised as early as 2020 and has become the default architecture of any serious enterprise AI, because it solves three problems at once: knowledge going stale, the inability to cite sources, and the need to keep data out of the model.

But "fetching the relevant passages" covers a chain of seven links. Each one can break silently, and the user will see only one thing: an answer wide of the mark, delivered with confidence.

1. Chunking: the most underestimated decision

A document does not fit whole into the model's context: it is split into passages. The size of those passages determines everything else.

Too small and they lose their context: an isolated paragraph starting with "In this case, the deadline is extended to two months" no longer says which case is meant. Too large and they dilute the useful information in filler, so the passage stops being distinctive at search time.

The real gain is not in tuning a character count, but in respecting the structure of the document: chunking at headings and subheadings rather than blindly, never breaking a table, allowing a slight overlap between consecutive passages, and above all attaching each passage to the heading of its section. An extract from a council decision that remembers it belongs to the "Planning" chapter of the 12 March session is worth ten anonymous extracts.

Every passage must also carry its metadata: source, date, issuing department, confidentiality level. These will serve for filtering, for citation, and for access rights.

2. Embeddings: the language-specific link

An embedding model converts a text into a numerical vector, so that two texts with a similar meaning end up close together in the space. That is what allows "Which documents do I need for planning permission?" to retrieve a document titled "Composition of a planning application file", without a single word in common.

Two points to watch in a French deployment:

• Multilingual support is not optional. Many embedding models are excellent in English and mediocre on French administrative language, its acronyms and its legal phrasing. That choice is tested on your documents, not on a generic leaderboard.
• Changing embedding model means reindexing everything. The vectors of one model are not comparable with those of another. It is an architecture decision, not a parameter you adjust along the way, which is exactly why it should be settled during the pilot.

3. Re-ranking: the link that makes the difference

Vector search is fast but approximate: it compares vectors computed separately for the question and for each passage. As a result it happily returns ten passages "on the right topic", of which only two actually answer it.

Re-ranking corrects this. A second model, a cross-encoder, which reads the question and the passage together, rescores the candidates and keeps only the best. It is more expensive, so it is applied to around twenty preselected passages, not to the whole store.

The effect is twofold: the useful passage rises to the top, and above all the context sent to the model is shorter and cleaner. Less noise means less hallucination and fewer tokens to process, so also less VRAM and more throughput. That is why our production stack puts a BGE reranker between retrieval and the LLM.

4. Access rights: the risk nobody tests

This is the most serious point and the one least often handled in pilot projects. A vector index built "over the whole network drive" is a machine for surfacing, to any staff member, an annual appraisal record, an HR note or a disciplinary file, simply because the passage was relevant to the question asked.

The rule is simple: rights apply to retrieval, not to the answer. Filtering afterwards, by asking the model "not to disclose", is not access control, it is politeness. The filter must be applied in the query to the index, based on the user's identity and the metadata of each passage.

It is also a compliance requirement: the GDPR imposes a minimisation duty that plays out precisely here, and the full reasoning is set out on our AI & GDPR page and in our security guarantees.

5. Evaluation: you only improve what you measure

Most RAG chains are tuned "by feel": you ask three questions, it looks better, you keep it. That is not sustainable as soon as several parameters are in play.

The method that works fits in one sentence: build a reference set of questions. Between 50 and 100 questions actually asked by users, each paired with the document that contains the answer. This set is built once, with the business teams, and becomes the yardstick for every technical decision that follows.

Three indicators are enough:

• Retrieval rate: in how many cases does the right document appear among the selected passages? If that figure is low, no model in the world will make up for it.
• Faithfulness: is the answer entirely supported by the passages supplied, with nothing invented added?
• Citation rate: does the answer point to a source the user can verify?

With that set, every change, a new chunking scheme, a different embedding model, the addition of re-ranking, becomes a measurement rather than an opinion.

6. So what about the model?

It comes last, and that is excellent news. When the retrieval chain does its job properly, the task asked of the model comes down to: read three to five passages, extract the answer, phrase it clearly and cite where it came from. That is a summarising task, not an encyclopaedic knowledge task.

An open weight model in the region of 24 billion parameters is more than enough, which is what we run in production, and the question of choosing one is covered in detail on our LLM models page. That changes the economics of the project completely: no need for the largest model on the market, therefore no need for a hyperscaler API, therefore sovereignty becomes affordable. The matching hardware sizing is worked out with our VRAM calculator.

Key takeaways

If an internal AI project disappoints, the reflex "let us change the model" is almost always the wrong one. In order, you first need to check that the documents are correctly extracted, that the chunking respects their structure, that retrieval surfaces the right passage, and that re-ranking cleans up the context. The model is only the last link, and by far the easiest to replace.

This is also what makes RAG compatible with sovereignty: your documents are never used to train anything, they stay in an index you control, and a piece of data removed from the index disappears from the AI the same day.