RAG Knowledge Bases: Precision vs Breadth

Why This Is an Evaluation Tutorial, Not a Vendor Setup Guide

Retrieval-augmented generation, usually shortened to RAG, is often described as a straightforward recipe: retrieve relevant passages, add them to a language-model prompt, and expect more reliable answers. That description is incomplete. The available verified evidence does not support the claim that RAG improves every quality dimension, every task, or every model.

A June 2025 research paper, Reasoning with RAGged events: RAG-Enhanced Event Knowledge Base Construction and reasoning with proof-assistants, compares direct generation, knowledge-graph enhancement, and RAG for extracting structured historical events from narrative texts. Its case study uses the first ten chapters of Thucydides and evaluates multiple large language models, including GPT-4, Claude, and Llama 3.2.

The important lesson for teams planning a knowledge base is not that one architecture won universally. The paper finds that enhancement strategies optimize different performance dimensions. Direct generation favored coverage and historical breadth with Claude and GPT-4. RAG improved precision, coordinate accuracy, and metadata completeness, but reduced breadth. In other words, retrieval may help a system return more exact, better-supported details while also causing it to omit relevant events or facts that a broader approach might surface.

That is a practical engineering result. A knowledge-base team should not begin by asking, “Which RAG stack should we deploy?” The better first question is, “Which failure matters most for this use case: imprecision, missing coverage, incomplete metadata, or inconsistent behavior?” Your answer should shape the evaluation, the retrieval policy, and the acceptable trade-offs.

What the Verified Research Establishes

The study examines the difficult task of extracting structured representations of historical events from narrative sources. Manual construction is expensive, and the authors compare three approaches: pure or direct base generation, knowledge-graph enhancement, and retrieval-augmented generation. The work also addresses limitations in RDF/OWL reasoning by translating RDF outputs into Coq proof-assistant specifications for deeper temporal and semantic analysis.

For a production knowledge base, the most transferable evidence concerns evaluation discipline. The study shows that RAG is not a blanket substitute for direct model generation. Its value depends on what the team measures. Where precision, coordinate accuracy, and metadata completeness are especially important, retrieval can be beneficial. Where comprehensive extraction and breadth are more important, a direct-generation approach may have an advantage.

The paper also identifies model-dependent behavior. Larger models showed robust baseline performance with incremental benefits from RAG. Llama 3.2 displayed substantially greater variance, ranging from competitive outcomes to catastrophic ones. This means that an architecture decision cannot safely be separated from model evaluation. A retrieval pipeline that appears effective with one model may behave very differently with another.

These findings should change how an organization frames “grounding.” Retrieved material can improve a system’s precision, but retrieval also constrains what the model sees. If the retrieved evidence is narrow, incomplete, or poorly aligned with the task, the final output can become narrow as well. A system that only measures whether an answer contains a citation may miss this problem entirely.

Step 1: Define the Knowledge Task Before Choosing an Architecture

Start by writing a one-page task definition. Identify the input material, the desired output, the users, and the cost of errors. A historical-event extraction workflow is different from an internal policy assistant, but both need an explicit definition of success.

• Input: What source material will the system use? Define the document collection and its boundaries.
• Output: Is the expected result a concise answer, a structured record, a set of extracted facts, or a ranked set of passages?
• Coverage: Must the system identify all relevant facts, or only the most strongly supported facts?
• Precision: How costly is an incorrect fact, incorrect date, incorrect entity, or incorrect attribute?
• Metadata: Which fields must be complete and correct for the result to be useful?
• Human review: Which outputs require expert verification before they can be acted upon?

This step prevents an avoidable mistake: evaluating a system on the metric that is easiest to collect rather than on the metric that reflects user value. A compliance-oriented knowledge base...