RAG 5: evaluate improvements without manufacturing evidence
How I would compare two systems using real questions and reviewed evidence without counting related chunks as independent tests.
In this article
The Taesa study showed that thousands of chunks do not guarantee thousands of independent observations. The next evaluation has another risk: replacing chunks with very similar synthetic questions while preserving the same statistical mistake. Before computing an interval, I need to define the target population and the units that generated those questions.
Build a reference set that can be challenged
I would start with questions from users and specialists, recording task, reference date, permissions and required evidence. Some questions allow several equivalent sources; others need two or more passages. I would include missing information, contradictions, older versions and queries outside the user’s access. A reference answer should not be reduced to one string when valid alternatives exist.
Two reviewers can examine a sample and resolve disagreements while preserving decision history. I would split development and test sets by families or cases and, when required by the application, by time. I would not claim this represents the entire company if questions are a convenience sample. An evaluation can be useful while remaining limited in scope.
Measure each stage and the complete answer
For retrieval I would measure evidence recall and, where graded relevance exists, nDCG. For generation I would review correctness, completeness, claim support and citations. An existing citation may not support the sentence. I would also report unsupported answering, unnecessary refusals, access violations, cost and p95 latency. An average must not conceal a serious failure in a document category.
EnterpriseRAG-Bench adds conflicts and duplicates to a synthetic enterprise corpus; its flattened documents do not reproduce PDF parser losses. EnterpriseRAG, a different 2026 work, combines noise, gaps and response constraints. These are distinct benchmarks. Both help design difficult cases, but neither replaces real documents or establishes that an LLM judge is sufficient for our evaluation.
EnterpriseRAG-Bench: synthetic corpus and limitations
EnterpriseRAG: constraints and insufficient evidence
Compare differences, respect groups
I would run variants on the same questions and compute paired differences. If independent cases were sampled with several questions within each, I would resample cases while retaining all their questions and both systems. When a question connects several documents, document grouping may be insufficient: cases, families or another defensible dependence structure are needed. Bootstrap alone does not fix biased sampling.
# Design pseudocode: independently sampled cases
for b in range(B):
sampled_cases = sample_cases_with_replacement(case_ids)
# Preserve multiplicity; keep questions and systems paired.
delta[b] = metric(B_system, sampled_cases) - metric(A_system, sampled_cases)
# Declare weighting, interval method and degenerate-replicate policy.I would not automatically use percentiles with few groups: the Taesa case already showed undercoverage. I would report sample size and composition, interval, method and calibration limits. Before testing I would set the minimum useful improvement and unacceptable regressions. If a reranker adds little while doubling latency, retaining the simple system may be the right decision.
Taesa case: resampling unit and limitations