Data science and AI

Regulatory RAG: from chunks to provisions with identity and temporal validity

Part 2 of the ANEEL–TAESA case: I examine retrieval errors and the information a chunk needs to distinguish regulations and versions.

Jean Michael Estevez AlvarezPublished: 11 Oct 2026

In Part 1 the question was statistical: how far can we trust a number computed over thousands of chunks. The results also left a practical question. BM25 recognized the source of only 18.46% of chunks, and 639 of the 1,300 documents were never recognized at all. Why does a search engine get a regulatory collection so wrong, and what would have to change in how these texts are stored for RAG to answer with the right rule?

One hypothesis is confusion between similar versions. For example, a query about a 2021 statement might retrieve its 2022 counterpart; this is an illustration, not an individual error documented by the CSVs. The year was stored as metadata but not used as a filter; it could also occur in the text. Before attributing failures to temporal validity, I need to preserve and review rankings.

My argument here is that the quality of a regulatory RAG system starts before embedding, with what is preserved and how it is organized. First I describe what the experiment did and what it did not test. Then what preparation and chunking left behind along the way. Finally I propose a provision-level representation and a test plan. The proposals are mine and have not yet been validated.

Part 1: findings and limits of the ANEEL–TAESA evaluation

What the experiment tested and what it left out

The experiment measured document attribution with BM25 and TF-IDF. It did not implement legal segmentation, temporal-validity resolution, dense retrieval or answer generation. The 18.46% and the 639 documents with no hits (49.15%) are therefore a signal to investigate versions and the ability to tell similar documents apart. They do not measure answer accuracy, nor do they prove that every confusion is temporal.

The pattern by document type points the same way. Reference forms and annual statements, which repeat with small changes, stayed below 10%. Normative resolutions and orders, which are more distinctive, exceeded 40%. There is another asymmetry in size: the 130 largest documents hold almost 60% of the chunks, so whatever happens to them weighs heavily on any average.

Chunking: what gets cut and what gets lost

Chunking means splitting a document into units a search engine can index and return. It looks like an implementation detail, but size decides how much information travels together. A short chunk isolates a concept well and may separate a condition from its exception. A long one keeps context and mixes topics that compete in search. Overlap repeats some text between units to soften the cuts, at the cost of duplication, storage and redundant context.

In the study code, the fatiar function splits text on whitespace into blocks of 50, 100, 200 or 400 words, without overlap. If the last block is shorter than half the chosen size, it is discarded: with 100-word blocks, a 49-word remainder disappears. A 20,000-word cap per document is applied before splitting, and the first 20 words of each block form the query. Articles, items and annexes are not detected. It is length-based segmentation, suited to the experiment's goal, not a provision parser.

That is also why the study does not say which chunk size to use in production. Changing the size changes the queries and the task itself. Comparing splitting strategies would require holding fixed the real questions, acceptable evidence, corpus, context budget and evaluation criteria.

Before the cut: what preparation left out

RAG cannot retrieve what never made it into the index, and text preparation settles that before any model is involved. The 20,000-word cap affected only 56 documents, 4.3% of the total. Even so, it cut the text from 7.0 to 3.35 million words, about 52% of the volume (calculated from rounded figures). These were mostly reference forms and prospectuses of up to 200,000 words. A small share of documents can hide a large loss of content. Before raising top-k or switching models, I would check whether annexes and final clauses were left out.

Cleaning produced a curious case. To strip institutional headers, lines appearing in at least 5% of a source's documents were removed. Among them was the standard effective-date clause ("Art. X. This resolution enters into force..."), precisely the line that says when a rule starts to apply. For the attribution experiment the choice makes sense, since that line is identical across hundreds of acts. For RAG that has to answer questions about validity, it would be a serious loss. Cleaning also excluded 20 PDFs without OCR and 279 texts with fewer than 150 useful words, mostly orders that shrank to a few lines once the header was gone.

These are reasonable experimental choices that need review before reaching production, because a short or repeated provision can be essential. I would keep the original, the extracted and cleaned text, pages, tables, units, hashes and an exclusion log. And I would test OCR and table extraction on a human-reviewed sample.

What a chunk needs to carry with it

If the problem is that a chunk reaches the model without knowing where it comes from, when it dates from and what it depends on, the way out is to make it carry that information. In a regulatory RAG system, a document should be represented as a set of provisions, each with five elements.

A fictional example shows why each element matters. Article 5 sets an obligation, a paragraph opens an exception and a later act changes only item II. If the search engine returns the item without the article's opening text, the reader cannot tell who it applies to. If it returns the original article after the amendment, it shows superseded wording. What I would test is storing each provision with its parent and relationships, retrieving the relevant passage and then expanding the context with the rule, the exception and the applicable version. One caveat: document hierarchy does not by itself establish legal precedence between instruments.

This way of modeling is not new. OASIS Akoma Ntoso already treats identification, references and lifecycle as part of how a legal document is represented. A pilot does not need the full XML vocabulary or a graph database: identifiers, relationship tables and reviewed metadata are enough to start. What does require evidence and expert review is resolving conflicts, exceptions and partial amendments. A date in a filter does not settle applicability on its own.

OASIS: Akoma Ntoso vocabulary and document lifecycle

Split by structure, not by word count

In practice, I would first try splitting articles and their subdivisions while keeping headings and links. If a unit does not fit the token budget, I would split it by paragraphs or sentences, always keeping the parent provision identifier and without inventing new legal provisions. Tables need to keep headings, units and notes; annexes, their references. The result can pair small units for locating evidence with a larger context for answering. Overlap stays a test variable, not a substitute for explicit relationships.

First the right version, then hybrid retrieval

My starting point would be BM25 with reliable metadata: act identifier, entity, type, publication, reference period, effective start and end when known, and amendment or replacement relationships. Publication is not the same as validity. A question about 2021 should not automatically pick the latest edition. If the date is ambiguous, the system should ask for clarification or say which date it is using.

Next I would compare BM25, dense retrieval and the two combined. BM25 is a good baseline for act numbers and exact terms; embeddings may help with paraphrases. RRF fusion combines list positions instead of adding scores on different scales, and a reranker can reorder candidates before generation. All of these are improvement hypotheses, not report findings. BEIR is a useful reminder that retrievers behave differently from one task to another.

Cormack, Clarke and Büttcher (2009): Reciprocal Rank Fusion

Thakur et al. (2021): BEIR, heterogeneous retrieval evaluation

Read from left to right, the flow keeps documents and versions, extracts passages with their provenance, combines lexical and semantic retrieval with date and permission filters, reranks candidates and generates an answer with citations that can be checked. If evidence is missing or validity conflicts, the system can abstain and send the query for review. Monitoring links each answer error back to the passage, index version and configuration that produced it.

Which parameters I would test and what they cost

As a starting point, not a validated recipe, I would test section-based chunks of 256 and 512 tokens, 0 and 15% overlap, 20 and 50 candidates per retriever, and 5 and 10 passages after reranking. Tokens are not words, so this does not compare directly with the experiment's 100 words. I would try to keep articles, tables and headings together. Every knob has a price: more overlap adds duplication and storage; more candidates and reranking add latency and cost. That is why I would also test a version with no embeddings and no reranker.

Nor would I send the whole archive to the model to make up for weak retrieval. Lost in the Middle showed that, for the models and tasks it studied, where evidence sits in the context affects the result. That does not prove every current model fails the same way, but it justifies testing context order, length and relevance with the model you actually plan to use. And a larger context still cannot repair an extraction step that deleted the evidence.

Liu et al. (2024): Lost in the Middle

A test plan that can change the decision

I would start with a pilot of real questions reviewed by specialists, including the hard ones: dates, similar versions, exact numbers, tables, annexes and questions with no answer. For each I would record the acceptable evidence and the reason for the label. Training, tuning and the final test would be split by document family and by time, because splitting chunks at random can put almost the same text on both sides. Final sample size would follow from the precision needed and the risk involved, not from the rows available.

I would measure retrieval and answers separately. Recall@k needs a reasonably complete set of relevant evidence; MRR measures the rank of the first relevant result and nDCG allows graded relevance. I would add correct version, claims backed by citations, faithfulness, numerical accuracy and appropriate abstention, plus p95 latency and cost per query. If an automated judge is used, it has to be calibrated against people and its disagreements audited, especially for the highest-impact errors.

Configurations would be compared on the same questions, with paired differences and intervals that respect the design, as discussed in Part 1. In the report, BM25 beat TF-IDF by only 0.62 points on the micro-average and 0.27 on the macro-average, with ties in 70% of documents, and document bootstraps covered between 85% and 89% in that comparison. A method's name is no substitute for checking its calibration. I would freeze versions of the corpus, extractor, chunking, embeddings, index, reranker, prompts and model so that every change can be explained.

Before the final test, I would agree with users on a minimum useful improvement and limits for cost, latency and temporal errors. A new stage would stay only if it improved the quality that matters without breaking those limits. If filtered BM25 ties with the hybrid system, I would keep the simpler one. If evidence is missing from the index, the problem is ingestion. If the generator makes things up even with the right evidence in front of it, I would review prompts and behavior before expanding the index. None of this has been validated in production yet.

The analyst’s work does not end at the metric

What a data scientist brings here is the link between the business question and the evidence: defining what is being estimated, spotting exclusions, designing the sample and finding the mechanism behind errors. In production that includes watching for shifts in the question mix, stale documents, extraction failures and reindexing costs. It also means checking permissions before retrieval, treating instructions found inside documents as untrusted data, and testing for false citations, leaks and unauthorized queries. A high average can hide a serious failure in a rare document family.

My recommendation is to start with document coverage, versions and a representative evaluation, and only then justify each additional layer. The report gives solid evidence against treating chunks as independent replicates, but it does not pick a winning architecture. I think that is a healthy limit. It keeps a convincing number from turning into a promise the system cannot keep.

Provenance and next step

Source: the revised report, resultados.json and desfechos_L100.csv. Concentration and the 639 zero-hit documents were calculated from saved outcomes. The statistical revision added a design check; it did not evaluate this architecture. The diagrams remain proposals. The next study should compare representations on the same questions before adding hybrid retrieval.

Additional aggregates and source hashes (JSON)

Back to the blog index

Methodological revision · October 11, 2026

I conducted the original experiments as Jean M. Estevez Alvarez. This revision distinguishes those results from subsequent checks: it preserves the original outcomes and coverage results, corrects derived quantities and adds 5,000 repetitions of the two-stage design. There was no new retrieval run, training or production evaluation.

RI selection had two filters: up to 35 candidates per type after initial screening and up to 25 included after text cleaning and review. Final exclusions were not automatically replaced. Original rankings use implicit tie-breaking and do not preserve retrieved candidates; I therefore do not attribute each error to a wrong version without additional evidence.

Design revision results (JSON)

Field and Welsh (2007): models and limits of the cluster bootstrap

Bell and McCaffrey (2002): original BRL paper