RAG 1: a collection you can audit
How I collected guides and studies with Scrapling and saved their versions so I could trace each decision back to its source.
In this article
After reviewing the ANEEL–TAESA experiment, I did not want to change the search system merely because a recipe sounded convincing. I needed to know what each source recommended, what it had tested and what I could reproduce. I started with Contextual Retrieval, its notebook and the Claude RAG catalog. This series describes that review and a proposed procedure; it does not report a newly measured improvement on Taesa.
First I bounded the collection
I used the local Scrapling 0.4.15 checkout with Fetcher for ordinary HTTP requests. The catalog contained 19 guides tagged RAG & Retrieval. Its HTML also contained links from other categories: following every anchor would have produced a different collection. I selected rows by their category label and saved the URL list before downloading their detail pages.
The capture preserved HTML, text and links, plus linked notebooks and twelve selected research texts. The log contains 62 HTTP 200 responses; these are not 62 independent documents because redirects and duplicates exist. Coverage includes the identified pages and notebooks, not a mirror of the entire domain, its images and datasets. Third-party code was downloaded, not executed.
What I kept for reproducibility
Each response has its requested and final URL, UTC timestamp, status, size and SHA-256. Extracted text has its own hash. I saved the notebook commit and hashes of the local collector code. A hash detects byte changes; it does not prove truth, completeness or legal validity. Those require separate checks.
# PowerShell: new snapshot; scripts in the research workspace
$env:RAG_COLETA_DIR = "C:/Desenvolvindo/CientistaDeDados/rag_snapshot_novo"
python -X utf8 rag_pesquisa_2026/coletar.py
python -X utf8 rag_pesquisa_2026/coletar_complementos.py
python -X utf8 rag_pesquisa_2026/coletar_notebooks.py
python -X utf8 rag_pesquisa_2026/verificar_coleta.py
python -X utf8 rag_pesquisa_2026/testar_coleta.pyOn another machine, install the checkout dependencies and adjust its path. These commands run from the research repository, not the blog server. A new directory prevents overwriting the previous capture. Then inspect coverage, titles, body text and code; HTTP 200 alone can also return an empty page or an access challenge.
A paper catalog is not a complete review
The bibliography repository lists 265 works, but its analysis explains that only two received full-text reading. Captured commit 8f548e6 contains README.md and Analise.md; the referenced portfolio directory and PDFs are absent. I used the catalog to discover sources and consulted primary texts for central claims. I recorded these gaps rather than treating the missing material as reviewed.
The Portuguese synthesis is original; full source copies remain in the local research archive. I also distinguished the 2024 editorial date from the current code version. This phase produces an auditable collection, recorded limitations and engineering questions. It is not yet evidence that a new architecture performs better.
Starting sources: bibliography repository