RAG 6: from evaluation to reproducible operation
What I would prepare to maintain the system as documents, permissions or models change, and when I would consider retraining it.
In this article
A notebook finding evidence today does not prove the system will keep working when a file changes, a permission is revoked or a model is updated. After a favorable experiment, the next task is turning decisions into verifiable contracts. This is an engineering proposal for the next cycle, not a description of controls already deployed at Taesa.
A system version contains more than a model
I would version the corpus, parser, splitting rules, metadata, enrichment prompt, embeddings, index, retrievers, reranker, generator and evaluation. I would save configuration and dependencies with every result. An embedding change requires compatibility and index-rebuild checks; a permission correction requires invalidating affected results and caches. Ingestion must be idempotent: replaying an event must not duplicate documents.
To update the collection I would build a candidate version, check integrity and regression queries, and only then switch the active-index reference. I would retain a recoverable rollback version. Deletions and revocations cannot wait indefinitely for the next batch: they need a verifiable propagation policy, including the cache layer.
Tests that correspond to real failures
I would separate parser tests using difficult documents, schema validation, retrieval integration tests and answer evaluation. Cases would include broken references, headerless tables, duplicates, new versions and revoked access. A retrieved document is untrusted content: its instructions must not change permissions, execute tools or replace the user’s request. Injection scenarios belong in system tests.
Each run would record document IDs, index version, candidates, scores, filters and configuration under retention and access rules matching data sensitivity. There is no need to indiscriminately log complete questions and documents. Observability should locate failures in the corpus, extraction, retrieval or generation without creating another uncontrolled confidential collection.
When to train and when to repair the pipeline
I would not train to compensate for a table lost by a parser or a missing document. If a model receives correct evidence and systematically fails at a well-defined task, supervised training may be a hypothesis. A 2026 multi-hop study found different benefits for RAG, unsupervised training and supervised training, with the latter achieving the best accuracy in its comparison. This rules out claiming that RAG always outperforms any training.
Knowledge-injection comparison for multi-hop questions
SPA studies synthetic-data generation for continued pretraining. Its findings do not validate contextualized indexes. If I tested synthetic data, I would control diversity, faithfulness and test separation, comparing against a simple alternative. Changing facts requiring citations would still need a retrievable, updated source.
SPA: scope of the synthetic-augmentation study
Responsibilities and delivery criteria
The data scientist leads objective definition, sampling, metrics and error analysis. Data engineering owns ingestion, versions, lineage and updates. Software engineering integrates permissions, tests, deployment and recovery. Domain specialists review evidence and meaning. These are shared responsibilities with concrete deliverables, not a transfer of the problem to the model.
In the next cycle I want to save the corpus manifest, reviewed questions and configurations alongside rankings, errors and intervals. I would also report which changes did not help. That protocol has not run yet; what I present here is the test plan.
The October 2026 AGO AI Quality Gate preprint reinforces another caution: a judge can produce impeccable JSON while poorly distinguishing incorrect answers. Its judge and release studies have different scopes, including an experiment using synthetic outcomes. I would adopt the lesson of calibrating evaluators against human references, distinguishing missing evidence from poor results and measuring unsafe release decisions. I would not copy its thresholds without local validation.
AGO: validate the judge before promoting a version