EAAPL-KNW006: Corpus Quality Assurance

Pattern ID: EAAPL-KNW006 Status: Proven Complexity: Medium Tags: observability model-risk traceability medium-complexity Version: 1.0 Last Updated: 2026-06-12

1. Executive Summary

Corpus Quality Assurance (CQA) is the automated pipeline that evaluates the fitness of documents for inclusion in an AI knowledge corpus — before ingestion and on a continuous basis after ingestion. It is the quality control function that stands between raw enterprise content and the retrieval systems that power AI answers.

This pattern defines six quality dimensions — completeness, accuracy, duplication, staleness, coverage, and structure — and specifies the automated measurement, threshold gating, and alerting mechanisms for each. It also covers the quality trend dashboard and the escalation path when automated quality assurance cannot make a determination.

For CIOs and CTOs, the core argument is: AI answer quality cannot exceed corpus quality. Teams investing in LLM selection, prompt engineering, and retrieval architecture while neglecting corpus quality are optimising the wrong variable. A well-tuned AI system on a poor corpus will outperform a poorly tuned system on a good corpus in the short term, but the corpus quality deficit compounds — AI answers degrade as documents age, duplicate, and diverge, while the AI model remains fixed. CQA is the ongoing quality investment that protects AI answer quality as the corpus grows and ages.

Implementation is medium complexity. Unlike knowledge graph or semantic layer patterns, CQA does not require new data infrastructure — it adds a quality measurement and gating layer to existing document ingestion pipelines.

2. Problem Statement

2.1 Business Problem

Enterprise knowledge corpora degrade without active management. Documents become outdated as policies and products change, but the old versions remain in the retrieval index — the AI continues citing superseded information. Duplicate documents accumulate as the same content is ingested from multiple sources with minor variations, causing inconsistent retrieval. Poorly formatted or truncated documents produce low-quality retrieval chunks that confuse rather than inform the LLM. The business consequence is AI answers that become less reliable over time, eroding user trust in proportion to the corpus quality deficit.

2.2 Technical Problem

Retrieval quality in RAG systems is directly determined by the quality of the documents retrieved. Standard vector similarity search has no quality awareness: a low-quality, outdated document with high semantic similarity to a query will be retrieved in preference to a high-quality, current document with slightly lower similarity. Without quality scores attached to documents and factored into retrieval ranking, quality degradation is invisible to the retrieval algorithm.

2.3 Symptoms

• AI cites version-superseded documents (e.g., a policy withdrawn 18 months ago)
• Same question receives different answers on different days because duplicate documents with conflicting content are retrieved inconsistently
• Truncated or corrupted documents appear in retrieval results; LLM produces incoherent answers for those queries
• No metric exists for corpus health; the team does not know if quality is improving or declining
• Coverage gaps are discovered reactively (users complain the AI "doesn't know" about a topic) rather than proactively

2.4 Cost of Inaction

• AI adoption reversal: business units that experience repeated quality failures disengage and revert to manual research
• Regulatory risk: AI answers based on superseded regulatory or compliance documents produce incorrect guidance with potential legal consequences
• Compounding quality debt: the longer quality management is deferred, the more documents require remediation, and the larger the quality remediation project becomes
• Lost insight: coverage gaps mean entire knowledge domains are unrepresented in AI answers — the system is not aware of what it doesn't know

3. Context

3.1 When to Apply

• Any production RAG system with >500 documents — below this scale, manual review is feasible; above it, automation is necessary
• Corpora with multiple document sources and types of varying quality (the diversity creates the quality variance that requires automated management)
• Domains with regulatory compliance implications where document currency is essential (compliance, legal, product, medical)
• Organisations with high document update velocity — quality degrades fastest where content changes frequently
• As a companion to EAAPL-KNW003 (AI Knowledge Corpus Management) — CQA is the quality assurance function; KNW003 is the lifecycle management function

3.2 When NOT to Apply

• Single-source corpora from a single authoritative owner with manual review already in place — CQA overhead is not justified
• Pure experimental/prototype deployments where AI answer quality is not yet a business concern
• Corpora updated in batch by a controlled process with built-in quality controls upstream — additional CQA layer may be redundant

3.3 Prerequisites

• Document metadata standard: at minimum, source system, author, effective date, expiry date, document type
• Document ingestion pipeline with an interception point where quality checks can be executed before final ingestion
• Storage for quality scores and quality history per document
• Alerting infrastructure for quality threshold violations

3.4 Industry Applicability

4. Architecture Overview

The Corpus Quality Assurance architecture comprises two operational phases: Pre-Ingestion Quality Gating and Post-Ingestion Continuous Quality Monitoring, unified by a shared quality score store and health dashboard.

4.1 Pre-Ingestion Quality Gating Pipeline

Every document submitted for corpus ingestion passes through six quality checks in sequence:

Completeness Check. The completeness scorer evaluates whether the document is whole and self-contained. Checks include: minimum word count for the document type; absence of truncation indicators (sentences that end abruptly, missing conclusion sections, "Page X of Y" indicators suggesting missing pages); presence of expected structural elements for the document type (a policy document without a "Scope" or "Effective Date" section is flagged as incomplete); broken internal references (citations to sections that don't exist in the document). Completeness score: 0–1.

Accuracy Validation. Accuracy validation operates at two tiers. Tier 1 (automated): for documents in high-stakes domains, automated fact-claim extraction identifies specific factual assertions (percentages, thresholds, named entities with specific attributes) that can be cross-checked against a trusted reference source (a regulatory database, a product master data system, an authoritative ontology). Claims that contradict the reference source reduce the accuracy score. Tier 2 (human): a statistically sampled proportion of documents from high-stakes domains are routed to a human accuracy reviewer — a domain expert who validates a representative sample of claims against primary sources. The human sampling rate is configurable per domain (typically 2–10% for high-stakes; 0.1–0.5% for informational domains).

Duplication Detection. Exact duplication is detected via cryptographic hash (SHA-256 of document content, normalised for whitespace and formatting). Near-duplicate detection uses cosine similarity of document-level embeddings: documents with similarity above a configurable threshold (default 0.95) are flagged as near-duplicates. For near-duplicate pairs, the deduplication strategy is configurable: reject the newer document (preserve canonical version), merge metadata (combine source attributions), or route to human review to determine which is authoritative. Exact duplicates are always rejected silently.

Staleness Evaluation. The staleness scorer evaluates document freshness relative to domain-specific maximum age thresholds. Threshold configuration is per document type and domain: regulatory instruments (12 months), internal policies (6 months), product technical specifications (3 months), market data summaries (1 week). The staleness score decays from 1.0 (fully fresh) to 0.0 (at maximum age) and goes negative (below 0) when past expiry — a document past expiry cannot be ingested. The score factors in not just age but also the velocity of change in the domain: regulatory areas with a recent burst of amendments require more frequent refresh.

Structural Integrity Check. The structural checker validates that the document can be processed by the downstream chunking pipeline. Checks: document is machine-readable (not a scanned image without OCR text layer); character encoding is valid UTF-8; no binary artefacts that would confuse chunking; minimum retrievable text content (>100 words of coherent prose). A structurally invalid document cannot produce useful retrieval chunks.

Coverage Assessment. The coverage assessor checks whether the document adds genuine value to the corpus by mapping its content to the knowledge ontology. If the document's topic is already represented by ≥N high-quality, current documents, the new document's incremental value is low and it is deprioritised or queued for later ingestion. If the document covers a topic with <N representations, it is flagged as a coverage gap filler and prioritised.

Composite Quality Score and Gate. Each of the six dimension scores is combined into a composite quality score with configurable weights per document type. Documents above the high-quality threshold are auto-ingested. Documents in the middle band enter a quality review queue where a document owner is notified with specific dimension-level feedback. Documents below the minimum threshold are rejected with a detailed rejection report.

4.2 Post-Ingestion Continuous Quality Monitoring

Quality degrades after ingestion as time passes and the broader knowledge landscape changes. The continuous monitoring layer runs scheduled jobs to re-evaluate the active corpus.

Freshness Monitor runs daily: re-scores all documents' staleness dimension; flags documents approaching the pre-expiry warning threshold; triggers automated expiry at the hard threshold.

Recall Probe runs weekly: executes a golden query set against the active corpus; measures recall@k for each query category; a decline in recall for a specific category indicates that the documents covering that topic have degraded in quality or have been removed.

Duplication Drift Monitor runs weekly: checks for near-duplicates introduced since the last run; newly ingested documents are compared against the existing corpus; cross-document contradictions (same topic, conflicting factual claims) are detected and flagged.

Coverage Gap Monitor runs monthly: maps the active corpus against the knowledge ontology; identifies topics with declining document counts (documents aged out without replacement); generates a prioritised ingestion backlog for the content management team.

4.3 Quality Score Store and History

All quality scores (pre-ingestion and post-ingestion re-evaluation) are stored per document with timestamps. This enables quality trend analysis: is a domain's average quality improving or declining? Are specific document types consistently failing particular quality dimensions? The quality history also supports root cause analysis when AI answer quality issues are investigated.

5. Architecture Diagram

6. Components

7. Data Flow

7.1 Primary Data Flow — Pre-Ingestion Quality Gate

7.2 Error Flow

8. Security Considerations

8.1 Authentication and Authorisation

The quality gate API authenticates document submissions using the same source authentication mechanism as the corpus management pipeline. Quality scores are internal operational data and are accessible to corpus administrators, data stewards, and the AI platform engineering team. Quality history records for a specific document are accessible to the document owner. The quality review queue interface requires MFA-enabled SSO.

8.2 Secrets Management

Reference source API credentials (for accuracy cross-checking), embedding model API keys (for duplication detection), and quality score database credentials are stored in a secrets vault with standard rotation.

8.3 Data Classification

Quality scores are metadata and carry the same classification as the document they describe. A quality report for a Confidential document is itself Confidential. The quality health dashboard aggregates are typically Internal classification (no individual document details).

8.4 Encryption

Quality score store: encrypted at rest (AES-256). Data in transit: TLS 1.3. Accuracy reviewer interface: HTTPS-only with session token management. Documents processed by the quality pipeline are processed in-memory only where possible; no sensitive content written to intermediary disk storage.

8.5 Auditability

All quality gate decisions are logged: document ID, timestamp, submitter identity, all dimension scores, composite score, gate decision (auto-ingest/review/reject), rejection reason if applicable. For documents entering the review queue, the reviewer's identity, decision, and timestamp are logged. This audit trail enables investigation of any document's quality history.

8.6 OWASP LLM Top 10 Mapping

9. Governance Considerations

9.1 Responsible AI

Quality thresholds are value judgements encoded as configuration. A high completeness threshold that rejects documents with non-standard formatting may systematically exclude content from certain sources or geographies that format differently. Quality threshold calibration should include a bias audit: do the thresholds disproportionately exclude any legitimate document types, sources, or domain perspectives? Results are reviewed annually.

9.2 Model Risk Management

The accuracy validator's claim extraction and reference cross-checking component is a model. Its false negative rate (claims it fails to flag as inaccurate) determines the probability of inaccurate facts reaching the corpus. This model is subject to model risk management: model card documenting training data, precision/recall on validation set per claim type, known failure modes, and refresh schedule. The human accuracy review sampling programme provides an independent validation signal.

9.3 Human Approval Gates

All documents in the quality review queue require human action: either the document owner remediates the quality issue and resubmits, or the document is permanently rejected. Rejected documents cannot be automatically re-submitted — re-submission requires the owner to explicitly acknowledge the original rejection reason. Human accuracy reviewers for high-stakes domains complete competency validation before being assigned review tasks.

9.4 Policy Ownership

Quality threshold policy (minimum dimension scores, composite weights, human review sampling rates, domain-specific freshness schedules) is owned by the Corpus Governance Board. Quality threshold changes require a 10-business-day review period and simulation of the impact on the existing corpus (what percentage of currently active documents would fail under the new thresholds). Threshold changes that would invalidate >5% of the active corpus require executive approval.

9.5 Traceability

Every document in the active corpus has a complete quality history: all dimension scores at each quality gate evaluation, all continuous monitoring scores, all human review decisions, and the current quality score. This history supports both root cause analysis (why did AI answer quality degrade in domain X?) and compliance reporting (confirm that all documents in the corpus met quality standards at ingestion).

9.6 Governance Artefacts

10. Operational Considerations

10.1 Monitoring and SLOs

10.2 Logging

All quality gate events are logged as structured JSON: {document_id, timestamp, source, dimension_scores{}, composite_score, gate_decision, rejection_reason, reviewer_id}. Continuous monitoring events: {run_id, timestamp, check_type, documents_evaluated, alerts_generated}. Recall probe results: {run_id, timestamp, query_category, recall_at_k, threshold, pass_fail}. Log retention: 90 days operational; 7 years archive.

10.3 Incident Management

P1: Recall probe shows recall@5 below 0.70 — immediate investigation of recently ingested documents; potential recall of batch ingestion. P2: Average corpus quality score below threshold in a critical domain (compliance, medical) — same-day quality audit; halt of new ingestion until root cause identified. P3: Review queue SLA breach; single domain coverage gap — next business day response.

10.4 Disaster Recovery

10.5 Capacity Planning

Quality gate processing is CPU-intensive for large documents (structural parsing, embedding generation for deduplication). At high ingestion rates (>1,000 documents per day), parallelise quality gate workers with a job queue. The quality score store grows at approximately 1 KB per quality evaluation per document; a corpus of 100,000 documents with monthly re-evaluation accumulates ~1.2 GB per year — manageable.

11. Cost Considerations

11.1 Cost Drivers

11.2 Scaling Risks

• Human accuracy review is the primary cost scaling risk: if the high-stakes document volume grows and sampling rates are maintained, review labour grows proportionally
• Accuracy validator LLM cost can be significant for large, complex documents with many factual claims — restrict deep accuracy validation to genuinely high-stakes domains
• Recall probe cost scales with golden query set size and retrieval computation — keep golden set to 200–500 representative queries

11.3 Optimisations

• Hash deduplication is free (CPU-only) — always run before embedding-based near-duplicate detection
• Tier accuracy validation by document classification: Restricted documents get full claim extraction and human review; Internal documents get automated checks only
• Cache quality scores for documents that have not changed between evaluation runs (hash-based change detection)
• Use smaller embedding models for deduplication (the absolute embedding values matter less than the similarity ranking)

11.4 Indicative Cost Ranges

12. Trade-Off Analysis

12.1 Quality Gate Strictness Options

12.2 Accuracy Validation Approaches

12.3 Architectural Tensions

13. Failure Modes

13.1 Cascading Failure Scenarios

Scenario 1: Quality Gate Configuration Drift. Over 18 months, quality thresholds are relaxed in small increments to keep pace with ingestion volume pressures. No single change is significant enough to trigger governance review. Average corpus quality declines from 0.82 to 0.68. AI answer quality declines proportionally. Detection occurs when a business unit escalates multiple AI errors in a high-profile project. Root cause analysis reveals the threshold drift. Resolution requires: threshold reset to original values; corpus-wide retrospective quality re-evaluation; purge of documents now below threshold; 3-month remediation project.

Scenario 2: Reference Source Compromised. The external regulatory database used as an accuracy validation reference source is compromised and injects incorrect threshold values for a compliance domain. The accuracy validator cross-checks document claims against the now-incorrect reference and accepts inaccurate documents. AI begins providing incorrect compliance guidance. Detection: compliance team notices AI answers contradict known regulatory requirements. Resolution: immediate removal of affected documents from corpus; reference source integrity investigation; temporary switch to human accuracy review until reference source is validated clean; add cross-check against secondary reference source.

14. Regulatory Considerations

15. Reference Implementations

15.1 AWS

15.2 Azure

15.3 GCP

15.4 On-Premises

16. Related Patterns

17. Maturity Assessment

Overall Maturity Label: Proven

18. Revision History