[EAAPL-PLT008] AI Experiment Tracking

Category: Platform Engineering Sub-category: MLOps / Evaluation Version: 1.1 Maturity: Proven Tags: experiment-tracking, mlops, model-evaluation, a-b-testing, evaluation-datasets, metric-tracking, model-registry, promotion-decision Regulatory Relevance: EU AI Act Article 9 (Risk Management), Article 17 (Quality Management), ISO 42001 Clause 9

1. Executive Summary

AI systems that cannot demonstrate systematic evaluation before production deployment are a governance liability. When a new model version, prompt change, or RAG configuration is promoted without structured comparison data, the organisation has no evidence that the change improves outcomes and no baseline to detect if it degrades them. Regulators, auditors, and increasingly, procurement processes require this evidence.

The AI Experiment Tracking pattern establishes the infrastructure for systematic, reproducible evaluation of all AI configuration changes. It covers the metadata schema for experiments (what was changed, what was measured, over what dataset), evaluation dataset management (the golden datasets that make metrics reproducible), metric computation and comparison (including human evaluation workflows for nuanced quality assessment), multi-run comparison dashboards, and the promotion decision audit trail that links every production configuration to the experiment that justified its deployment. This pattern transforms AI quality management from anecdotal to evidence-based, satisfying both engineering and regulatory audiences.

2. Problem Statement

Business Problem

AI feature quality degrades without detection because there is no systematic measurement. A model upgrade that performs better on marketing copy but worse on technical documentation goes undetected until customers complain. A prompt change that reduces cost also reduces accuracy in a way that only manifests at edge cases. Without experiment tracking, these regressions are invisible until they cause business impact.

Technical Problem

AI experiments are informal: engineers test a new model in a local notebook, eyeball a few outputs, and deploy. There is no reproducible evaluation framework, no standard metric set, no golden dataset, no comparison to baseline, and no record of the decision. When the promotion is later questioned, there is no evidence to review.

Symptoms

• Model or prompt changes deployed without documented evaluation results
• Different teams using different metrics to evaluate the same AI capability (no standard metric set)
• Golden datasets living in individual engineers' laptops or ad hoc S3 buckets with no versioning
• Post-incident analysis unable to identify whether the AI change or a data change caused quality degradation
• Regulators or auditors requesting evaluation evidence for production AI systems; none available

Cost of Inaction

• AI quality regressions reaching production that structured evaluation would have caught
• Regulatory non-compliance due to absence of quality management documentation
• Duplicated evaluation effort across teams using incompatible methodologies
• Inability to demonstrate AI improvement over time to business stakeholders

3. Context

When to Apply

• Organisation has AI models or prompts in production that change over time
• Multiple teams evaluate AI changes with no standard methodology
• Regulatory obligations require quality management documentation for AI systems
• A/B testing of model configurations is needed for production traffic comparison
• AI programme stakeholders require evidence of continuous improvement

When NOT to Apply

• Static, never-changing AI system with no evaluation lifecycle
• One-time AI analysis project with no ongoing deployment: lightweight evaluation sufficient
• Research prototype: production-grade experiment tracking overhead not warranted

Prerequisites

• Evaluation datasets (golden datasets) for each AI use case; these are the prerequisite that most organisations lack; they must be built before this pattern fully delivers value
• AI API Gateway for A/B traffic routing integration (PLT002/PLT003)
• Model Registry for linking experiment results to model versions (PLT001 Layer 2)
• Observability infrastructure for metric ingestion

Industry Applicability

4. Architecture Overview

The AI Experiment Tracking system is the measurement and evidence layer for all AI quality decisions. It is structurally analogous to a scientific lab notebook for AI—each experiment has a defined setup, methodology, results, and conclusion—but operationalised at engineering scale with automation.

Experiment Metadata Schema defines what an experiment record contains. Every experiment must record: a unique experiment ID, the component under evaluation (model name + version, prompt name + version, or RAG configuration version), the baseline configuration it is being compared to, the evaluation dataset reference (name + version), the evaluation metrics computed, the evaluation execution timestamp, the person or automated system that executed the evaluation, and the promotion decision record (approved/rejected with reason and approver). This schema is the foundation of the audit trail; every production configuration must have a traceable experiment record.

Evaluation Dataset Management is the most operationally demanding part of this pattern. Golden datasets for AI evaluation must be representative, version-controlled, and regularly maintained. A golden dataset consists of: input examples (representative queries or prompts), expected outputs (or reference outputs for similarity scoring), and metadata (creation date, curator, domain coverage statistics, known limitations). Datasets are stored in versioned object storage (S3, GCS) with a content-addressed hash ensuring reproducibility. A dataset update triggers re-evaluation of the current production configuration as a new baseline, ensuring metrics are always comparable on the same dataset version.

Metric Computation Framework standardises the metrics computed across all experiments. Common AI quality metrics include: accuracy (for classification tasks), factuality score (for knowledge-retrieval tasks, computed via citation checking or reference comparison), format compliance rate (for structured output tasks), latency percentiles (P50, P95, P99 inference time), token efficiency (output tokens per quality unit), and for safety-critical applications: toxicity rate, bias score, and hallucination rate. Metric computation is automated in a standardised evaluation harness that can be invoked from CI/CD pipelines and from the experiment tracking service.

Human Evaluation Workflow extends automated evaluation with human judgment for nuanced quality attributes that automated metrics cannot capture reliably: tone and brand voice consistency, logical coherence of long-form outputs, appropriateness of creative content, and clinical appropriateness in healthcare applications. Human evaluation is routed to qualified evaluators via a task queue; results are aggregated with inter-rater reliability scoring (Cohen's kappa) to ensure evaluation quality. Human evaluation is rate-limited by cost and evaluator availability; the pattern defines when human evaluation is required (high-risk use cases, major version changes) versus when automated evaluation is sufficient.

Multi-Run Comparison Dashboard provides the analytical view across experiments. The dashboard allows comparison of metrics across all experiments for a given component, over time (to detect trend improvements or regressions), and against the current production baseline. Statistical significance testing (two-proportion z-test for classification metrics, t-test for continuous metrics) is applied automatically; the dashboard distinguishes between statistically significant improvements and noise.

Promotion Decision Audit Trail links the decision to deploy a new configuration to the experiment that justified it. When a platform team member promotes a model or prompt version to production via the Prompt Registry (PLT005) or Model Registry (PLT001), they must reference an experiment ID that documents the evaluation. This reference is recorded in the promotion record and in the production configuration's metadata. Auditors can trace any production AI configuration to the experiment evidence that justified its deployment.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow — Automated CI Evaluation on Prompt Change

Error Flow

8. Security Considerations

• Evaluation datasets may contain sensitive representative inputs; they must be classified and stored with the same access controls as production data
• Experiment results (including individual output comparisons) may reveal model behaviour on sensitive inputs; access to raw outputs restricted to model owner and platform team
• Human evaluators must be bound by confidentiality agreements if evaluation involves sensitive content

OWASP LLM Controls

9. Governance Considerations

Quality Management System

• Experiment tracking constitutes the quality management system for AI as required by EU AI Act Article 17; every production configuration must have a traceable experiment record
• Promotion without experiment evidence is a governance policy violation; the promotion workflow enforces experiment ID as a required field

Governance Artefacts

10. Operational Considerations

Monitoring

SLOs

Disaster Recovery

11. Cost Considerations

Cost Drivers

Optimisations

• Use the cheapest model that produces comparable signals for evaluation runs where the evaluation target is a more expensive model
• Maintain a tiered evaluation strategy: fast automated eval (10 examples) for PR gate; comprehensive eval (250 examples) for promotion to staging; extended eval (1000+ examples) for production promotion of high-risk changes

Indicative Cost Range

12. Trade-Off Analysis

Evaluation Strategy Options

Metric Strategy Options

Architectural Tensions

13. Failure Modes

14. Regulatory Considerations

EU AI Act Articles 9 and 17

• Article 9 requires risk management measures for high-risk AI; experiment evaluation records constitute the technical documentation of risk management
• Article 17 quality management system requirement is satisfied by the experiment tracking infrastructure and promotion audit trail
• Technical documentation required by Article 11 must reference the evaluation methodology, datasets, and metrics used

ISO 42001 Clause 9 (Performance Evaluation)

• Experiment tracking directly implements Clause 9.1 (monitoring and measurement of AI system performance)
• Regular production quality sampling satisfies Clause 9.3 (management review of AI performance)

NIST AI RMF MEASURE 2.3

• Metrics for tracking AI performance over time must be defined and measured; the experiment tracking framework and metric time series DB satisfy this requirement

15. Reference Implementations

AWS

Azure

Open Source / SaaS

On-Premises

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven Experiment tracking with MLflow, W&B, and Azure ML is production-proven. AI-specific evaluation frameworks (Ragas, DeepEval) are maturing rapidly. The promotion-decision audit trail link is the least standardised component.

Scoring Matrix

18. Revision History