[EAAPL-GOV006] Model Bias Detection

Category: Governance / Fairness Engineering Sub-category: Continuous Bias Monitoring Version: 1.2 Maturity: Proven Tags: bias-detection, fairness, demographic-parity, equalised-odds, calibration, retraining, continuous-monitoring Regulatory Relevance: EU AI Act Article 9(7), NIST AI RMF MEASURE 2.5, APRA CPS230 §20, Anti-Discrimination Act, ASIC RG 271

1. Executive Summary

The Model Bias Detection pattern implements a continuous pipeline for detecting, measuring, and alerting on statistical bias in AI model outputs post-deployment. It provides the ongoing counterpart to the pre-deployment fairness assessment conducted via the AI Risk Assessment Framework (GOV002).

Bias in AI systems is not static. A model that passes pre-deployment fairness assessment can develop bias as the world changes: population demographics shift, user behaviour patterns evolve, feedback loops form, or retraining data introduces new skews. Without continuous monitoring, this drift is invisible until a regulator, journalist, or customer complaint surfaces it—at which point the enterprise has both an operational problem and a governance failure to explain.

The pattern implements three measurement approaches—demographic parity, equalised odds, and calibration—as a continuous pipeline consuming live inference logs. When bias exceeds configured thresholds, the pattern triggers a graduated response: alert (notify), restrict (limit model scope), retrain (initiate model update), or escalate (human governance review). This graduated response prevents both under-reaction (ignoring findings) and over-reaction (shutting down useful models on minor statistical fluctuations).

For regulated Australian entities, the pattern provides the technical control that satisfies obligations to prevent AI-driven discrimination under the Australian Human Rights Commission AI Framework and ASIC's responsible lending guidance (RG 271).

2. Problem Statement

Business Problem

AI models can produce discriminatory outcomes across protected customer segments (age, gender, race, disability) without anyone in the organisation detecting it in time to prevent harm. Manual auditing is infrequent and retrospective. Regulatory investigations expose bias that has been operating for months or years.

Technical Problem

Fairness metrics cannot be computed in real time at inference (demographic attributes are often not available for every request). A windowed batch approach is required, but batch frequency must be short enough to detect and remediate bias before material harm accrues. The statistical significance of fairness measurements depends on sample size, requiring careful threshold calibration to avoid spurious alerts on small subgroups.

Symptoms

• Fairness testing is a one-time pre-deployment activity with no continuous monitoring
• Bias complaints from customers or regulatory bodies are the first indication of a fairness problem
• Model retraining decisions made purely on accuracy metrics without fairness reassessment
• Protected attribute data not collected or retained, making retrospective fairness auditing impossible
• Different business units defining fairness differently, creating inconsistent measurement

Cost of Inaction

• Regulatory: Anti-Discrimination Act enforcement; ASIC enforcement for discriminatory credit decisioning; EU AI Act Article 9(7) non-compliance
• Legal: Class action from affected demographic group; damages proportional to harm period × affected population
• Reputational: Media exposure of AI discrimination; long-term trust damage with affected communities
• Financial: Model rollback cost; investigation cost; remediation and affected customer redress

3. Context

When to Apply

• All AI models making consequential decisions affecting individuals (credit, insurance, hiring, healthcare)
• Any model processing data about protected attributes or proxies for protected attributes
• Models where fairness obligations exist under anti-discrimination legislation
• Following deployment of a model that passed pre-deployment fairness assessment (continuous counterpart)

When NOT to Apply

• Models with no consequential impact on individuals (internal operations, no personal data)
• Models deployed in environments without demographic data available (cannot measure what cannot be observed — document as a governance gap)
• Very low volume models (<1,000 inferences/day per segment — insufficient statistical power; use extended window or aggregate with similar models)

Prerequisites

• Pre-deployment fairness assessment (GOV002) establishing baseline thresholds
• Inference logs with sufficient metadata to support fairness computation
• Data governance approval to retain inference logs with demographic proxy data
• Defined protected attributes for each model use case

Industry Applicability

4. Architecture Overview

The Model Bias Detection pipeline is architected as a streaming-to-batch architecture: inference events stream into a log aggregation system, and fairness computations execute on windowed batches with configurable frequency. This approach balances real-time visibility with statistical validity—fairness metrics require sufficient sample size for statistical significance, which streaming-per-event cannot provide.

Three-Metric Framework. The pipeline computes three distinct fairness metrics because each captures a different dimension of bias. Using only one metric creates a false sense of assurance—a model can be perfectly fair on demographic parity while systematically disadvantaging a group on equalised odds.

Demographic Parity (also called statistical parity): the ratio of positive prediction rates across demographic groups. A credit model with demographic parity ratio of 0.6 for women vs men means women receive 40% fewer approvals than men—potentially discriminatory depending on legitimate factors. Threshold: 0.8–1.25 is commonly accepted as the "80% rule" from US employment discrimination law.

Equalised Odds: the difference in true positive rate (sensitivity) and false positive rate across groups. A healthcare model with poor equalised odds may correctly identify high-risk patients in the majority population but miss the same proportion in minority populations—systematically underserving them. Threshold: equalised odds difference <0.05 for high-stakes medical decisions.

Calibration: whether confidence scores mean the same thing across groups. A model with 80% confidence on a prediction should be correct 80% of the time, equally across all demographic groups. Poor calibration means the model is systematically over- or under-confident for specific groups—dangerous when confidence scores drive downstream decisions (loan approval thresholds, treatment triage).

Window Strategy. Fairness metrics are computed over rolling windows (24-hour, 7-day, 30-day) with different alert thresholds per window. The 24-hour window detects sudden bias shifts (e.g., from a model update or data feed change). The 7-day window reduces statistical noise for day-of-week effects. The 30-day window provides the trend baseline aligned to pre-deployment assessment. This multi-window approach distinguishes transient anomalies from systemic drift.

Protected Attribute Handling. Computing fairness metrics requires demographic attribute data that may be sensitive. The pipeline implements a privacy-preserving approach: demographic attributes are stored in a separate, access-controlled attribute vault, joined to inference logs only within the fairness computation environment, and purged from the computation result before results are distributed to dashboards. Computation results contain only aggregate statistics, never individual-level demographic associations.

Graduated Response Architecture. The pattern implements four response tiers based on finding severity:

• Alert: metrics breach threshold; notifications sent to RAI Officer and model owner; no model impact
• Monitor-Enhanced: persistent breach; monitoring frequency increased; business owner informed
• Restrict: sustained high-severity breach; model scope restricted to reduce exposure while investigation proceeds (e.g., block high-consequence decision types)
• Escalate: Critical severity or failure to remediate within SLA; AI Ethics Review Board convened; GOV008 incident created

Feedback Loop Detection. A second-order monitoring capability detects whether model outputs are feeding back into training data in a way that amplifies existing bias (the "feedback loop" problem). This detects when historical model decisions are incorporated into training datasets for model updates, potentially encoding and amplifying historical discrimination. Detection is based on training data provenance analysis at retraining time.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Bias Detection Flow

Error Flow

8. Security Considerations

Protected Attribute Data Protection

• Demographic attributes stored encrypted at rest in Attribute Vault; AES-256
• Vault access restricted to fairness computation service account; no human direct access without explicit approval
• Demographic data never included in alert notifications or dashboard visualisations (aggregates only)
• Retention: inference logs with demographic join purged after fairness computation; aggregates retained per regulatory schedule

Auditability

• All fairness computation runs logged with input data range, sample sizes, metric outputs, threshold comparison
• Threshold change audit: any change to fairness thresholds requires GOV002 re-assessment reference

OWASP LLM Top 10 Mapping

9. Governance Considerations

Threshold Governance

Fairness thresholds are owned by the AI Governance function, not ML engineering. Changes require Compliance sign-off. Threshold provenance is stored with GOV002 assessment reference. Threshold relaxation requires documented justification.

Governance Artefacts

10. Operational Considerations

SLOs

11. Cost Considerations

Indicative Cost Range

12. Trade-Off Analysis

Option Comparison

13. Failure Modes

14. Regulatory Considerations

EU AI Act

• Article 9(7): High-risk AI systems must be regularly tested to ensure compliance with requirements throughout lifecycle. This pattern implements that continuous testing.

NIST AI RMF

• MEASURE 2.5: AI system fairness and bias is evaluated on a regular basis. Pipeline implements quantitative measurement at configurable frequency.

Australian Anti-Discrimination Law

• Age Discrimination Act 2004, Disability Discrimination Act 1992, Racial Discrimination Act 1975, Sex Discrimination Act 1984: All prohibit algorithmic discrimination on relevant attributes. Monitoring demographic parity provides the detection mechanism.

ASIC RG 271

• Responsible lending obligations: AI-driven credit assessments must not produce discriminatory outcomes. Demographic parity monitoring for credit models directly supports compliance.

15. Reference Implementations

AWS

Azure

Open Source

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven (Level 3)

18. Revision History