[EAAPL-PLT009] Feature Store Integration

Category: Platform Engineering Sub-category: ML Infrastructure / Data Engineering Version: 1.1 Maturity: Proven Tags: feature-store, feature-serving, online-inference, offline-training, feature-pipeline, point-in-time, feature-monitoring, training-serving-skew Regulatory Relevance: EU AI Act Article 10 (Data Governance), ISO 42001 Clause 6, NIST AI RMF MAP 3.5

1. Executive Summary

Feature stores solve a deceptively simple problem: when an ML model needs a feature during inference, how does it get the right value, freshly computed, at low latency? And when training a new model, how does it get the exact same feature values that would have been available at prediction time in the past—preventing the data leakage that invalidates backtests and production evaluations?

The Feature Store Integration pattern establishes a shared infrastructure layer that decouples feature computation from feature consumption, enabling features to be computed once and reused across models, teams, and use cases. The online store serves low-latency feature retrieval for real-time inference; the offline store enables point-in-time correct training data generation. Feature pipelines manage computation and freshness; feature monitoring detects drift that would degrade model performance before it reaches users. For enterprises with multiple ML models consuming overlapping signals, the feature store is the difference between duplicated, inconsistent feature computation and a shared, governed, quality-assured data layer.

2. Problem Statement

Business Problem

Multiple ML models within the same organisation compute the same features independently, consuming redundant engineering effort and producing inconsistent values (e.g., "30-day spend" computed differently for fraud, recommendation, and credit risk models). Business decisions made on these models are implicitly inconsistent. When models are retrained, the historical features used for training may not match what would have been available at prediction time, leading to overoptimistic evaluation metrics and production performance gaps.

Technical Problem

Online inference requires feature values available in <10ms at the model API boundary; this requires a pre-computed, low-latency store. Training requires point-in-time correct historical feature values to avoid look-ahead bias. Without a feature store, teams either accept this bias or build expensive, fragile point-in-time joins from raw data. Feature pipelines are duplicated across teams with no shared infrastructure.

Symptoms

• Same feature (e.g., customer 30-day transaction count) computed differently in 3 different model codebases
• Production model performance consistently below offline evaluation metrics (training-serving skew)
• Feature pipeline failures causing model inference to serve stale or missing features
• No visibility into when a feature was last updated or what its current distribution is
• Training datasets built from current feature values rather than the values available at the historical prediction time

Cost of Inaction

• Training-serving skew causing production models underperforming by 5–20% vs offline evaluation
• 30–50% of ML engineering time spent on feature engineering that duplicates existing work
• Model regressions caused by undetected feature drift going undetected for weeks
• Regulatory audits unable to reproduce model predictions due to no record of feature values at decision time

3. Context

When to Apply

• Organisation has ≥2 ML models sharing overlapping input features
• Real-time inference latency requirements (<50ms) demand pre-computed feature values
• Training pipelines require point-in-time correct historical data
• Feature reuse across teams is a stated engineering goal
• Model performance monitoring requires feature drift detection

When NOT to Apply

• Single simple model with unique features: feature store overhead not warranted
• LLM-only organisation with no traditional ML models: most LLM use cases don't benefit from traditional feature stores (embeddings have their own infrastructure path)
• Research experiments: use pandas and raw data; migrate to feature store when productionising

Prerequisites

• Operational data sources (databases, event streams) producing features
• Feature computation infrastructure (Spark, Flink, or dbt for offline; streaming processor for online)
• Online store infrastructure (Redis or equivalent <10ms lookup)
• Offline store infrastructure (data warehouse or object storage for point-in-time joins)
• ML model serving infrastructure that can retrieve features at inference time

Industry Applicability

4. Architecture Overview

The feature store architecture is defined by the separation between its online and offline paths, each serving a different consumer with different latency and freshness characteristics.

The Online Store is a low-latency key-value store containing pre-computed feature values, indexed by entity key (e.g., customer_id, product_id, session_id). Lookup latency must be <10ms at P99 to be compatible with real-time inference SLAs. The online store is populated by the feature materialisation pipeline, which computes features from source data and writes them on a schedule (for batch features) or in near-real-time (for streaming features). Redis is the canonical technology for the online store; its GET operation with a compound key (entity_type:entity_id:feature_set) delivers sub-millisecond lookup at scale.

The online store does not store feature history—only the current value for each entity. This makes it fast and cheap. When a model is called for inference, the feature serving layer assembles the feature vector by looking up all required features for the request's entity IDs from the online store, combining them with request-time context (features that cannot be pre-computed because they depend on the current request), and passing the assembled feature vector to the model.

The Offline Store serves training data generation and batch inference. Unlike the online store, the offline store retains historical feature values—specifically, the feature value that was current at any given point in time. This enables point-in-time correct training data generation: given a set of training examples with timestamps, retrieve the feature values that were available just before each timestamp. This prevents look-ahead bias (using future data to predict the past), which is the most common source of training-serving skew. The offline store is implemented as a time-partitioned table in a data warehouse (BigQuery, Redshift, Snowflake) or as Parquet files in object storage, with a time dimension on every feature record.

Feature Pipelines compute and refresh feature values from source data. Batch pipelines run on a schedule (hourly, daily) using Spark or dbt and write to both the offline store (appending the new time-partitioned record) and the online store (overwriting the current value). Streaming pipelines consume event streams (Kafka, Kinesis) and compute features in near-real-time using Flink or Spark Streaming, writing to the online store with low latency. The choice between batch and streaming for a feature depends on its staleness tolerance: fraud detection features require seconds-old values; monthly customer metrics can be daily.

Feature Registry is the metadata layer for the feature store. It records: feature name, description, data type, computation logic (the transformation that produces the feature), data source, update frequency, entity type, business owner, and deprecation status. The feature registry is the discovery mechanism that enables engineers to find existing features before building new ones. It also serves as the configuration source for the feature materialisation pipeline and the feature serving layer.

Feature Monitoring is the operational quality layer. For each feature, monitoring tracks: distribution statistics (mean, std, percentile distribution) on a rolling basis, freshness (time since last update vs. configured threshold), null rate (unexpected nulls indicate pipeline failures), and drift (statistical distance between the current distribution and the training-time distribution, using measures like PSI or Jensen-Shannon divergence). Alerts on feature drift enable proactive model retraining before production performance degrades significantly.

5. Architecture Diagram

6. Components

7. Data Flow

Primary Flow — Real-Time Inference with Feature Store

Error Flow

8. Security Considerations

• Feature data may contain derived personal information (spending patterns, risk scores, health indicators); access to the online store must be restricted to authorised model serving infrastructure
• The offline store contains historical PII-derived features; access requires the same data classification controls as the source data
• Entity keys in the online store must not leak information about underlying entities; compound keys should use opaque IDs (UUIDs), not readable identifiers

OWASP LLM Controls

9. Governance Considerations

Data Governance

• Every feature must have a registered owner responsible for pipeline health and data quality
• Features derived from personal information must document the legal basis and retention policy in the feature registry
• Deprecated features must be retained in the registry with deprecation date and migration guidance; never silently deleted

Model Risk

• Point-in-time join methodology must be validated and documented as part of the model development process; incorrect point-in-time logic is a model risk event
• Feature drift alerts must be routed to the model owner, not just the platform team; the model owner is accountable for model performance

Governance Artefacts

10. Operational Considerations

Monitoring

SLOs

Disaster Recovery

11. Cost Considerations

Cost Drivers

Indicative Cost Range

12. Trade-Off Analysis

Feature Store Architecture Options

Online Store Technology Options

Architectural Tensions

13. Failure Modes

14. Regulatory Considerations

EU AI Act Article 10 (Data Governance)

• Feature computation logic must be documented (in feature registry) as part of the training data governance requirements for high-risk AI systems
• Point-in-time join methodology must be documented to demonstrate absence of data leakage in training data

Privacy Act / GDPR

• PII-derived features (spending patterns, health indicators) must have a documented legal basis in the feature registry
• Data subject deletion requests must propagate to the online store (delete entity's feature values) and be documented in the offline store (mark as deleted rather than hard delete, to preserve training data integrity)

NIST AI RMF MAP 3.5

• Feature monitoring and drift detection implement MAP 3.5's requirement for ongoing monitoring of AI system inputs

15. Reference Implementations

AWS

GCP

On-Premises / Open Source

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven Feature stores are production-proven at major technology and financial services companies. Open-source tooling (Feast) and managed services (Tecton, SageMaker Feature Store) are both mature. Point-in-time joins are well-understood. Feature monitoring is less standardised.

Scoring Matrix

18. Revision History