EAAPL-MDL007 — Model Compression and Optimisation

1. Executive Summary

Model compression reduces the computational footprint of an AI model — memory usage, inference latency, and cost per query — through techniques including quantisation, knowledge distillation, and pruning. For large language models, compression is not optional at enterprise scale: a 70B parameter model in full 16-bit precision requires 140GB of GPU memory; INT8 quantisation reduces this to 70GB with approximately 1% quality loss; INT4 reduces it to 35GB with 3–5% quality loss. The right compression strategy can reduce inference costs by 50–80% while maintaining quality within acceptable business thresholds. For CIOs, compression is a cost governance tool: without it, LLM serving costs scale prohibitively. For CTOs, compression is an inference engineering discipline — each technique involves trade-offs that must be evaluated against the specific quality requirements of the application. For risk officers, compression introduces a material model risk: a compressed model is a different model from its uncompressed predecessor. If the original model was validated for regulatory purposes (EU AI Act high-risk, APRA model risk), the compressed version requires re-validation. This pattern provides a benchmarking protocol and re-validation framework to address that requirement.

2. Problem Statement

2.1 Business Problem

LLM inference costs are the primary cost driver for AI applications at scale. A large language model generating 1,000 tokens costs $0.001–$0.03 per query depending on the model. At 10M queries/day, that is $10,000–$300,000 per day. Without compression, organisations either accept these costs, rate-limit usage below business requirements, or limit deployment to use cases where the cost-value ratio is clearly positive. Compression unlocks broader deployment by reducing per-query costs to a level where a wider range of use cases are economically viable.

2.2 Technical Problem

Foundation models are trained in high precision (BF16 or FP32) for numerical stability during gradient descent. Inference does not require the same precision. The excess precision represents wasted memory bandwidth and compute cycles. However, naively reducing precision degrades model quality — the compression must be calibrated against representative input data to minimise quality loss for the specific application's input distribution.

2.3 Symptoms

• Model serving infrastructure is GPU-constrained by model memory requirements, not compute throughput.
• Inference cost is the primary objection to broader AI deployment within the organisation.
• Model responses have acceptable latency at low load but degrade significantly under peak load due to memory bandwidth limitations.
• The organisation is paying for a larger GPU tier than is needed if the model were compressed.

2.4 Cost of Inaction

3. Context

3.1 When to Apply

• Models serving high-query-volume applications where inference cost is a primary concern.
• Models that require deployment on memory-constrained infrastructure (edge devices, smaller cloud instances).
• After fine-tuning (EAAPL-MDL006) — the fine-tuned model is the compression target.
• When latency reduction is required and the bottleneck is memory bandwidth rather than compute.

3.2 When NOT to Apply

• Safety-critical outputs: Models making high-stakes decisions (medical diagnosis, credit decisions, safety systems) where even 1% quality degradation is unacceptable — evaluate compression benefits against quality risk carefully.
• High-precision numerical tasks: Models producing numerical outputs (financial calculations, measurements) where precision matters — quantisation error accumulates in numerical computations.
• Models that are already at acceptable cost and latency targets — do not compress without clear business justification.
• Models subject to active regulatory examination where re-validation cost exceeds compression savings.

3.3 Prerequisites

3.4 Industry Applicability

4. Architecture Overview

4.1 Quantisation

Quantisation converts model weights from high-precision floating point (BF16/FP32) to lower-precision integer or reduced-precision float representations.

INT8 (Post-Training Quantisation): Reduces each weight from 16-bit to 8-bit integer. Memory reduction: 2×. Quality loss: approximately 0.5–1% on typical LLM benchmarks. Inference speedup: 1.5–2× on hardware with INT8 acceleration (NVIDIA A100, H100, consumer GPU). Suitable for most production LLM deployments where minor quality loss is acceptable. Calibration requires a representative sample of 512–2,048 inputs.

INT4 / GPTQ / AWQ: Reduces weights to 4-bit. Memory reduction: 4× vs FP16. Quality loss: 2–5% depending on model size (larger models tolerate quantisation better). Inference speedup: 2–4× on compatible hardware. The AutoGPTQ and AutoAWQ libraries implement calibrated 4-bit quantisation that minimises perplexity increase. Use for deployments where memory constraint is the primary driver and 2–5% quality loss is acceptable.

When NOT to quantise: As stated, avoid INT4 for safety-critical, high-precision numerical, or regulated decisions where re-validation cost or quality risk is unacceptable. Always measure quality on the full evaluation suite — do not rely on perplexity as a proxy for task-specific quality.

4.2 Knowledge Distillation

Knowledge distillation trains a smaller "student" model to mimic the behaviour of a larger "teacher" model. The student is trained on the teacher's output distribution (soft targets) rather than hard labels. This transfers the teacher's generalisation to a smaller architecture.

Architecture requirement: The student architecture is defined before distillation begins. The student is typically 10–30% the size of the teacher (e.g., a 7B student from a 70B teacher). The student must have sufficient capacity to capture the knowledge — a student that is too small cannot learn the teacher's full capability regardless of training budget.

Dataset requirement: Distillation requires a large, diverse dataset — typically the original training data or a similarly broad corpus. The teacher generates soft probability distributions over vocabulary for each training example; the student is trained to match these distributions. Dataset size requirements are similar to the original training run.

Quality ceiling: Knowledge distillation has a quality ceiling: the student cannot exceed the teacher's quality. Empirically, a well-distilled student achieves 90–95% of the teacher's task performance at 20–30% of the inference cost. For tasks where the teacher's quality exceeds the requirement, this is highly cost-efficient.

Use cases: Distillation is the right approach when the organisation needs a model that is dramatically smaller (not just 2–4× reduction from quantisation) and can invest in the training cost of the distillation run.

4.3 Pruning

Pruning removes weights or neurons that contribute least to model outputs, reducing model size.

Structured pruning removes entire neurons, attention heads, or layers — the resulting model has fewer parameters and can be directly accelerated by standard hardware without special kernels. Quality loss is higher than unstructured pruning for the same size reduction.

Unstructured pruning removes individual weights based on magnitude or gradient importance — the resulting sparse weight tensor requires sparse matrix multiplication support for inference speedup. On standard GPU hardware without sparse acceleration, unstructured pruning reduces model size but does not necessarily reduce inference latency.

Industry evidence for LLM pruning: Research and production evidence (Meta, Google, Microsoft) indicates that LLMs tolerate moderate pruning (up to 20% of parameters) with < 2% quality loss on most benchmarks. Beyond 30% pruning, quality loss becomes significant and recovery via fine-tuning is required. Pruning is less mature than quantisation as a production technique for LLMs as of 2026 — prefer quantisation for production deployments unless there is a specific architectural reason for pruning.

4.4 ONNX Export for Inference Optimisation

ONNX (Open Neural Network Exchange) is a cross-platform model representation that enables deployment with optimised runtimes (ONNX Runtime, TensorRT, OpenVINO) that provide hardware-specific inference acceleration independent of the training framework.

Cross-platform benefit: An ONNX-exported model can run on CPUs, GPUs, and specialised hardware accelerators without rewriting serving code. This is particularly valuable for hybrid infrastructure (some workloads on GPU, some on CPU).

TensorRT acceleration: NVIDIA TensorRT converts ONNX models to optimised inference engines for NVIDIA GPUs. For transformer models, TensorRT provides 2–4× latency reduction vs PyTorch inference on the same hardware, with equivalent quality.

Limitations: Not all model operations are ONNX-exportable without custom operators. Verify the full model graph exports correctly before relying on ONNX in production. ONNX export must be validated against the quality threshold — graph-level optimisations can occasionally affect numerical precision.

4.5 Benchmarking Protocol Before and After Compression

The benchmarking protocol must satisfy one requirement: statistical confidence that quality degradation is within the pre-agreed threshold. The protocol:

1. Run the full evaluation suite on the baseline model, recording all metric values.
2. Apply the compression technique to produce the compressed model artefact.
3. Run the full evaluation suite on the compressed model.
4. Compute the quality degradation for each metric: (baseline_score - compressed_score) / baseline_score.
5. Apply a statistical significance test (bootstrap confidence interval on evaluation metrics).
6. Compare degradation to the pre-agreed threshold.
7. If all metrics are within threshold: record evaluation results and proceed to PATCH version registration.
8. If any metric exceeds threshold: reject compression; document result; explore alternative technique or adjust compression parameters.

The evaluation must use the full evaluation suite, not a sample. Compression techniques can cause disproportionate degradation on rare input types or long-tail tasks that are underrepresented in small sample evaluations.

4.6 Compliance Consideration for Regulated Models

If the original model was validated under EU AI Act, APRA CPS 234 model risk management, or any other regulatory framework, the compressed model requires re-validation. Compression changes the model's weights and internal representations — regulators treat the compressed model as a distinct model for validation purposes. The re-validation must demonstrate that the compressed model's regulatory-relevant performance (accuracy, fairness, explainability) has not been materially degraded by compression. The re-validation evidence must be attached to the compressed model's PATCH version artefact bundle.

5. Architecture Diagram

6. Components

7. Data Flow

7.1 Primary Flow

7.2 Error Flow

8. Security Considerations

8.1 Controls Summary

8.2 OWASP LLM Top 10 Relevance

9. Governance Considerations

9.1 Responsible AI

Compression can disproportionately impact model quality for minority subgroups — a model that performs similarly overall may have larger quality degradation for underrepresented groups. Fairness evaluation after compression is mandatory: compare subgroup performance metrics (where applicable) between baseline and compressed model. Any subgroup quality degradation exceeding the threshold for any subgroup is a blocking condition.

9.2 Model Risk Management

A compressed model is a PATCH version change per EAAPL-MDL001. For models registered in the MRM framework, the PATCH version requires a model risk validation event demonstrating that the compression has not materially changed risk-relevant model behaviour. This is a lighter-weight validation than a MINOR version but is not zero.

9.3 Human Approval Gates

Compression results (benchmark degradation report) must be reviewed and approved by the model owner before the PATCH version is registered. For regulated models, the AI Governance function reviews the re-validation evidence. No compression technique may be applied silently — every compression is a versioned, approved change.

9.4 Governance Artefacts

10. Operational Considerations

10.1 SLOs

10.2 Monitoring and Logging

Post-deployment monitoring for compressed models mirrors production monitoring with one addition: a monthly quality drift check. If the quality metric for the compressed model drifts over time at a rate faster than the baseline model, this may indicate that the compression has created sensitivity to data distribution shift that was not present in the original model.

10.3 Incident Response

If a compressed model shows unexpected quality degradation in production (post-deployment), the standard rollback procedure (EAAPL-MDL004) applies: rollback to the last-known-good version (the uncompressed or previous PATCH). The incident triggers a review of the compression benchmark — was the quality degradation visible in the evaluation and accepted, or was it not detected?

10.4 Disaster Recovery

10.5 Capacity Planning

Compression reduces memory requirements — this enables deployment on smaller (cheaper) GPU instances or increases the number of models per GPU instance. After compression, re-evaluate the serving infrastructure sizing: INT8 compression enabling deployment from A100-80GB to A100-40GB halves the GPU memory cost. Model this before procurement to capture the cost saving.

11. Cost Considerations

11.1 Cost Drivers

11.2 Scaling Risks

Distillation training cost scales with student model size and training dataset size — cost is comparable to fine-tuning. For organisations doing distillation of very large teachers (70B+), budget accordingly. Quantisation compute is much lower (typically < 1 hour for INT8 calibration of a 7B model).

11.3 Optimisations

• Start with INT8 post-training quantisation: highest quality-to-effort ratio; no training required.
• Use AutoAWQ or AutoGPTQ for INT4 before considering distillation — they achieve close to distillation quality with dramatically less compute.
• Profile the actual memory and latency bottleneck before choosing a technique: if the bottleneck is compute (not memory bandwidth), quantisation may have limited benefit.

11.4 Indicative Cost Range and Savings

12. Trade-Off Analysis

12.1 Compression Technique Comparison

12.2 Architectural Tensions

13. Failure Modes

13.1 Cascading Failure Scenarios

If a compressed model is deployed without adequate quality regression testing and serves as the base for a subsequent fine-tuning run (EAAPL-MDL006), the fine-tuned model inherits the quality degradation of the compressed base. Mitigation: fine-tuning pipelines must use uncompressed (full precision) base models; compression is always a post-training PATCH applied to the final model, not a pre-training base model step.

14. Regulatory Considerations

15. Reference Implementations

15.1 AWS

• Quantisation: AWS SageMaker Neo (hardware-aware optimisation); bitsandbytes on SageMaker Processing Job; AutoGPTQ on GPU instance.
• Distillation: SageMaker Training Jobs (same pipeline as fine-tuning, student architecture).
• ONNX Export: SageMaker Processing Job with PyTorch ONNX export; Triton Inference Server on SageMaker for TensorRT.
• Evaluation: SageMaker Processing Jobs for benchmark runs.
• Artefact Storage: S3 (PATCH version bundle).

15.2 Azure

• Quantisation: Azure ML with bitsandbytes/AutoAWQ compute job; Azure Neural Network Intelligence for hardware-aware compression.
• Distillation: Azure ML Training (same pipeline as fine-tuning).
• ONNX Export: Azure ML with Optimum (Hugging Face); ONNX Runtime inference on Azure.
• Evaluation: Azure ML Pipelines for benchmark runs.
• Artefact Storage: Azure Blob Storage (PATCH version bundle).

15.3 GCP

• Quantisation: Vertex AI Custom Training with bitsandbytes/AutoAWQ; GCP Model Garden compression tools.
• Distillation: Vertex AI Training (student model).
• ONNX Export: Vertex AI Custom Training with Optimum; TensorRT on T4/A100 VMs.
• Evaluation: Vertex AI Pipelines.
• Artefact Storage: GCS (PATCH version bundle).

15.4 On-Premises / Hybrid

• Quantisation: bitsandbytes, AutoGPTQ, AutoAWQ on on-premises GPU; llama.cpp for GGUF quantisation on CPU-capable inference.
• Distillation: Standard PyTorch training on GPU cluster (same as fine-tuning pipeline).
• ONNX Export: torch.onnx.export + Optimum; TensorRT for NVIDIA GPU optimisation.
• Evaluation: Eleuther LM Eval Harness on self-hosted compute.
• Artefact Storage: MinIO S3-compatible storage.

16. Related Patterns

17. Maturity Assessment

Overall Maturity: Proven

18. Revision History