---
title: "Production Playbook: Continuous Monitoring and Drift Detection for Reliable ML in Production"
newsletter: "MLOps Community"
date: 2025-04-15
source: https://aaif.live/newsletters/mlopscommunity/2025-04-15-production-playbook-continuous-monitoring-and-drift-detectio
---

# Production Playbook: Continuous Monitoring and Drift Detection for Reliable ML in Production

*MLOps Community — Agentic AI Foundation, 2025-04-15*

Looking forward to the lyric sheet for my favorite meditation and spa bangers when the whale version of this [https://go.mlops.community/cy2teu] drops.

## What Should You Be Watching?

PRODUCTION PLAYBOOK: CONTINUOUS MONITORING AND DRIFT DETECTION FOR RELIABLE ML IN PRODUCTION

You trained the model. You deployed the model. But in production, things shift.

User behavior changes, data pipelines mutate, and what worked well last week might quietly degrade today. That’s why continuous monitoring and drift detection are essential. If you want your models to stay useful, you need to know when the ground has shifted underneath them.

Here’s how to set up practical, effective monitoring that catches problems early.

What Should You Be Watching?

Start by defining what “good” looks like. Useful signals to track:

 * Model performance: Accuracy, precision, recall, F1, AUC – ideally sliced by segments
 * Prediction distribution: Output skews, class imbalance, unexpected values
 * Feature drift: Changes in input feature stats compared to training baseline
 * Concept drift: Shifts in the relationship between input and target


TECHNIQUES FOR DETECTING DRIFT

1. Spotting Feature Drift

Feature drift is usually the canary in the coal mine. Two common ways to track it:

 * Sliding windows: Compare recent production data against a fixed or rolling baseline (e.g. 7 days vs training set)
 * Statistical tests: Compare feature distributions over time. The Kolmogorov-Smirnov (K-S) test is a simple way to compare distributions. Below, is an example to check whether a single feature has drifted in production compared to training

import scipy.stats as stats ks_stat, p_value = stats.ks_2samp(training_data['feature'], production_data['feature']) if p_value < 0.05: print("Feature drift detected!")

2. Tracking Model Performance

If you have labels (even delayed), calculate metrics continuously:

from sklearn.metrics import accuracy_score accuracy = accuracy_score(y_true, y_pred) if accuracy < 0.8: print("Performance alert: accuracy dropped")

If you don’t have labels in real time:

 * Watch model confidence scores or output entropy
 * Track downstream business metrics as proxies

3. Detecting Concept Drift

Concept drift is harder – it shows up when the world changes, not just the data.

Some useful signals:

 * Prediction variance: Is the model’s output less stable across similar inputs?
 * Segment monitoring: Accuracy by region, user group, or time window
 * Retrain triggers: If confidence drops or business KPIs fall, schedule a retrain

Building a Monitoring Workflow

Automate checks into a regular cadence. A simple pipeline might look like:

monitoring_pipeline: schedule: daily tasks: - fetch_recent_production_data - calculate_feature_and_performance_metrics - send_alerts_if_metrics_breach_thresholds - retrain_model_if_needed

This can run on whatever orchestration you already use – Airflow, Dagster, a lightweight CI runner.

Useful Tools (if you don’t want to build it all yourself)

 * Evidently – Drop-in drift detection with built-in statistical tests and visual dashboards. Can be scheduled in pipelines or run in real time, and integrates easily with Jupyter or batch workflows.
 * Prometheus + Grafana – Ideal for surfacing custom metrics from model serving endpoints, drift detectors, or pipeline systems. Grafana dashboards make it easy to set up alerts when thresholds are crossed.
 * MLflow – Track experiments, model versions, performance metrics, and deployment stages. Includes a model registry and supports integration with CI/CD workflows.
 * Kubeflow – A full MLOps platform with pipelines, experiment tracking, and serving. Useful if you're already running on Kubernetes and want tighter infra integration.
 * WhyLabs + whylogs – Instrument your models and data pipelines with lightweight logging. Whylogs generates statistical profiles of your data, and WhyLabs provides cloud-based drift and anomaly monitoring.
 * NannyML – Focuses on post-deployment model monitoring, especially for cases where you don’t have access to ground truth labels immediately. Handles confidence-based performance estimation and data drift detection.

A Few Good Habits

 * Baseline wisely: Use a stable, representative slice of data
 * Avoid alert fatigue: Tune thresholds so you don’t get false alarms
 * Mix automation and oversight: Retrain automatically if needed, but keep a human in the loop for promotion

Final Thought

Drift happens. But with the right monitoring, you’ll catch it before it becomes a production incident. A few checks, a few alerts, and your models stay useful longer. That’s the goal.

## “Your First MLOps Stack”

THINKING OF BUILDING YOUR OWN STACK?

If the tools in this piece felt familiar – or maybe just out of reach – our course “Your First MLOps Stack”, by Stefano Bosisio might be worth a look.

It walks through data processing (Apache Beam + Dataflow), model training (Kubeflow + VertexAI), and model tracking (MLflow), with a cloud-native focus throughout. 51 lectures, practical examples, and theory blended in.

It’s currently half price at $249, and we've got deals on our other courses too.

Check out the course → [https://go.mlops.community/FirstStackCourse]

## HERE TO HELP

Before you go, here are three ways I can help - just hit reply:

 * Curated intros to other community members
 * What problems are you dealing with? Let me help you find the best solutions through my network
 * Looking to augment your staff for an MLOps or AI project? I got you covered

Thanks for reading, catch you next time!

Interested in partnering with us? Get in touch: partners@mlops.community

Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin].

The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/].

Shoutout to Ranuga Disansa [https://go.mlops.community/i83caq] for his contributions.

---
Source: https://aaif.live/newsletters/mlopscommunity/2025-04-15-production-playbook-continuous-monitoring-and-drift-detectio