--- title: "Production Playbook: Safe Deployment" newsletter: "MLOps Community" date: 2025-05-13 source: https://aaif.live/newsletters/mlopscommunity/2025-05-13-production-playbook-safe-deployment --- # Production Playbook: Safe Deployment *MLOps Community — Agentic AI Foundation, 2025-05-13* They say giving is better than getting - why not do both? It'd be great if you could give 30 minutes to help community member Michael Weber [https://go.mlops.community/3j9wjr] with his PhD research. He's studying how and why MLOps setups differ across projects - think XGBoost vs agent-based systems, classical pipelines vs bleeding-edge experimentation. Book a call [https://go.mlops.community/tb0o0p] or reach out via micweber@stanford.edu On May 21, get two tracks, top-tier speakers, and sessions on lakehouses, data engineering, streaming, and analytics infra at OpenXData [https://www.openxdata.ai/?utm_source=mlopscommunity&utm_medium=email&utm_campaign=2025_05_openxdata&utm_content=20250513_newsletter]. ## Safe Deployment Strategies for ML Models: Canary, Shadow, and Blue-Green Rolling out a new model is rarely just a code push. Even if you've validated everything offline, it’s the real-world data — messy, spiky, unpredictable — that decides whether things hold up. Most production failures don’t come from your training code. They come from pushing something that quietly breaks under load or returns garbage predictions to real users. This piece walks through three deployment patterns that give you breathing room: canary, shadow, and blue-green. Each one helps reduce risk when rolling out a new ML model, and lets you spot problems early before they hit everyone at once. 1. Canary Deployments Canarying means rolling out the new model to a small slice of live traffic while the old one continues to serve most users. It’s a controlled test in production: 5% of requests hit the new model, 95% go to the current one. If nothing breaks, you gradually increase traffic to the new version. If something does break, you stop. Kubernetes + Istio example: yaml apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: mymodel-service spec: hosts: - mymodel-service http: - route: - destination: host: mymodel-service subset: v1 weight: 90 - destination: host: mymodel-service subset: v2 weight: 10 This keeps both models live and splits traffic based on the weights. SageMaker example: json "ProductionVariants": [ { "VariantName": "OldModel", "ModelName": "ml-model-v1", "InitialVariantWeight": 0.9 }, { "VariantName": "NewModel", "ModelName": "ml-model-v2", "InitialVariantWeight": 0.1 } ] You adjust these weights over time, moving to 20/80, 50/50, and so on. What to monitor: * Error rates * Latency (especially on cold starts) * Changes in output distribution * Business metrics (if available), like conversions or drop-off Canarying works best when you have solid observability in place. If you’re not logging model responses or surfacing key metrics, a quiet failure in that 10% can slip by. And don’t skip rollback logic. If you’re doing traffic splitting, reverting to 100% on the old model should be one step — ideally automated if metrics spike. 2. Shadow Deployments Shadow deployments send the same live traffic to both models — but only one model’s output is used. The new model’s predictions are logged, never returned to users. It’s like rehearsing a launch with the full load but no public consequences. Why use it? To test a major model change without touching user experience. You can compare predictions offline, monitor performance, and catch surprises — all without risk. Python example: python @app.post("/predict") def predict(request: Request): input_data = request.json() result = model_v1.predict(input_data) # Fire off shadow request asyncio.create_task( requests.post("http://model-v2/predict", json=input_data) ) return result Istio config to mirror traffic: yaml mirror: host: mymodel-service subset: shadow mirrorPercentage: value: 100.0 What to watch: * Prediction drift: is v2 making very different calls? * Latency under load: is it fast enough? * Runtime errors: does the new model choke on weird inputs? Shadowing is useful when: * You’re switching model architectures or frameworks * You’ve retrained on new data with major distribution changes * You're adding new input features that might not always be present Tradeoffs: You’re doubling inference cost per request. And you’ll need infra to line up predictions and analyse the logs. But it’s the safest way to test a new model on real traffic. 3. Blue-Green Deployments This one’s more of a full-swap strategy. You have two environments — Blue (live) and Green (the new version). You deploy the new model to Green, run smoke tests or shadow traffic, and when you’re happy, you switch all traffic from Blue to Green. If things go wrong, you flip back. Kubernetes-style switch: yaml # Before: selector: app: mymodel release: v1 # After: selector: app: mymodel release: v2 On SageMaker: Deploy a new endpoint config, then update the production endpoint to point to it. Rollback is just pointing back to the old config. Good for: * Major changes across the stack (e.g. new model and new serving logic) * Zero-downtime cutovers * Teams with tight rollback SLAs The main tradeoff is cost: you’re running two environments in parallel. You also need to be confident that your tests in Green caught anything that could go wrong in live traffic. Some teams pair this with a brief shadow or canary phase just to be sure. Quick Comparison A Few Notes on Rollback Regardless of strategy, rollback plans should be: * Automated where possible (triggered by metrics or alerts) * Tested regularly (especially if they rely on infra rewiring) * Decoupled from human judgement when time matters Canarying is only safe if you can actually revert fast. Final Thoughts If your current deployment process is "push and pray," even a basic version of these strategies is a step up. You don’t need full-blown platform support or fancy rollout controllers — you just need enough control to limit the blast radius and buy yourself time to react. These aren’t new ideas, but they’re easy to overlook in ML. And unlike most "best practices," they’re simple to test in staging and iterate on in production. Pick one. Try it. Sleep slightly better. Working on something tricky or planning ahead? Here’s how we can help - just hit reply: * Custom workshops tailored to your company’s needs * Hiring? I know some quality folks looking for a new adventure * Want to connect with someone tackling similar problems? I can introduce you Thanks for reading, catch you next time! Interested in partnering with us? Get in touch: partners@mlops.community Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin]. The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/]. Shoutout to Ranuga Disansa [https://go.mlops.community/i83caq] for his contributions. --- Source: https://aaif.live/newsletters/mlopscommunity/2025-05-13-production-playbook-safe-deployment