--- title: "Production Lessons from Agent Builds" newsletter: "MLOps Community" date: 2025-07-08 source: https://aaif.live/newsletters/mlopscommunity/2025-07-08-production-lessons-from-agent-builds --- # Production Lessons from Agent Builds *What’s working, what’s breaking, and what we’ll learn – a preview of Agents in Production Part 2* *MLOps Community — Agentic AI Foundation, 2025-07-08* https://go.mlops.community/imagejul8 Another example of the damage bad data can do. Show you're part of the solution - our “AI Training Data [https://go.mlops.community/TrainingTJul8]” tee is 30% off. AGENTS IN PRODUCTION PART 2: CONQUERING REAL-WORLD AI AGENT CHALLENGES WE’VE SEEN IT FIRST-HAND: YOU TWEAK A PROMPT, SWAP IN A NEW MODEL, ADJUST A RETRIEVER – AND SOMETHING ELSE BREAKS. RUNNING AI AGENTS IN PRODUCTION IS HARD. MEMORY MODULES FORGET OR GET BLOATED. EVAL IS MURKY WHEN SUCCESS ISN’T A SINGLE NUMBER. SYSTEMS THAT LOOKED GREAT IN DEMO COLLAPSE UNDER EDGE CASES OR REAL USER TRAFFIC. COSTS SPIRAL WITH EVERY API CALL OR GPU CYCLE. THAT’S THE STUFF WE’RE TACKLING AT AGENTS IN PRODUCTION PART 2 [https://go.mlops.community/aip2jul8] - OUR VIRTUAL CONFERENCE ON JULY 17, 5:30–11:00 PM CEST / 8:30 AM–2:00 PM PT. WHAT TO EXPECT AT PART 2 – TACKLING HARD PROBLEMS HEAD-ON PART 2 IS PACKED WITH SESSIONS THAT ADDRESS THE PAIN POINTS OF DEPLOYING AGENTS. HERE’S A PREVIEW OF SOME KEY SESSIONS AND WHAT YOU CAN LEARN FROM THEM: FROM GUESSWORK TO GREATNESS: SYSTEMATIC AI AGENT OPTIMIZATION Nimrod Busany (Founder, Traigent) How can you tune an agent’s prompts, models, and parameters without endless trial-and-error? This talk shows a structured approach to evaluating and optimizing agents across multiple objectives. Expect to learn how multi-dimensional testing (think accuracy and latency and cost) can yield 4–7× quality boosts and huge cost savings by intelligently exploring hundreds of configurations instead of guesswork. Underwriting Assist: A Multi‑Agent System in Production Somya Rai (Principal AI Engineer, EXL) A real case study of an insurance underwriting assistant that uses multiple agents (built with LangChain and Ray) to speed up underwriting by 3× and cut errors 40%. Somya will walk through the system’s architecture - how it uses RAG and a shared memory to let agents analyze policy documents, profile risk, and draft rationale with human-level accuracy. You’ll hear about the guardrails and evaluations they put in place (for compliance and correctness), and how they achieved tangible business impact. It’s a rare peek into an agent system that’s actually running in a high-stakes industry. Cost Optimization for Multi-Agent Systems Mohamed Rashad (DevisionX) Every extra model call and backend process has a dollar cost - and it adds up fast. Mohamed confronts the stark economics of large-scale AI agents: how infrastructure and inference costs can break the bank if unchecked. He’ll share strategies to rein in expenses without sacrificing performance, from clever cloud/on-prem hybrid deployments to fine-grained cost tracking tools. If you’re responsible for your team’s cloud bill (or your startup’s runway), don’t miss the hard-won tactics for keeping GPU budgets under control. How to Build Execution Layers That Don’t Burn Out Tanmay Tiwari (Rivian) It’s easy to build an agent that works once; it’s much harder to build one that runs reliably at scale. Tanmay shares how he designed an execution framework handling thousands of daily operations without melting down. In under 10 minutes, he breaks down how to keep autonomous workflows on track when no one’s watching - designing for consistent behavior, self-correcting task flows, sane memory use, and graceful error handling. If you worry about your agents drifting off or crashing overnight, this session offers battle-tested tips for resilience. The Future of Compute: How AI Agents Are Reshaping Infrastructure Diego Oppenheimer (Hyperparam) Traditional cloud infrastructure wasn’t built for AI agents’ bursty, stateful workloads. Diego’s keynote cuts through buzzwords to address why scaling agents is straining current compute paradigms (VMs, containers, even serverless) and piling up technical debt. He’ll candidly explore whether we can stretch existing infrastructure or need a ground-up rethink for the AI era. If you’re wrestling with deployment headaches - state management, unpredictable scaling, monitoring blind spots - this talk will validate those pain points and outline bold ideas (and hard questions) about the road forward. These are just a few highlights – the full lineup dives into security (e.g. applying “zero trust” principles to agent collaboration), continuous learning (building self-improving agents that get better with each interaction), memory architectures (organizing knowledge so agents don’t forget key info), and much more. Every session is geared towards practical insights: you’ll hear how practitioners are building and breaking things, not abstract theory. Level-Up Before Part 2 To help you get up to speed (or refresh your knowledge) before the event, here are a few recommended talks from Agents in Production Part 1 and Agent Hour sessions. These quick summaries highlight foundational lessons that tie into Part 2’s themes - consider them your pre-game prep: Generative AI Agents in Production: Best Practices and Lessons Learned [https://go.mlops.community/PatrickJul8] Patrick Marlow (from Agents in Production Part 1) Patrick shared practical lessons from deploying generative AI agents in real-world systems, focusing on what tends to go wrong and how to design for stability. He covered prompt refinement through metaprompting, implementing safety measures beyond basic prompt injection checks, and setting up evaluation workflows to catch regressions early. Grounded in hands-on experience, this talk offered clear guidance on moving from prototypes to production. Best Practices from a Live European AI Agent in Logistics [https://go.mlops.community/VanessaJul8] Vanessa Escande (from an Agent Hour session) Rather than theory or roadmap talk, this session focused on an agent already in production, handling logistics claims across Europe. Vanessa explained how they designed it to be deterministic, using structured templates and sandboxed reasoning to avoid hallucinations. The agent integrates into existing workflows, processes both structured and unstructured data, and flags edge cases to a human team. It’s a strong example of how constraint-driven design, confidence thresholds, and orchestration can deliver real autonomy without giving up reliability. Building Reliable Agents [https://go.mlops.community/EnoJul8] Eno Reyes (from Agents in Production Part 1) Reliable agents aren’t just about better prompts or smarter models - they need structure. Eno walked through the systems-level thinking behind building agent workflows that don’t drift, stall, or silently fail. He covered planning techniques inspired by robotics, like Kalman filters and model predictive control, along with consensus mechanisms and simulation-based decision making. The talk also dug into tool design, feedback processing, and where to involve humans for fallback. It's packed with practical strategies for making agents more stable in complex, real-world environments. Intelligent Autonomous Multi‑Agent Systems [https://go.mlops.community/NatanJul8] Natan Vidra (from an Agent Hour session) Natan explored how to design collaborative multi-agent systems that go beyond single-use tools and start to resemble full teams. He broke down coordination patterns, monitoring strategies, and the trade-offs involved in orchestrating agents with different roles and capabilities. The talk also covered early ideas for a generalist agent framework that can dynamically spawn and assign agents to tasks using an orchestrator. It’s a thoughtful look at scaling agents from single flows to flexible, modular systems with reusable components. AI AGENT WORLD TOUR: IN-PERSON MEETUPS The Agents in Production virtual event isn’t all we're doing. The AI Agent World Tour is still rolling, with meetups happening around the world. * San Francisco: The Rerun [https://lu.ma/aiagentsummit] - September 4 The spring edition was packed, so we’re doing it in SF again. 30+ booths showing real AI agent systems - memory hacks, evaluation pipelines, routing tricks, failure handling, and more. Science-fair-style setup: walk up, ask questions, learn from what people have actually built. No marketers. * New York City [https://lu.ma/agentsummitNYC] - October 16 Same vibe, different coast. Dozens of teams demoing working agents and infra. Not polished, not theoretical - just engineers talking through what’s live, what’s failed, and what they’ve learned. These are just two stops. We’re also hitting: * Amsterdam [https://lu.ma/m7m5847s] – July 9 * Munich [https://lu.ma/xh0natks] – July 16 * Atlanta [https://lu.ma/xqbtaqpy] – July 17 * Seattle [https://lu.ma/g42ppkok] – August 14 * Lisbon [https://lu.ma/evk7oocj] – August 14 * Delhi [https://lu.ma/vdwc1ia7] – August 23 * Barcelona [https://lu.ma/xlazzzyw] – September 18 * Berlin [https://lu.ma/ggedfeyq] – September 18 * Stockholm [https://lu.ma/7jbphgzz] – September 25 * London [https://lu.ma/i51hrr7n] – November 6 Stay up to date with our full calendar [https://lu.ma/user/usr-5NONIt93S6laFHY]. Join Us Live on July 17 The agent space is moving fast, and honestly, it’s hard to keep up. Agents in Production Part 2 [https://go.mlops.community/aip2jul8] is a chance to stop, compare notes, and figure things out with others who are actually building this stuff. Whether you're trying to get agents to act reliably, scale sensibly, or just not torch your cloud budget, this should help. Make sure to join live on July 17 if you can. You’ll get to ask questions directly in the interactive Q&A, enjoy some surprises and snag some swag, and just be part of the real-time conversation. Look forward to seeing you there and hearing your questions and thoughts. [Agents in Production Part 2](https://go.mlops.community/aip2jul8) Working on something tricky or planning ahead? Here’s how we can help - just hit reply: * Custom workshops tailored to your company’s needs * Hiring? I know some quality folks looking for a new adventure * Want to connect with someone tackling similar problems? I can introduce you Thanks for reading, catch you next time! Interested in partnering with us? Get in touch: partners@mlops.community Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin]. Shoutout to Rohit Ghumare [https://www.linkedin.com/in/rohit-ghumare/] for his contributions. --- Source: https://aaif.live/newsletters/mlopscommunity/2025-07-08-production-lessons-from-agent-builds