--- title: "Agent of Chaos: Lessons from 20+ Experiments" newsletter: "MLOps Community" date: 2025-02-25 source: https://aaif.live/newsletters/mlopscommunity/2025-02-25-agent-of-chaos-lessons-from-20-experiments --- # Agent of Chaos: Lessons from 20+ Experiments *Plus, making ETL painless with Airflow & Composer, and a closer look at BAML in Tuesday Tool Time* *MLOps Community — Agentic AI Foundation, 2025-02-25* https://go.mlops.community/73674y Yeah, it's great that it's going to save lives, but can it fix version conflicts in Python environments? ## The Agent Landscape - Lessons Learned Putting Agents Into Production 2 min read The Agent Landscape - Lessons Learned Putting Agents Into Production Paul van der Boor // Senior Director Data Science @ Prosus Group Floris Fok // AI Engineer at @ Prosus Group When is an AI agent not an AI agent? When it's a search bar. This was one of the painful lessons the Prosus team shared when we talked about what makes AI agents actually useful in production. They've been running agent experiments since 2022, testing over 20 projects. One big challenge is making sure agents aren’t just cool but actually useful. If you can’t measure whether an agent’s working, it’s probably not. The best ones tend to have a few things in common: * A clear “did it work?” test – Coding agents succeed because code either runs or doesn’t. But if an agent’s output is just “vibes” (e.g., summarizing user feedback), it’s much harder to tell if it’s doing a good job. * Doesn’t annoy users – If an agent complicates things, people won’t use it. A good example is search: users want quick answers, not a back-and-forth chat just to find a blue couch. * Worth the cost – Cheaper tokens don’t always mean cheaper systems. If an agent calls an LLM five times to complete a task, that adds up. Instead of looking at cost per token, it makes more sense to check cost per completed task. We also talked about why so many agents fail: noisy outputs, hard-to-define success, and the simple fact that users don’t behave the way you think they will. Here's hoping you behave the way I think you will and click below to listen! Video [https://go.mlops.community/l9rr24] || Spotify [https://go.mlops.community/ezgr54] || Apple [https://go.mlops.community/ie5lu3] ## Bridging the Culinary Gap - Part 2 2 min read Bridging the Culinary Gap - Part 2 With thanks to Jessica Rudd for their contribution. It might not seem it, but working in ML and a professional kitchen have quite a bit in common: a pressure environment, demanding customers, the need for reliable tools... Running with this comparison, this blog looks at how Google Cloud Composer and Apache Airflow take the hassle out of managing ETL pipelines by handling scheduling, monitoring, and dependencies. They make sure each step runs in the right order, prevent failures, and keep things running smoothly. A big win here is dynamic DAGs, which automate Dataform job orchestration without needing to manually define each DAG. This means: * Less repetitive work – DAGs are generated from a config file, so you’re not writing the same boilerplate over and over. * Easier maintenance – All settings live in one place, making tweaks and updates straightforward. * Better scalability – Adding new jobs is as simple as updating the config, no extra code needed. The blog also includes a sample DAG showing how Composer/Airflow structures workflows in a way that’s clean, efficient, and scalable. If you've been swearing at your ETL pipeline more than Gordon Ramsey, might be worth a read! Read it here [https://go.mlops.community/d4cj7k] ## Job of the Week [https://go.mlops.community/fyom3j](https://go.mlops.community/fyom3j) ## Tuesday Tool Time 🔧 1 min read Tuesday Tool Time 🔧 BAML: The Boundary AI Markup Language ## Tuesday Tool Time Welcome to this week’s Tuesday Tool Time - where we explore the tools shaping the MLOps ecosystem and help you decide what’s worth your time. This week, Stephen Oladele [https://go.mlops.community/733wfg] takes a look at BAML [https://github.com/BoundaryML/baml](Boundary AI Markup Language) - a domain-specific language that makes creating AI apps easier by treating LLM prompts as functions. If you’ve been juggling messy prompt engineering workflows or struggling to integrate LLMs into multi-language projects, BAML might be the tool you’ve been waiting for. https://mlops-community.slack.com/archives/C04T55KFV8S/p1740080523705069?thread_ts=1739999400.124009&cid=C04T55KFV8S 👉 See discussion in the community [https://mlops-community.slack.com/archives/C04T55KFV8S/p1740080523705069?thread_ts=1739999400.124009&cid=C04T55KFV8S] 🔧 Tool Overview: BAML in a Nutshell BAML lets developers define prompts with explicit input parameters and expected outputs. It makes prompt engineering feel more like writing reusable code modules than creating one-time strings. The BAML approach brings structure, consistency, and modularity to how you develop AI apps. 💡 Who’s it for? MLOps engineers, data scientists, and AI engineers. Teams looking to integrate LLMs into diverse programming environments (Python, TypeScript, Ruby, etc.). Anyone looking to clean up and scale their prompt engineering workflows. ➡️ Check it out on GitHub [https://github.com/BoundaryML/baml] ⚙️ Setup & Usability: Built for Developers BAML’s setup is straightforward. You can: * Cross-language support: Write AI-specific logic in BAML while keeping the rest of your codebase intact—whether it’s in Python, TypeScript, Ruby, or other languages. * Use the BAML Visual Studio Code (VSCode) extension for: * Real-time prompt previews * Schema validation * Interactive prompt execution * Minimal setup: BAML is super easy to set up, with awesome docs and a helpful community. This makes prompt engineering way easier and quicker, especially for devs who are new to LLM integrations. ➡️ Get started with BAML using the Python Integration [https://docs.boundaryml.com/guide/installation-language/python] 💡 Key Features & Strengths ✅ Language-agnostic integration: You can use LLMs without refactoring your existing codebase because you can call BAML functions from various languages [https://docs.boundaryml.com/guide/installation-language/rest-api-other-languages]. ✅ Structured prompt definition: Define prompts with clear inputs and expected outputs, turning them into predictable, testable components. ✅ Robust tooling support: The VSCode extension has awesome tools to help with writing and testing prompts, with schema-aware editing and real-time execution. ✅ Active community & open source: BAML is open-source (Apache 2.0) with an active community. Get support, contribute, or explore on GitHub [https://github.com/orgs/BoundaryML/discussions] and Discord [https://discord.gg/BTNBeXGuaS]. ⚠️ Limitations & Weaknesses ⚡ Learning curve for new users: I think you may face an initial adjustment period if you're new to domain-specific languages. ⚡ Evolving ecosystem: Obviously, BAML is pretty new and still growing, so there will be frequent updates. This might lead to some breaking changes every now and then, so make sure to stay on top of version updates. 🚀 Real-World Use Case: Building a Tone-Sensitive Chatbot Let’s say you want to build a chatbot that adjusts its tone based on user input. Using BAML, you can define a function that takes user messages and a desired tone as inputs and returns an appropriate response. Example Python Integration (from GitHub): from baml_client import b from baml_client.types import Message, StopTool messages = [Message(role="assistant", content="How can I help?")] while True: print(messages[-1].content) user_reply = input() messages.append(Message(role="user", content=user_reply)) tool = b.ChatAgent(messages, "happy") if isinstance(tool, StopTool): print("Goodbye!") break else: messages.append(Message(role="assistant", content=tool.reply)) 👆 This clean integration keeps your AI logic modular, testable, and maintainable while making it easy to tweak prompts and outputs. ➡️ Explore more examples in the docs [https://docs.boundaryml.com/examples/interactive-examples] 🔄 Comparisons: BAML vs. Traditional Prompt Engineering Traditional Approach: * Embedding prompts directly into code (often as long strings) * No clear structure for inputs/outputs * Messy and hard to maintain BAML Approach: * Prompts as structured functions with defined schemas * Reusable across multiple projects and languages * Clear separation between AI logic and app code Verdict: If you’ve struggled with prompt sprawl in large codebases, BAML gives a cleaner, more scalable solution. Comparison with other frameworks: * BAML vs Pydantic [https://docs.boundaryml.com/guide/comparisons/baml-vs-pydantic] * BAML vs Marvin [https://docs.boundaryml.com/guide/comparisons/baml-vs-marvin] 🧑‍⚖️ Final Verdict: Is BAML Worth It? So I’d say BAML is a pretty cool tool that makes it way easier to work with LLMs by letting you create reusable functions for your prompts. It's super helpful for teams building complex AI systems or scaling LLM-powered apps. There's a bit of a learning curve, but the benefits, like maintainability, modularity, and productivity, make it valuable for any LLMOps toolkit. ✅ Pros: * Structured and maintainable AI code integration * Good cross-language support * Excellent developer experience (VSCode extension) ❌ Cons: * Initial learning curve * Rapid ecosystem changes may require frequent updates 📺BAML: What it is and how it helps [https://youtu.be/S9jxdVLFDJU] 🔗Explore BAML on the docs [https://docs.boundaryml.com/home] 💬 Have You Tried BAML? We’d love to hear from you! Have you integrated BAML into your workflows? Got any tips, tricks, or challenges? Join the conversation in #llmops [https://mlops-community.slack.com/archives/C04T55KFV8S], and let’s geek out over prompt engineering done right! 🚀 ## The Sandbox 1 min read The Sandbox A little place to test some ideas Trying out a few things here - let us know what you think here [https://go.mlops.community/hx6agg] or email steve@mlops.community Back to the Feature A highlight from last week Last week's blog on workflow orchestration was music to people's ears! It explored DAG-based vs. event-driven models, tool comparisons (Airflow, Argo, Kubeflow), and key factors like scalability, automation, and fault tolerance. Read it here [https://go.mlops.community/gceymz] Tech Teaser Quiz A mini MLOps mindbender Question: Whataburger is celebrating its 75th anniversary by displaying digital fan art at the South by Southwest (SXSW) festival in Austin, and Apple has committed $500 billion to the development of a new AI server facility in Houston. How many Double Meat Whataburger Whatameals could they have bought instead? a) About 37 billion b) About 82 billion c) About 63 billion d) About 41 billion Click here [https://go.mlops.community/TechTeaserAnswers] for the answer. Interested in partnering with us? Get in touch: partners@mlops.community Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin]. The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/]. --- Source: https://aaif.live/newsletters/mlopscommunity/2025-02-25-agent-of-chaos-lessons-from-20-experiments