--- title: "The Good, The Bot, and The Ugly: LLMs in Production" newsletter: "MLOps Community" date: 2025-02-06 source: https://aaif.live/newsletters/mlopscommunity/2025-02-06-the-good-the-bot-and-the-ugly-llms-in-production --- # The Good, The Bot, and The Ugly: LLMs in Production *Plus, is there anybody out there?, a blog that'll be poetry to you, hidden gems, Slack spotlight, and the sandbox.* *MLOps Community — Agentic AI Foundation, 2025-02-06* https://go.mlops.community/tpbxxe After some deep research of our own about how big the newsletter's getting, plus the new things we’ve got planned, we're gearing up to hit your inbox with a combo - two editions a week coming soon! ## Real LLM Success Stories: How They Actually Work 2 min read Real LLM Success Stories: How They Actually Work Alex Strick van Linschoten // ML Engineer @ ZenML Listening to this episode could help you match Klarna's valuation of $14.6 billion. It’s a big claim, but so’s Klarna’s that AI took on the workload of 700 full-time agents. A more realistic claim? You’ll have a clearer picture of real-world LLM use cases after this episode. Alex talked me through how he compiled a database of real-world LLM use cases, inspired by Evidently AI’s ML tracking efforts. He pulled together blog posts, company write-ups, and MLOps Community discussions to map out what’s actually happening in production. Chatbots dominate, but a few teams are pushing further. Some embed LLMs into products to guide users and drive conversions, while others are testing agent workflows - though true automation is rare. The most successful deployments keep agents on a tight leash, limiting them to well-defined, repeatable tasks. A few high-profile cases, like Klarna’s customer support system, claim to have demonstrated financial impact, but broader success remains elusive: * Scope limitations – Open-ended tasks remain unpredictable, so most working agent deployments stick to rigid, constrained processes. * Enterprise caution – Large companies hesitate to hand over too much control, preferring structured workflows with human oversight. * Technical bottlenecks – Debugging is a nightmare. Failures could be bad prompting, broken workflows, or missing constraints - it’s not always clear where things went wrong. The database is open-sourced on Hugging Face, and Alex wants to add tool-based filtering next. Got a cool LLM use case? Send it his way. Have a listen - it might just save you hiring 700 agents. Video [https://go.mlops.community/rrj86o] || Spotify [https://go.mlops.community/50dp4n] || Apple [https://go.mlops.community/onkfu6] ## AI & Aliens: New Eyes on Ancient Questions 2 min read AI & Aliens: New Eyes on Ancient Questions Richard Cloete // Laukien-Oumuamua Postdoctoral Research Fellow @ Harvard University I’m not exaggerating when I say this episode is out of this world! Richard discussed how AI is being used to detect unidentified aerial phenomena (UAPs) through the Galileo Project, which deploys sensor arrays to track objects across different spectra. Training YOLO-based models for object detection in astronomical and aerial research—whether spotting unknown objects in the sky or classifying celestial bodies from telescope data—required a specialized dataset that didn’t exist. To address this, his team created AeroSynth, a synthetic data generator that: * Generates 3D simulations: Using Blender and Python, they created realistic scenes with aircraft, birds, drones, and balloons at different angles, distances, and lighting conditions. * Improves model performance: Synthetic data enabled training on relevant sky-based perspectives, leading to better detection accuracy for real-world observations. His expertise in AI-powered object detection extends to work with the Minor Planet Center, where he's developing models to classify different types of space objects from telescope data, including potential interstellar visitors. Beyond these sky-watching initiatives, he is also working on ocean monitoring with Seeker Robotics, using AI-powered autonomous surface vehicles to track aerial and marine activity. You don’t need a high-powered telescope to see how good this episode is! Video [https://go.mlops.community/720ep4] || Spotify [https://go.mlops.community/iq0df4] || Apple [https://go.mlops.community/01adzi] ## DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning 1 min read DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning Upcoming MLOps Community Reading Group. DeepSeek-R1 talk has been non-stop, so let’s actually figure out what’s worth paying attention to. The MLOps Community Reading Group is tackling DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning [https://go.mlops.community/DeepSeekPaper] to explore its RL approach, reasoning claims, and what it all really means. Join us Feb 13, 11 AM ET. Register here [https://go.mlops.community/DeepSeekReadingGroup] [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://go.mlops.community/DeepSeekPaper) ## Poetry Was Good, Uv Is Better: An MLOps Migration Story 1 min read Poetry Was Good, Uv Is Better: An MLOps Migration Story With thanks to Médéric Hurier for their contribution. I saw the best minds of my generation destroyed by madness, starving, hysterical - waiting for Poetry to install. In an epic worthy of The Iliad, this blog covers the migration from Poetry to Uv for managing Python dependencies in an MLOps project. Uv, built with Rust, offers faster dependency resolution and better adherence to Python standards, making it an attractive alternative. The transition improved CI/CD efficiency, simplified configuration files, and streamlined Docker builds. A major benefit was the cleaner and more efficient dependency management: * Uv removed the need for Poetry-specific configurations in pyproject.toml, ensuring a more standardized setup. * Dependency resolution became significantly faster, reducing installation times in both local development and CI/CD pipelines. * Uv’s strict adherence to Python Enhancement Proposals (PEPs) minimized complexity and improved maintainability. The result was faster installs, a more standardized setup, and better integration with the broader Python ecosystem, making Uv a compelling choice for MLOps teams. Merrily, merrily, merrily, merrily, workflows are a dream. Read it here [https://go.mlops.community/wsn8ln] ## Job of the Week [https://go.mlops.community/MetaJob](https://go.mlops.community/MetaJob) ## Hidden Gems ## Building Steam with a Grain of Salt // // Building Steam with a Grain of Salt // Gem [https://go.mlops.community/az9ja5] // Song [https://www.youtube.com/watch?v=HORLJvUMs08] An in-person meetup in SF on Feb 12 to build your own agentic RAG system using Airbyte’s toolset, showing how to prepare, manipulate, and transform data, and trace and evaluate that system using Arize Phoenix Talk Talk // Gem [https://go.mlops.community/OptimizedAICall] // Song [https://www.youtube.com/watch?v=2IgjUYrDbWI] A call for speakers from the Optimized AI Conference, seeking proposals on LLMs, RAG, AI infrastructure, model optimization, and real-world applications of AI in production. Lessons // Gem [https://go.mlops.community/1f9yb2] // Song [https://www.youtube.com/watch?v=AcSh_tfQa3w] Lessons learned after a decade in software development, covering shifts in perspective on simplicity, typed languages, functional programming, database choices, and the importance of communication in engineering teams. Multi-Love // Gem [https://go.mlops.community/op7676] // Song [https://www.youtube.com/watch?v=bEtDVy55shI] A multi-agent AI workflow using LLaMA 3 8B accurately detects cognitive concerns in clinical notes, achieving a 0.90 F1-score, improved specificity (1.00), and efficient prompt refinement, offering scalable clinical applications. [Gem](https://go.mlops.community/az9ja5) ## Slack Spotlights 2 min read Slack Spotlights Stephen Oladele [https://go.mlops.community/733wfg] shares some of the chat you might have missed ## Best Object Detection & Instance Segmentation Models for Production 🔍 Best Object Detection & Instance Segmentation Models for Production Last week in #computer-vision [https://mlops-community.slack.com/archives/C026ED0PZEZ], Will started a discussion on the best object detection and instance segmentation models for production. With no one-size-fits-all answer, the community shared insights on their go-to models, deployment stacks, performance optimization techniques, and tooling choices. ⏩ Quick Summary: The best model really depends on what you have available (like GPUs or CPUs), whether you need batch processing, and how much latency you can tolerate. YOLO (Ultralytics) [https://docs.ultralytics.com/models/] and RT-DETR [https://docs.ultralytics.com/models/rtdetr/] are still popular, but NVIDIA's TAO Toolkit [https://developer.nvidia.com/tao-toolkit] and Triton Inference Server [https://developer.nvidia.com/triton-inference-server] have optimized deployment pipelines that some people like. Others prefer custom PyTorch Lightning [https://github.com/Lightning-AI/pytorch-lightning] setups because they're more flexible. Detectron2 [https://github.com/facebookresearch/detectron2] and MMDetection [https://github.com/open-mmlab/mmdetection] are still okay but might need some extra work to get them running optimally. Some people are looking into newer stuff like D-FINE [https://github.com/Peterande/D-FINE], Detrex [https://github.com/IDEA-Research/detrex], and SAM [https://github.com/facebookresearch/segment-anything] models to stay ahead of the curve. 🔥 Key Takeaways from the Discussion:💡 Popular Model Choices 🟢 NVIDIA TAO Toolkit: Iain suggested NVIDIA's ready-to-use object detection models for commercial use. He also mentioned that TensorRT optimization [https://docs.nvidia.com/deeplearning/triton-inference-server/archives/tensorrt_inference_server_180/tensorrt-inference-server-guide/docs/optimization.html] can be used to deploy them efficiently. Nice feature: 🔹 Good licensing, easy fine-tuning with NGC containers [https://catalog.ngc.nvidia.com/containers]. 🟢 YOLO (Ultralytics): This is the go-to tool for A. Soellinger and Burhan when they need fast and efficient inference. It has strong tooling (especially for prototyping and real-time applications) and super fast inference speeds with INT8 quantization when paired with TensorRT (on NVIDIA hardware). 🟢 RT-DETR: Derek Austin found RT-DETR to outperform YOLO in speed and accuracy for object detection workflows. 🟡 MMDetection & Detectron2: George Pearse has been trying out MMDetection's Mask R-CNN with a Swin Transformer backbone but wants to find another option. He thinks Detectron2 is reliable but it hasn't been updated in a while. 🟡 D-FINE & Detrex: Possible future contenders, but lack instance segmentation support. 🟡 Segment Anything Model (SAM [https://github.com/facebookresearch/segment-anything]): George Pearse is checking if the SAM can run smoothly in ONNX format [https://github.com/onnx/onnx]. 🟡 Custom Implementations: Will is currently running Detectron2 with R-CNN but exploring NVIDIA TAO for efficiency gains. 🚀 Deployment & Optimization Strategies Triton Inference Server: Both Iain Wallace and Ricard Borras use Triton to serve their models. Triton is pretty cool because it can handle deploying multiple models, do A/B testing, and even convert models to ONNX format so they run more efficiently. ONNX Challenges: Will & Iain Wallace found that converting Detectron2 models to ONNX didn’t yield the same performance. Custom Model Training: George Pearse debated transitioning to PyTorch Lightning or extending RT-DETR to support instance segmentation. 🤔 The Big Question: What’s Next? So, old-school frameworks like Detectron2 and MMDetection still get the job done, but there are newer and faster options out there now. The big question is whether to stick with what you know, try to optimize with NVIDIA's tools, or jump on the bandwagon and move to something new like RT-DETR. Got your own preferred model or serving stack? Join the conversation [https://mlops-community.slack.com/archives/C026ED0PZEZ/p1737727867174639] on #computer-vision [https://mlops-community.slack.com/archives/C026ED0PZEZ] and share your insights! 🚀 ## The Sandbox 1 min read The Sandbox A little place to test some ideas Trying out a few things here - let us know what you think here [https://go.mlops.community/hx6agg] or email steve@mlops.community Back to the Feature A highlight from last week Those struggling with AI agent performance must have breathed a sigh of relief when they saw this blog last week. It covered six common failure points and data-driven techniques to troubleshoot and improve AI agent performance effectively, from query generation and retrieval to reranking, looping, synthesis, and cost. Read it here [https://go.mlops.community/noxuaa] Gatewaze Grooves Sharing music picks from our latest members through the Gatewaze [https://go.mlops.community/GwazeEmail]. From all your replies, it's hard to imagine if there was an IRL event for the whole community, what the night out after would be like! I think it'd take a skilled DJ to mix this [https://www.youtube.com/watch?v=RJkeKmtLnDY], with this [https://www.youtube.com/watch?v=0XhdnwDOQjk], or mash up this [https://www.youtube.com/watch?v=NrLkTZrPZA4], and this [https://www.youtube.com/watch?v=pCgFZMLico0]. For any adventurous part-time producers who want to try, the updated list here [https://go.mlops.community/Grooves]. For everyone else, I’ll leave you with this video [https://www.youtube.com/watch?v=cae5Y1TAezw] that's the most recent release on the list. Tech Teaser A mini MLOps mindbender How many hot dogs would world record holder Joey Chestnut eat in the time it takes DeepResearch to typically complete a query? a) 10 - 30 b) 20 - 350 c) 40 - 250 d) 60 - 400 Click here [https://go.mlops.community/TechTeasers] for the answer. Interested in partnering with us? Get in touch: partners@mlops.community Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin]. The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/]. --- Source: https://aaif.live/newsletters/mlopscommunity/2025-02-06-the-good-the-bot-and-the-ugly-llms-in-production