--- title: "Smart Networks vs. False Facts" newsletter: "MLOps Community" date: 2025-01-09 source: https://aaif.live/newsletters/mlopscommunity/2025-01-09-smart-networks-vs-false-facts --- # Smart Networks vs. False Facts *Plus, evaluating gen AI systems, moving 16 live ML models, agents and autonomy, a reading group on taxonomy of AgentOps, hidden gems, and the sandbox.* *MLOps Community — Agentic AI Foundation, 2025-01-09* https://go.mlops.community/an1e90 Happy New Year! And welcome back with a bumper edition of the newsletter as we catch up on everything you might have missed over the break. There are 4 podcasts, 2 blogs, a reading group, and more, so let's get cracking! [Join the reading group](https://go.mlops.community/an1e90) ## Unleashing Unconstrained News Knowledge Graphs to Combat Misinformation Unleashing Unconstrained News Knowledge Graphs to Combat Misinformation Robert Caulk // Founder @ Emergent Methods I will always love you, all along the watchtower, proud Mary. Not an unusual declaration of love, but a few cover songs that Robert made me think of with his analogy about how your first experience with something can shape how you judge it later. In his case, he was talking about ontologies, as he shared insights on building ontology-free graphs, emphasizing flexibility over rigid metadata structures. Several key design decisions drive the system's effectiveness: * Ontology-free graphs reduce complexity by letting relationships form naturally, eliminating the need for predefined schemas. * Ephemeral MEM Graph instances can be quickly spun up and discarded, ensuring efficient handling of complex user queries. * Fine-tuned models like Phi 3 Mini, combined with keyword search, enhance entity extraction and disambiguation, making the data both precise and prompt-ready. We also talked about how Ask News constructs real-time knowledge graphs from thousands of news sources using AI, vector search, and keyword extraction to deliver nuanced, context-rich stories. No covers with this one, just original insights! Video [https://go.mlops.community/shikl7] || Spotify [https://go.mlops.community/d4g6c0] || Apple [https://go.mlops.community/ipp7tf] ## Holistic Evaluation of Generative AI Systems Holistic Evaluation of Generative AI Systems Jineet Doshi // Staff Data Scientist @ Intuit If you’re using an LLM as a judge, who judges the judge? What about an LLM jury? But then, who judges the jury…? This conundrum surfaced during a discussion on evaluation, alongside agent orchestration challenges, and generative AI complexities. Jineet described how Intuit’s AI infrastructure scales to 58 billion predictions daily and underscored the need for robust evaluation throughout the ML lifecycle. He highlighted three primary evaluation methods: * Human-based evaluations: Effective, but costly, involving manual labeling and red teaming. These methods provide high-quality results but lack scalability. * LLM as judges: A scalable approach requiring careful prompt design, bias mitigation, and regular recalibration to maintain reliability. * Benchmarks and traditional NLP techniques: Useful for structured tasks, but limited in handling open-ended outputs, necessitating custom benchmarks for specific use cases. He also stressed the need for both unit tests, which assess components like retrieval accuracy, and integration tests for end-to-end system performance. Emerging ideas, such as using models like Claude to simulate user interactions, were suggested as promising future directions. You don’t need a judge and jury to tell you it’d be a crime not to listen. Video [https://go.mlops.community/1y3bvd] || Spotify [https://go.mlops.community/lgseud] || Apple [https://go.mlops.community/e20of6] ## Re-Platforming Your Tech Stack Re-Platforming Your Tech Stack Michelle Marie Conway // Lead Data Scientist @ Lloyds Banking Group Andrew Baker // Data Science Delivery Lead @ Lloyds Banking Group They say that moving house is one of the most stressful things you can do, so imagine how Andrew, Michelle, and their MLOps team felt moving 16 live ML models. If they were stressed, it didn’t show in this chat as they talked about Lloyds Banking Group’s large-scale move from on-prem to Google Cloud. Perhaps because it resulted in significant performance improvements - one critical pipeline went from five hours to just 20 minutes. They also shared key lessons in managing production ML models, particularly around maintaining robustness and dealing with ongoing governance, including: * Collaboration with platform teams: Early involvement helped shape infrastructure decisions to ensure the platform was fit for purpose. * Incremental updates over big-bang changes: This approach reduced risks and made it easier to troubleshoot issues when things inevitably went wrong. * Balancing accuracy and reliability: The team prioritized explainability and long-term operability over marginal gains in model accuracy. Click to listen now, and feel your stress melt away! Video [https://go.mlops.community/koait6] || Spotify [https://go.mlops.community/4doq29] || Apple [https://go.mlops.community/zzeqa2] ## ML, AI Agents, and Autonomy ML, AI Agents, and Autonomy Egor Kraev // Head of AI @ Wise Plc I was a little surprised in the intro when Egor said he drank green tea and not grog - mainly because I misunderstood his role as one of the founders of the Swiss Pirate Party. He set me right on that, and shared a range of insights on applied AI at Wise, where he focuses on classic ML techniques, like XGBoost for fraud detection and treasury operations. He highlighted how LLMs help turn unstructured data into structured formats, making them particularly useful for automating tasks like customer support classification. One standout innovation was MotleyCrew, an open-source framework that allows flexible use of agent tools without locking into a specific ecosystem. He outlined: * Interoperability across agent frameworks: LangChain, LlamaIndex, Crewai. * Support for patterns like forced validation to ensure robust agent responses. Additionally, he discussed CausalTune, a Wise-tested library for causal inference in marketing, enabling customer-level targeting by estimating impacts from A/B tests and hypothetical scenarios. Have a listen, you’ll be hooked! Video [https://go.mlops.community/gxbuuo] || Spotify [https://go.mlops.community/a3fj32] || Apple [https://go.mlops.community/bfmv0s] ## A Taxonomy of AgentOps for Enabling Observability of Foundation Model-based Agents A Taxonomy of AgentOps for Enabling Observability of Foundation Model-based Agents MLOps Community Reading Group AgentOps? More like Agent-Oops when the taxonomy fails - making this reading group discussion essential. They dissected key elements outlined in the paper A Taxonomy of AgentOps for Enabling Observability of Foundation Model based Agents [https://go.mlops.community/lt7bwv], such as planning, reasoning, memory, and workflows. They debated how existing MLOps practices apply - or don’t - when dealing with agent systems, given their complexity and lack of clear control. Tracing and evaluation were major themes, reflecting the paper’s emphasis on observability: * Traces capture the full journey of a task through the agent system, while spans break it into smaller, traceable actions. * Evaluation methods need to assess not only the final outcome but also each intermediate step, ensuring every part of multi-step tasks is tracked and improved. The paper’s mention of guardrails prompted debate on their role in maintaining agent reliability. Should they be embedded in prompts, or handled externally for better predictability? The group agreed that while the taxonomy is a great start, the field remains in its early days, with plenty of room for refining how observability is done. Have a watch and soon your AI agent’s taxonomy will be smart enough to classify its own classification errors. Watch it here [https://go.mlops.community/DecReadingGroup] [A Taxonomy of AgentOps for Enabling Observability of Foundation Model based Agents](https://go.mlops.community/lt7bwv) ## ML piplines in the age of LLMs: from local containers to cloud experiments ML piplines in the age of LLMs: from local containers to cloud experiments With thanks to Gleb Lukicov for thier contribution. With the AI hype cycle, you can get used to being disappointed when things are announced as "being in the pipeline". This blog on pipelines won’t disappoint, though, as it solves the problem of transitioning from local, container-based approaches to more scalable cloud-based experimentation methods, especially crucial for resource-intensive LLMs. It shows how developing and testing pipelines locally first ensures smoother scaling when moving to the cloud. A practical example walks through setting up a pipeline: * Running a complete ML pipeline locally with steps such as data retrieval, processing with an LLM, and evaluation. * Executing the entire pipeline with a single command, speeding up iteration and debugging. It also highlights key tools for seamless development: * Docker Desktop to enable containerized local workflows. * Kubeflow Pipelines v2 for orchestrating and scaling in the cloud. This approach boosts consistency and efficiency across development environments. Have a read and master pipelines like Mario! Read it here [https://go.mlops.community/c9qtal] ## What on Earth is RAG? What on Earth is RAG? With thanks to Steven Jieli Wu for thier contribution. There’s a saying that if you can't explain it to a six-year-old, you don't understand it yourself. Four years off, but Steven nails it by explaining RAG to a 10-year-old before detailing practical insights for deploying RAG in production environments. He highlights how retrieval systems like Coveo or Lucidworks can reduce reliance on high-cost models like GPT-4, with GPT-3.5 Turbo proving both cost-effective and efficient. In regulated industries, meeting strict accuracy (>90%) and low latency (<200ms) targets often requires combining vector search (Databricks) with sparse retrieval for better relevance tuning. A couple of key takeaways: * Streaming responses improves user perception by making the system feel faster. * Effective evaluation requires both automated LLM scoring and SME feedback—time-consuming, but necessary. Be sure to read and make RAG child’s play! Read it here [https://go.mlops.community/ghbxtv] ## Hidden Gems ## In Deep // // In Deep // Gem [https://go.mlops.community/uh95w6] // Song [https://go.mlops.community/7844iy] An in-depth analysis of Amazon's Trainium2 architecture, detailing its specifications, networking capabilities, and deployment strategies, highlighting AWS's efforts toward AI self-sufficiency and reduced reliance on third-party hardware. Guide // Gem [https://go.mlops.community/cbltne] // Song [https://go.mlops.community/c25qly] A comprehensive guide detailing the process of developing LLM-native applications, covering stages from ideation to experimentation, evaluation, and productization, with a focus on standardizing workflows and embracing a research-oriented mindset. Feel Good Inc. // Gem [https://go.mlops.community/651fjb] // Song [https://go.mlops.community/97y5eu] A blog post exploring the integration of AI-generated clinical text into electronic health records, discussing potential benefits, challenges, and implications for healthcare providers and patients. Chips // Gem [https://go.mlops.community/40haiv] // Song [https://go.mlops.community/1ak4uy] A podcast episode analyzing the AI semiconductor landscape, covering topics such as NVIDIA's competitive edge, the shift to GPUs in data centers, challenges in scaling AI pre-training, synthetic data generation, and evolving memory technology. ## Job of the Week [https://go.mlops.community/fgx1pe](https://go.mlops.community/fgx1pe) ## The Sandbox The Sandbox A little place to test some ideas Trying out a few things here - let us know what you think here [https://go.mlops.community/hx6agg] or email steve@mlops.community Slack Spotlights Sharing some of the chat you might have missed In the #discussions [https://go.mlops.community/fr30i7] channel, there was some chat about the switch from NVIDIA to AMD [https://go.mlops.community/vi95tx], a question about why the same OCR setup [https://go.mlops.community/lm6wgh] produces consistent but different results between two environments, and a request to chat [https://go.mlops.community/k84hvy] with anyone who's worked with the Enterprise Snorkel Flow platform. It’s been busy in the #mlops-questions-answered [https://go.mlops.community/wcwrrv] channel too, with: * answers about how to retrieve N nodes from a knowledge graph for RAG [https://go.mlops.community/0ajquo]. * a question on the best way to automate periodic updates [https://go.mlops.community/i5q8uk] to a vector database in an LLM RAG app using Langchain, Chroma, AWS, and Netsuite data * a discussion [https://go.mlops.community/p6qob8] on whether serverless cloud AI APIs (like those from GCP and AWS) still hold value in an AI/Data Stack. Back to the Feature A highlight from last week year From our Year in Review mails, the milestone 200th episode [https://go.mlops.community/nEdHhB] on founding and funding was a highlight from January to April. Between May and August, you all enjoyed the this look [https://go.mlops.community/vbf5gv] at recommender systems with Spotify’s Senior ML Engineer. Finally, September through to December, our Data Engineering for AI/ML conference [https://go.mlops.community/1h430r] collection was, unsurprisingly, the biggest hit. Tech Teaser A mini MLOps mindbender New year, new look you! You decide to create an AI stylist that matches outfits. If the model has nnn shirts and mmm pants, how many combinations does it need to evaluate for a single outfit recommendation? Click here [https://go.mlops.community/AIStylistAnswer] for the answer. Interested in partnering with us? Get in touch: partners@mlops.community Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin]. The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/]. --- Source: https://aaif.live/newsletters/mlopscommunity/2025-01-09-smart-networks-vs-false-facts