--- title: "Better Data, Better Models: The Conversation We Need" newsletter: "MLOps Community" date: 2025-02-20 source: https://aaif.live/newsletters/mlopscommunity/2025-02-20-better-data-better-models-the-conversation-we-need --- # Better Data, Better Models: The Conversation We Need *Plus, conducting a survey of workflow orchestration systems, hidden gems, Slack spotlight, and can AI really write efficient code?* *MLOps Community — Agentic AI Foundation, 2025-02-20* https://go.mlops.community/l6mgmj A busy week for releases, with Perplexity dropping R1 1776 [https://go.mlops.community/czhy72], its first 'open-source' model, and I use quotes because it wasn't without some pushback [https://go.mlops.community/l6mgmj]. There was also the launch of Grok 3, with some initial vibes [https://go.mlops.community/xcwthm] saying it's competitive with top models but still has limits. Much like my old report card: has potential, but could try harder. But at least my dad never went around calling me 'scary smart'. ## Look At Your ****ing Data 2 min read Look At Your ****ing Data Kenny Daniel // Founder @ Hyperparam Staying hydrated, getting enough sleep, wearing sunscreen, exercising - all widely acknowledged as important but mostly ignored. Add data to that. Kenny and I talked about how data quality isn’t getting enough attention, even though it’s key to model performance. At NeurIPS, barely any talks covered it, and major labs keep their curation methods secret. Even LLaMA and DeepSeek only hint at how they clean their datasets. Yet, better filtering - like LLaMA 3's model-based selection - can massively improve results. Although everyone agrees high-quality data matters, transparency on best practices is lacking, and Kenny shared how this leads to key challenges: * Nobody shares how datasets are cleaned - Big labs treat their data pipelines as proprietary, and research papers barely mention filtering methods. * More data isn’t always better - LLaMA 3 only used 10% of its available dataset, carefully selected through model-based filtering. Quality beats quantity. * Exploring large datasets is painful - Standard tools struggle with scale. Even Parquet lacks good exploration tools, which is why Kenny built a browser-based Parquet parser to make inspection easier. We also got into ML infrastructure, where hosting models remains an unsolved challenge, and how, though Python dominates ML tooling, it’s not great for UI development. Don’t make this podcast another thing you ignore - click below to listen. Video [https://go.mlops.community/dtydvl] || Spotify [https://go.mlops.community/1tnalf] || Apple [https://go.mlops.community/h1jj3c] ## A Survey of Workflow Orchestration Systems 2 min read A Survey of Workflow Orchestration Systems With thanks to Alex Miłowski for their contribution. Does your workflow orchestration make you feel less like a conductor and more like a random person waving a stick? This article explores workflow orchestration systems for MLOps, covering automation, scalability, and fault tolerance. It looks at declarative vs. imperative approaches, DAG-based vs. event-driven models, and how different tools manage dependencies and retries. It covers Kubernetes-native options like Argo Workflows, general-purpose systems such as Apache Airflow, and ML-specific tools like Kubeflow Pipelines. There’s also a look at performance trade-offs, security considerations, and how these tools fit into cloud or on-prem setups. A big focus is how different systems handle dependencies and execution flow: * DAG-based orchestration: Tools like Apache Airflow follow a strict directed acyclic graph (DAG) structure, where tasks run in a set order. This works well for predictable workflows but can be rigid when things change. * Event-driven orchestration: Systems like Prefect trigger tasks based on real-time events, making them more flexible for workflows that don’t always follow the same path. As well as exploring these different systems, it also covers how to pick the right tool based on workload complexity, execution speed, and operational overhead. Once you read it, you’ll be conducting an orchestration symphony in no time. Read it here [https://go.mlops.community/gceymz] ## Job of the Week [https://go.mlops.community/vmsl3j](https://go.mlops.community/vmsl3j) ## Hidden Gems ## Code Monkey // // Code Monkey // Gem [https://go.mlops.community/pinnmo] // Song [https://go.mlops.community/hugs4s] An article about teaching AI to generate optimized GPU code using reinforcement fine-tuning (RFT), detailing the use of Triton and reinforcement learning techniques to convert PyTorch functions into efficient, architecture-aware kernels. Check out Unsupervised Thoughts section below for more on this. Make It Easy On Yourself // Gem [https://go.mlops.community/ga5mmx] // Song [https://go.mlops.community/qqjcox] Reflecting on lessons from a decade in software engineering, emphasizing the challenges beyond coding - such as team alignment, navigating uncertainty, and handling complex project dynamics. We Were Wrong // Gem [https://go.mlops.community/n4r13c] // Song [https://go.mlops.community/eotri2] Fly.io sharing their experience with GPU-powered cloud infrastructure, detailing the challenges of hardware integration, security, and market demand - ultimately concluding that most developers prefer LLM APIs over direct GPU access. Beginnings // Gem [https://go.mlops.community/n5eyqx] // Song [https://go.mlops.community/0yqhwq] A GitHub repository by Microsoft, offering a 10-lesson course on building AI agents for beginners, covering fundamental concepts, practical examples, and code samples for hands-on learning. ## Slack Spotlights 2 min read Slack Spotlights Stephen Oladele [https://go.mlops.community/733wfg] shares some of the chat you might have missed ## 📝 Evaluating Text-to-SQL Agents: Challenges & Best Practices 📝 Evaluating Text-to-SQL Agents: Challenges & Best Practices This week in #agents [https://mlops-community.slack.com/archives/C0861BQ65A7], Kahl Kritzinger kicked off a discussion on how teams are evaluating the accuracy of their Text-to-SQL agents. While these agents promise to free up analysts from constant ad-hoc SQL requests, ensuring query correctness, avoiding hallucinations, and reducing bad assumptions remain a major challenges. ⏩ Quick Summary: Evaluating text-to-SQL agents is a tough nut to crack due to hallucinations, schema complexity, and the risk of incorrect queries. But hey, there are some best practices that can help! Think metadata-driven schema selection, few-shot examples, and retrieval-based approaches. And don't forget the human touch – having a human-in-the-loop for oversight and validation is crucial. Plus, guardrails, versioning, and continuous feedback loops are key to preventing those pesky hallucinations and incorrect queries. 🔥 Key Takeaways from the Discussion: ⚠️ Key Challenges with Text-to-SQL * Hallucinated queries – LLMs might generate syntactically correct but semantically wrong queries that could lead to misleading results. “Bad queries are worse than no queries.” * Lack of schema knowledge – Users often don’t fully understand the underlying database structure, which makes it hard to phrase precise questions. (David, Ross Katz) * Automating SQL query generation is harder than expected – Even skilled analysts struggle with new data ecosystems, and LLM-generated queries often require review—fully removing humans from the loop is unrealistic. (Ross Katz) 📊 How to Evaluate Text-to-SQL Accuracy * Execute the SQL & validate results – The most direct way to test accuracy is to run the generated queries and compare the results against a golden dataset. (Médéric Hurier) * Use hand-crafted benchmarks – Create a set of typical business questions and their expected SQL queries. (Ross Katz, Alex Strick van Linschoten) * Leverage evaluation frameworks and tools – * DSPy [https://github.com/stanfordnlp/dspy] – More focused on prompt fine-tuning, but some use its evaluation tools. * RAGA & QueryCraft [https://github.com/explodinggradients/raga] - Popular frameworks for measuring query validity and semantic accuracy. * DeepEval [https://github.com/confident-ai/deepeval] - Recommended by Ross Katz for benchmarking and tracking accuracy over time. * Faker [https://faker.readthedocs.io/en/master/] – Generate fake SQL databases for testing. (Médéric Hurier) * Retrieval Augmented Generation (RAG) for schemas – Some prefer live database access to pull schema details dynamically and improve query accuracy. (Meb, Ross Katz) 🚀 Strategies to Improve Performance * Metadata-driven schemas – Adding detailed descriptions, usage hints, column relationships, and sample values at the schema level significantly improves model accuracy. (Kahl Kritzinger) * Pre-generated SQL views – Instead of dynamically constructing queries, some teams predefine SQL views for common questions, reducing error rates. (David, Ross Katz) * Few-shot learning with analyst examples – Training the model on real-world SQL queries written by analysts greatly improves its ability to generate accurate responses. (Kahl Kritzinger) * Function calling for schema lookups – Consider dynamically fetching schema and sample queries during inference to improve context awareness. (Meb) 💬 Thoughts On “Fully Automated” Text-to-SQL * David: Many use cases feel like “a solution looking for a problem” given how much effort is needed to verify accuracy. * Ross Katz: Guardrails are the hardest part—bad SQL that runs can be disastrous. Maybe a human-in-the-loop approach is the way forward. 🔗 Bonus Resources Shared: * Hamel.dev: Your AI Product Needs Evals [https://hamel.dev/blog/posts/evals/] – A great breakdown on how to evaluate LLM-generated SQL. (Alex Strick van Linschoten) * Text-to-SQL: The Ultimate Guide for 2025 [https://medium.com/@ayushgs/text-to-sql-the-ultimate-guide-for-2025-3fa4e78cbdf9] – A fresh perspective on the state of text-to-SQL in GenAI. (Médéric Hurier) * Lessons Learned from Text-to-SQL [https://www.linkedin.com/pulse/text-sql-translation-problem-data-disciplines-ross-katz-pg87c/] – Hands-on experiences from rolling out a text-to-SQL system. (Ross Katz) * LLM Eval For Text2SQL [https://parlance-labs.com/education/evals/ankur.html] – Parlance Labs on Text-to-SQL. (Alex Strick van Linschoten) 🤔 The Big Question: Can Text-to-SQL Fully Replace Analysts? The consensus? Not yet. While retrieval-based systems and metadata-driven improvements help, LLM-generated SQL still requires validation before being trusted for decision-making. The best approach? A hybrid model where LLMs assist analysts by automating routine queries, but human oversight remains essential for complex analysis. 💬 Got insights on making text-to-SQL more reliable? Join the conversation [https://mlops-community.slack.com/archives/C0861BQ65A7/p1739508833501779] on #agents and share your experiences! 🚀 Interested in partnering with us? Get in touch: partners@mlops.community Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin]. The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/]. --- Source: https://aaif.live/newsletters/mlopscommunity/2025-02-20-better-data-better-models-the-conversation-we-need