--- title: "Is there more to evaluation than technical metrics?" newsletter: "MLOps Community" date: 2024-08-01 source: https://aaif.live/newsletters/mlopscommunity/2024-08-01-is-there-more-to-evaluation-than-technical-metrics --- # Is there more to evaluation than technical metrics? *Plus, a panel of extraordinary scale, don't be embarrassed by hallucinations, LLMOps are the tops, Semantic Search-Part 3, and hidden gems.* *MLOps Community — Agentic AI Foundation, 2024-08-01* ## Reliable LLM Products, Fueled by Feedback // Chinar Movsisyan // MLOps Podcast #251 Doing her Ph.D. to become Dr. Movsisyan, I think Chinar can already diagnose some cases of Metrics Myopia. We had a great chat about evaluation going beyond technical metrics, involving various stakeholders—not just engineers—in assessing AI product success. She discussed the need for tools to track product usage and performance, and shared her experiences addressing challenges like model optimization for drones and managing user expectations in regulated industries. Give it a listen—just what the doctor ordered! Video [https://home.mlops.community/home/videos/reliable-llm-products-fueled-by-feedback] || Spotify [https://open.spotify.com/episode/5zdlgaFQXifVDuDA6OX9lP?si=om8wKGKZT4m-sYuWAw7stQ] || Apple [https://podcasts.apple.com/us/podcast/reliable-llm-products-fueled-by-feedback-chinar-movsisyan/id1505372978?i=1000663817473] [https://podcasts.apple.com/us/podcast/reliable-llm-products-fueled-by-feedback-chinar-movsisyan/id1505372978?i=1000663817473](https://podcasts.apple.com/us/podcast/reliable-llm-products-fueled-by-feedback-chinar-movsisyan/id1505372978?i=1000663817473) [https://home.mlops.community/home/videos/reliable-llm-products-fueled-by-feedback](https://home.mlops.community/home/videos/reliable-llm-products-fueled-by-feedback) [https://open.spotify.com/episode/5zdlgaFQXifVDuDA6OX9lP?si=om8wKGKZT4m-sYuWAw7stQ](https://open.spotify.com/episode/5zdlgaFQXifVDuDA6OX9lP?si=om8wKGKZT4m-sYuWAw7stQ) ## Poor RAG performance? It could be your data. https://www.tonic.ai/textual?utm_campaign=MLOps%20Community%20Newsletter&utm_source[%E2%80%A6]0Community%20Newsletter&utm_medium=newsletter&utm_content=Aug1 Garbage in → garbage out. Data quality and data security are critical components of any production RAG system. Poorly parsed and chunked documents foster inaccurate and inefficient retrieval, and failing to de-identify your data [https://www.tonic.ai/blog/sensitive-data-in-text-embeddings-is-recoverable] before embedding can compromise data privacy. Tonic Textual [https://www.tonic.ai/textual?utm_campaign=MLOps%20Community%20Newsletter&utm_source[%E2%80%A6]0Community%20Newsletter&utm_medium=newsletter&utm_content=Aug1] enables you to create production-grade unstructured data pipelines that enhance your RAG system’s performance by supplying it with better data. Textual integrates directly with your knowledge store and automatically parses and normalizes your unstructured data. While in-flight, our NER models spring into action to extract semantic metadata and detect and redact or synthesize sensitive data. Finally, you can customize your chunking strategy and use the embedding model of your choice to load clean, secured, and enriched data from any file format into your vector store. Textual can help streamline data preparation for RAG and supply your models with quality, secure unstructured data. Get started for free today. Data privacy on your mind? You’re not alone. Join us on August 26th, 2024 at 12:00 pm PT for our webinar Enterprise Data Privacy in AI [https://tonic-ai.zoom.us/webinar/register/WN_mYb765C7QIGGsVEvIdBE5A#/registration], featuring Fortune 500 AI Advisor Allie K. Miller and George Mathew from Insight Partners. Try Tonic Textual Free [https://www.tonic.ai/textual?utm_campaign=MLOps%20Community%20Newsletter&utm_source[%E2%80%A6]0Community%20Newsletter&utm_medium=newsletter&utm_content=Aug1] ## A BLUEPRINT FOR SCALABLE & RELIABLE ENTERPRISE AI/ML SYSTEMS ## AIQCON Panel A BLUEPRINT FOR SCALABLE & RELIABLE ENTERPRISE AI/ML SYSTEMS Olympic climbing events start on Monday, but you don’t need to wait until then to see professionals on scaling. Our recent conference featured a panel with experts from Bank of America, NVIDIA, Microsoft, and IBM discussing the blueprint for scalable and reliable AI/ML systems. They covered aligning AI initiatives with business objectives, managing data silos, and addressing new risks introduced by generative AI. The panel emphasized the importance of consistent data standards, continuous monitoring of model performance, and adapting MLOps practices for LLMs. Future trends included agentic workflows, improved data accessibility, and making AI tools more user-friendly across various business functions. Give it a watch to start your training now! Video [https://home.mlops.community/home/videos/a-blueprint-for-scalable-and-reliable-enterprise-aiml-systems] || Spotify [https://open.spotify.com/episode/0k85dqVnScdGLYGQYRITRu?si=9557c29b7d084ffc] || Apple [https://podcasts.apple.com/us/podcast/a-blueprint-for-scalable-reliable-enterprise-ai-ml/id1505372978?i=1000663441536] ## Job of the Week [https://www.linkedin.com/jobs/view/3820989394/](https://www.linkedin.com/jobs/view/3820989394/) ## Mitigating Hallucinations & Embarrassing Responses in RAG Applications // Alon Gubkin // IRL #88 ## MLOps Community IRL Meetup Mitigating Hallucinations & Embarrassing Responses in RAG Applications // Alon Gubkin // IRL #88 Will AI ever advance to the point of staying awake at night, worrying about embarrassing things it said? This chat explores mitigating RAG hallucinations in customer-facing applications through prompt engineering, fine-tuning LLMs, and handling edge cases. It introduces a system that acts as a firewall, offering real-time monitoring and correction to prevent hallucinations, profanity, and sensitive content. Give it a watch to help prevent this existential crisis. Watch it here [https://home.mlops.community/home/videos/mitigating-hallucinations-and-embarrassing-responses-in-rag-applications] [Watch it here](https://home.mlops.community/home/videos/mitigating-hallucinations-and-embarrassing-responses-in-rag-applications) Understanding LLMOps: Navigating the waters of large language models [https://home.mlops.community/home/blogs/understanding-llmops-navigating-the-waters-of-large-language-models] Teardrops on laptops about LLMOps flops? Dry your eyes and have a read of this blog that looks at LLMOps, focusing on managing and operationalizing LLMs in production environments. It covers challenges such as preprocessing unstructured data, reducing hallucinations, and ensuring robust validation. It stresses the need for continuous monitoring, user feedback loops, and structured human intervention to maintain model reliability. Blog drops improve your chops at LLMOps. With thanks to David Weik for their contribution. Semantic Search to Glean Valuable Insights from Podcast [https://home.mlops.community/home/blogs/semantic-search-to-glean-valuable-insights-from-podcast-part-3] Amigos, Musketeers, and semantic search blogs, the best things come in threes! The conclusion to Part 1 [https://home.mlops.community/home/blogs/semantic-search-to-glean-valuable-insights-from-podcasts-part-1] and Part 2 [https://home.mlops.community/home/blogs/semantic-search-to-glean-valuable-insights-from-podcast-series-part-2], the finale covers the use of semantic search to query podcast transcripts. The process begins by transcribing podcast audio with OpenAI's Whisper model and storing it in ApertureDB. Transcripts are embedded using LangChain's token text splitter and Cohere's embed-v3. The "find similar" method in ApertureDB locates contextually relevant answers, explaining how to initialize embedding models and perform similarity searches. Additionally, it outlines building a question-answer chatbot with LangChain and Cohere’s Command-R model, enabling interactive queries on podcast data. Don't waste one of your three wishes on search, just read this! With thanks to Sonam Guptafor their contribution. [Understanding LLMOps: Navigating the waters of large language models](https://home.mlops.community/home/blogs/understanding-llmops-navigating-the-waters-of-large-language-models) ## Video Roundup Like an Olympics highlights package, community contributor Médéric watched more than 50 videos from Databricks AI/ML Summit 2024 [https://www.youtube.com/c/databricks] and picked some podium placing presentations for us: * Customizing your Models: RAG, Fine-Tuning, and Pre-Training [https://www.youtube.com/watch?v=N41R9lMLdXk]: best talk of the summit from my opinion, does a really great job at explaining the different Gen AI approaches while remaining easy to understand. * In the Trenches with DBRX: Building a State-of-the-Art Open-Source Model [https://www.youtube.com/watch?v=xJ54VFW4bUU]: more advanced, but really interesting REX on training DBRX (405B models from Databricks). * LLMs in Production: Fine-Tuning, Scaling, and Evaluation [https://www.youtube.com/watch?v=5Q2KGEN3APk]: shares great tools and practices for doing fine tuning. A feat worthy of a gold-medal! 🥇 ## Hidden Gems ## IRL Meetups San Jose [https://news.mlops.community/e/c/eyJlIjoxNTY4NzAsImVtYWlsX2lkIjoiZXhhbXBsZSIsImhyZWYiOiJodHRwczovL2x1Lm1hL3Nqem1xNTIyP3V0bV9jYW1wYWlnbj1XZWVrbHkrTmV3c2xldHRlcistKzIwMjQtMDctMjVcdTAwMjZ1dG1fY29udGVudD1XZWVrbHkrTmV3c2xldHRlclx1MDAyNnV0bV9tZWRpdW09ZW1haWxfYWN0aW9uXHUwMDI2dXRtX3NvdXJjZT1jdXN0b21lci5pbyIsInQiOjE3MjE5ODM5OTV9/9e14faa7d8f2edef129446c4e761dbd81dfa8c01d12dab0f02fa6c31b0551787] - August 6 (📣shout out to Google Cloud) [San Jose](https://news.mlops.community/e/c/eyJlIjoxNTY4NzAsImVtYWlsX2lkIjoiZXhhbXBsZSIsImhyZWYiOiJodHRwczovL2x1Lm1hL3Nqem1xNTIyP3V0bV9jYW1wYWlnbj1XZWVrbHkrTmV3c2xldHRlcistKzIwMjQtMDctMjVcdTAwMjZ1dG1fY29udGVudD1XZWVrbHkrTmV3c2xldHRlclx1MDAyNnV0bV9tZWRpdW09ZW1haWxfYWN0aW9uXHUwMDI2dXRtX3NvdXJjZT1jdXN0b21lci5pbyIsInQiOjE3MjE5ODM5OTV9/9e14faa7d8f2edef129446c4e761dbd81dfa8c01d12dab0f02fa6c31b0551787) Interested in partnering with us? Get in touch: partners@mlops.community Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity]. The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/]. --- Source: https://aaif.live/newsletters/mlopscommunity/2024-08-01-is-there-more-to-evaluation-than-technical-metrics