--- title: "Due Diligence Decoded: Preparing for Hidden Codebase Realities" newsletter: "MLOps Community" date: 2024-12-05 source: https://aaif.live/newsletters/mlopscommunity/2024-12-05-due-diligence-decoded-preparing-for-hidden-codebase-realitie --- # Due Diligence Decoded: Preparing for Hidden Codebase Realities *Plus, lighting the way for optimizing PyTorch, going in big on inference scaling for long-context RAG, hidden gems, and the sandbox.* *MLOps Community — Agentic AI Foundation, 2024-12-05* https://go.mlops.community/s5ctbn Getting paid, or just getting ‘a chance to be part of something big’? Take the State of AI Survey [https://go.mlops.community/s5ctbn] to find out where you stand and help us map the landscape for AI/ML roles. You’ll also be among the first to access the insights when the survey wraps up. ## AI-Driven Code: Navigating Due Diligence & Transparency in MLOps AI-Driven Code: Navigating Due Diligence & Transparency in MLOps Matt van Itallie // Founder and CEO @ Sema We’ve all been there, haven’t we? The ink hasn’t even dried on your multibillion-dollar acquisition when the legacy issues start popping up. After chatting with Matt about codebase scans and technical due diligence, I feel more prepared for my next one. We broke his process into eight focus areas, all designed to uncover the non-functional realities of a codebase. A few highlights: * Code Quality: Beyond linting, they assess complexity, test coverage, and how maintainable the code is without its original developers. * Cloud Optimization: They find underutilized resources (like idle GPUs) and suggest quick wins to cut costs. * Generative AI Code: Matt’s team developed the “G-bomb” to analyze AI-generated code and highlighting where extra review is needed to avoid risks. * Team Stability: A company’s strongest asset isn’t always the code—it’s often the people who wrote it. Having the original team stick around matters. Matt stressed the need to translate all this technical data into business-friendly insights for non-technical stakeholders like benchmarks and dollar impacts. We also discussed how GenAI is reshaping coding, from speeding up prototypes to introducing new challenges with security, quality, and IP. Do your due diligence and click below to listen. Video [https://go.mlops.community/47qu6o] || Spotify [https://go.mlops.community/dqh22g] || Apple [https://go.mlops.community/lq9aqc] ## PyTorch for Control Systems and Decision Making PyTorch for Control Systems and Decision Making Vincent Moens // Research Engineer @ Meta They say assumptions are the mother of all... well, I'm going to assume you know the rest. Vincent shared surprising insights on common assumptions about pin memory, along with tips for getting the most out of PyTorch. This conversation spanned everything from TensorDict’s modular capabilities to the importance of benchmarking workflows for real-world performance improvements, with some highlights being: * Why explicit use of pin memory for CUDA transfers isn’t always faster. * A simple trick to reduce CPU overhead in reinforcement learning workflows by leveraging dual-module setups. * TensorDict’s flexibility as a data structure, enabling seamless handling of complex models and diverse use cases like robotics, drug design, and generative AI. Vincent also discussed the evolution of PyTorch tools and tips for leveraging compile functionality to unlock substantial performance gains. I’ll assume you’ll click below to listen. Video [https://go.mlops.community/lnfvgj] || Spotify [https://go.mlops.community/48qvgr] || Apple [https://go.mlops.community/cdyzpm] ## Inference Scaling for Long-Context RAG Inference Scaling for Long-Context RAG MLOps Community Reading Group There are 2 types of people in this world, those that can infer what comes next, and those that needed the last part of this sentence. In this Reading Group session, we explored Inference Scaling for Long Context RAG [https://go.mlops.community/hq7df8], Google DeepMind’s latest work on long-context models capable of processing millions of tokens. The discussion highlighted: * How Iterative RAG improves multi-step reasoning for complex tasks by breaking questions into smaller subqueries. * Trade-offs between retrieval accuracy, noise, and compute costs. * Parameter optimization techniques and scaling laws to maximize performance. We also examined the challenges of current retrieval methods and their limits in real-world scenarios. You don’t need Iterative RAG to know what to do next! Watch it here [https://go.mlops.community/cjq538] The next session will be discussing A Taxonomy of AgentOps for Enabling Observability of Foundation Model based Agents [https://go.mlops.community/lt7bwv] on Dec 12. Register here [https://go.mlops.community/kcefwd] [Inference Scaling for Long Context RAG](https://go.mlops.community/hq7df8) ## Hidden Gems ## Vulnerable // Vulnerable // Gem [https://go.mlops.community/dexv2e] // Song [https://go.mlops.community/444rfy] An open-source tool for testing LLM vulnerabilities like hallucination, data leaks, and toxicity, supporting models from OpenAI, Hugging Face, and Replicate. Don't Sweat The Technique // Gem [https://go.mlops.community/81adu7] // Song [https://go.mlops.community/10j6ri] An article exploring reasoning techniques in compound AI systems, focusing on combining multiple AI architectures and post-training optimization to enhance decision-making and scalability. It's All Right Now // Gem [https://go.mlops.community/tsfdno] // Song [https://go.mlops.community/rbmlaa] An overview of the Institute for Ethical AI & Machine Learning, focusing on their commitment to responsible AI development through research, best practices, and community engagement. Cohesion // Gem [https://go.mlops.community/pg64o2] // Song [https://go.mlops.community/9qtfww] An article detailing the evolution of Mathem’s data platform into a unified, declarative system, focusing on enabling autonomous data product development through a serverless architecture, medallion layers, and YAML-based data contracts. ## Job of the Week [https://go.mlops.community/6lu5cw](https://go.mlops.community/6lu5cw) ## The Sandbox The Sandbox A little place to test some ideas Trying out a few things here - let us know what you think here [https://go.mlops.community/hx6agg] or email steve@mlops.community Slack Spotlights Sharing some of the chat you might have missed There was a question in #computer-vision [https://go.mlops.community/t8jq0c] about using open source tools to parse PDFs that are messy and contain handwritten notes. They’ve already tried phi3-vision, PaddlePaddle, moondream, so if you’ve got any suggestions, drop them in the channel. Looking to add a bit of personality to your model? There’s a discussion in #llmops [https://go.mlops.community/hss0y5] on how to do just that. This repository [https://go.mlops.community/qrq70k], created by Caleb Fahlgren and shared in #open-source [https://go.mlops.community/pl4c2f], offers a lightweight library for AI observability. Oversample? Undersample? How do you get the Goldilocks sample? Some discussion in #mlops-questions-answered [https://go.mlops.community/tn6w3k] on balancing imbalanced datasets: oversampling, undersampling, and other strategies. Back to the Feature A highlight from last week Michael Gschwind was a great guest - when someone with as much as experience as that talks, it pays to listen! He shared insights on TorchChat, focusing on rapid iteration, community-driven innovation, and the future of LLMs - from seamless workflows to new model architectures. Video [https://go.mlops.community/gsqwa4] || Spotify [https://go.mlops.community/x4mapa] || Apple [https://go.mlops.community/orjqos] Tech Teaser A mini MLOps mindbender Last week's question: You bought a new fridge that classifies foods and adding a new type of food doubles the output layer size but doesn't change the input or hidden layers. We asked how does this affect the number of parameters in a fully connected network? Answer: Doubling the output layer size in a fully connected network only increases the number of parameters by the product of the previous layer's size and the additional output neurons. This week's question: Insired by the openings to the podcasts, you’re training a model to recommend coffee. Each coffee shop adds 3 unique features to the dataset, but doubling the number of shops also doubles the number of users in your training data. How does the size of your dataset grow? Interested in partnering with us? Get in touch: partners@mlops.community Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin]. The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/]. --- Source: https://aaif.live/newsletters/mlopscommunity/2024-12-05-due-diligence-decoded-preparing-for-hidden-codebase-realitie