--- title: "Tricks of the Trade: Michael Gschwind’s Insights on TorchChat and LLMs" newsletter: "MLOps Community" date: 2024-11-26 source: https://aaif.live/newsletters/mlopscommunity/2024-11-26-tricks-of-the-trade-michael-gschwind-s-insights-on-torchchat --- # Tricks of the Trade: Michael Gschwind’s Insights on TorchChat and LLMs *Plus, AI Agents in Production VC panel, cost of answers going up, open-source guide, state of AI Agents, whitepaper on agents, mixture-of-exerts, and still testing ideas.* *MLOps Community — Agentic AI Foundation, 2024-11-26* https://go.mlops.community/v8g9ro Early edition this week to let you digest your turkey! When that relative asks you to build a website for their crystal chakra alignment business just because you work in tech, point them to this gem [https://go.mlops.community/v8g9ro] and tell them to do it themselves! Have a great Thanksgiving! ## PyTorch's Combined Effort in Large Model Optimization PyTorch's Combined Effort in Large Model Optimization Michael K. Gschwind - Software Engineer, Software Executive @ Meta So many owe Michael a thank you - and I don’t just mean those who mastered the Kickflip McTwist in Tony Hawk thanks to his work on the PS3. We chatted about his impressive journey, before doing a 'kickturn' and talking about the evolution of LLM infrastructure and his work on TorchChat. Quick bit of background on TorchChat, an integration of PyTorch: * Built on BetterTransformer, FlashAttention, and TorchCompile * Features advanced quantization enabling reduced memory use and faster on-device performance * Provides an end-to-end workflow for LLM optimization, including export to non-Python environments Throughout, he stressed the importance of avoiding premature optimization, iterating rapidly, and fostering community-driven innovation. We also explored what’s next for LLMs, focusing on: * Tighter integration and seamless workflows * New model architectures and transformer variants * Pushing boundaries with numeric formats like 4-bit floating point Click below to listen – easiest trick you'll ever land! Video [https://go.mlops.community/gsqwa4] || Spotify [https://go.mlops.community/x4mapa] || Apple [https://go.mlops.community/orjqos] ## LLMs to agents: The Beauty & Perils of Investing in GenAI LLMs to agents: The Beauty & Perils of Investing in GenAI AI Agents in Production // VC Panel They say the best investment you can make is in yourself. This is a close second. In this panel from AI Agents in Production, VCs from Prosus Ventures, Redpoint, and Sequoia discussed the opportunities and risks of investing in GenAI and agentic systems. They explored how AI agents are transforming industries like healthcare, construction, and customer support while addressing challenges in scaling, adoption, and economics. When it comes to picking winning startups, they emphasized a few key factors: * Team and Timing: Strong teams with a clear understanding of customer needs and realistic roadmaps for scaling their technology. * Proprietary Data: A competitive edge through access to domain-specific data, especially in areas like healthcare, where generalized models often fall short. * Business Viability: Sustainable business models that can handle the high cost of computation while delivering meaningful ROI. The panelists also noted that adoption relies on reshaping consumer behaviors and demonstrating tangible benefits. Looking ahead, they predicted advancements in personalized AI assistants, multi-agent collaboration, and innovative user interaction methods like voice and avatars. Click below to watch – a worthwhile investment of your time! Watch it here [https://go.mlops.community/y5t4t6] ## Price per token is going down. Price per answer is going up. Price per token is going down. Price per answer is going up. Ah, LinkedIn - a utopia of professional networking and showcasing expertise. Or, the perfect place to argue with someone who doesn’t know the difference between answer and response. This blog started as a response to a LinkedIn post claiming LLM costs are dropping because token prices are falling. While true on the surface, it ignores an important distinction: responses aren’t the same as answers. I got some push back on it, so thought I’d explain a bit more. Expectations for AI have grown: simple Q&A isn’t enough anymore. Users now demand systems that handle complex tasks, requiring more LLM calls, backend complexity, and sophisticated architectures. Take agents: every step in their workflows - planning, reasoning, refining - piles on extra calls and costs. Even fine-tuned open-source models aren’t immune, factoring in setup, hosting, and debugging. As expectations rise, inefficiencies grow, and low-quality outputs can derail results. Ultimately, while token costs fall, delivering meaningful, actionable answers has become more expensive and intricate than ever. Have a read to see if you agree. Not with me - I know I’m right - I mean with @Podcaster-k4j. Read it here [https://go.mlops.community/eo1aw5] ## Hidden Gems ## Six Ways // // Six Ways // Gem [https://go.mlops.community/ci7k09] // Song [https://go.mlops.community/5ugs62] A blog on democratizing AI engineering through open-source tools and models, with six components - models, prompts, knowledge bases, integrations, testing, and deployment. How It Is // Gem [https://go.mlops.community/o79mvn] // Song [https://go.mlops.community/t827lr] A report on the current landscape of AI agents, highlighting adoption trends, leading use cases, and challenges in deploying agents across various industries. White Paper // Gem [https://go.mlops.community/2hcyoq] // Song [https://go.mlops.community/xxdln0] A whitepaper exploring the development and deployment of AI agents, highlighting architectures, training methodologies, and real-world implementations. The Expert // Gem [https://go.mlops.community/y49kyo] // Song [https://go.mlops.community/nwzhbs] A paper on DeepSeek-Coder-V2, an open-source Mixture-of-Experts code language model with 6T tokens, GPT-4 Turbo-level coding, 338 languages, 128K context. ## Job of the Week [https://go.mlops.community/i3z13b](https://go.mlops.community/i3z13b) ## The Sandbox The Sandbox A little place to test some ideas Trying out a few things here - let us know what you think here [https://go.mlops.community/hx6agg] or email steve@mlops.community Slack Spotlights Highlighting some of the chat you might have missed Busy in the #mlops-questions-answered [https://go.mlops.community/r6ya8w] channel, a few picks: A discussion [https://go.mlops.community/4zc0zp] about implementing RAG systems that dynamically adjust between rule-based ML and LLM approaches based on user roles and business needs. Replies discussed related challenges, solutions like context-aware query tools, and tools like NomadicML for optimizing such systems. Someone's asking for advice [https://go.mlops.community/rrp7t0] on optimizing a VisionEncoderDecoderModel, particularly experimenting with quantization, and notes limitations with ONNX and Torch implementations, asking for alternative approaches or solutions. There's a question [https://go.mlops.community/kqb61z] about switching SageMaker scaling to CPU utilization- if you've got any tips on interpreting SageMaker CPU metrics jump in! Tech Teaser A mini MLOps mindbender Last week's answer: If you double the input size, output size, and the number of neurons per layer, the total number of parameters in the neural network ends up being 4 times more than before. This week's question: Preempting all the Thanksgiving leftovers and Black Friday deals, you consider buying a smart fridge. You find one with AI that classifies foods. Adding a new type of food doubles the output layer size but doesn't change the input or hidden layers. How does this affect the number of parameters in a fully connected network? Interested in partnering with us? Get in touch: partners@mlops.community Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin]. The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/]. --- Source: https://aaif.live/newsletters/mlopscommunity/2024-11-26-tricks-of-the-trade-michael-gschwind-s-insights-on-torchchat