--- title: "Proxy Problems, Pill Popping, and Python’s Limits" newsletter: "MLOps Community" date: 2025-03-20 source: https://aaif.live/newsletters/mlopscommunity/2025-03-20-proxy-problems-pill-popping-and-python-s-limits --- # Proxy Problems, Pill Popping, and Python’s Limits *Plus, reasoning engines, benchmark discipline, and GPU efficiency.* *MLOps Community — Agentic AI Foundation, 2025-03-20* Last week, Sam Partee gave a talk at AI in Production 2025 [https://go.mlops.community/ie6j0r]. This week, Arcade raises $12M [https://go.mlops.community/i5l6wg]. Coincidence? I don't think so! Huge congrats to Sam, Alex, and the Arcade team. Hit reply and tell me what you'd do with $12M - just keep it professional! ## GenAI Traffic: Why API Infrastructure Must Evolve... Again I feel this episode should come with a health warning for those like me who enjoy popping pills. The pills in question are supplements though, because Erica triggered memories for those of a certain age as she walked through the evolution of networking, from monoliths to microservices, and how proxies adapted to changing demands. But GenAI is forcing another shift, introducing challenges that existing infrastructure wasn’t designed for: * LLM requests are unpredictable – unlike traditional APIs, responses vary wildly in size and duration, making routing and resource allocation more complex. * Traffic is heavier – requests and responses are much larger, often exceeding gateway limits, requiring new approaches to scalability. * Streaming AI responses need better handling – unlike static content, LLM output must be processed dynamically, with real-time security checks. We also got into how Python-based gateways struggle at scale and why Envoy AI Gateway helps manage AI-driven traffic more efficiently. Head back to a better time, put your headphones on, pop your pills, and click play below. Video [https://go.mlops.community/k5ldmn] || Spotify [https://go.mlops.community/hf75gg] || Apple [https://go.mlops.community/efwblg] [https://go.mlops.community/efwblg](https://go.mlops.community/efwblg) [https://go.mlops.community/k5ldmn](https://go.mlops.community/k5ldmn) [https://go.mlops.community/hf75gg](https://go.mlops.community/hf75gg) ## SURVEY You're not alone dealing with data headaches in AI workflows. If you’ve ever struggled with getting the right data at the right time, it’d be great to get your input. It's a 5-minute survey [https://go.mlops.community/tat53t] - mostly just clicking. Your feedback will help shape better solutions for handling AI data. Thanks ## From Rules to Reasoning Engines ... wait, what was I doing...? Oh yeah, got distracted again. Wish I had an ambient agent to help with tasks, like George suggested when we talked about how AI-first software is shifting from automation to reasoning. * Rules vs. reasoning – the old approach relied on rigid rules, but we explored how AI systems can adapt in real-time, making software far more flexible. * Agents doing real work – instead of just assisting, AI agents are taking over tasks, from troubleshooting to writing code, freeing up engineers for bigger-picture thinking. * Rethinking enterprise tools – some teams are forcing AI into existing workflows, but others are building from scratch to make AI truly native. This shift is changing how we build and use software, and it’s happening fast. Click below to listen before you get distracted. Video [https://go.mlops.community/2b34rp] || Spotify [https://go.mlops.community/1fnved] || Apple [https://go.mlops.community/7vsfr4] [https://go.mlops.community/7vsfr4](https://go.mlops.community/7vsfr4) [https://go.mlops.community/2b34rp](https://go.mlops.community/2b34rp) [https://go.mlops.community/1fnved](https://go.mlops.community/1fnved) ## Job of the Week [https://go.mlops.community/t8ssug](https://go.mlops.community/t8ssug) ## Where's Your Pre-registration: A Physicist's Notes from the Cheap Seats on AI's Benchmarking Crisis What to me and AI benchmarking have in common? A discipline problem. For AI benchmarking, its problem is it lacks the scientific discipline seen in fields like physics. Instead of defining success upfront, we build tests, watch models ace them, and then scramble to make harder ones. The result? A cycle of meaningless scorekeeping, where models get better at tests rather than intelligence. Worse, many benchmarks are contaminated - LLMs have already seen the questions. This blog argues for: * Measuring true generalization - Can models apply knowledge to unfamiliar problems? * Pre-registering hypotheses - Define what’s being tested before running experiments. * Fixing contamination - If models train on test data, results mean nothing. Be disciplined and click below to read. Read it here [https://go.mlops.community/x760nl] [Read it here](https://go.mlops.community/x760nl) ## CALL FOR SPEAKERS You've got a lot to share, but if your dog’s glazing over every time you explain your latest project, maybe it’s time for a new audience. AI Summit London is looking for speakers this June - apply here [https://go.mlops.community/wrn03s]. ## How to Efficiently Use GPUs for Distributed Machine Learning in MLOps I appreciate the 30 minutes you spent coding a shortcut to open three apps at once instead of clicking them individually - efficiency matters, and this blog is for you. It breaks down how efficient GPU utilization is essential for distributed ML training, cutting costs and improving performance. Key topics include GPU-optimized communication, Kubernetes-based orchestration, and performance tuning. One major challenge is communication overhead between GPUs. To tackle this: * NCCL (NVIDIA) and RCCL (AMD) enable direct GPU-to-GPU transfers, reducing CPU bottlenecks. * RDMA-based data transfers bypass CPU overhead, improving scalability in multi-node setups. Kubernetes automates GPU resource management, while benchmarking collective operations ensures large-scale ML workloads run efficiently. Save time writing the Python script - just click below to read. Read it here [https://go.mlops.community/bcu806] [Read it here](https://go.mlops.community/bcu806) ## GATEWAZE GROOVES I ran the list of all your favorite artists in to ChatGPT and asked it to show when they were most popular. https://go.mlops.community/Grooves If the MLOps Community was one person, ChatGPT thinks they'd have been born in 1995. Its reasoning? "Their core music taste aligns with someone who was a teenager between 2008-2013, discovering indie rock, alt-pop, and electronic while still appreciating older classics." Not sure how it's defining peak popularity, but you can check the full list here [https://go.mlops.community/Grooves]. ## HERE TO HELP Before you go, here are three ways I can help - just hit reply: * Curated intros to other community members * What problems are you dealing with? Let me help you find the best solutions through my network * Looking to augment your staff for an MLOps or AI project? I got you covered Thanks for reading, catch you next time! Interested in partnering with us? Get in touch: partners@mlops.community Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin]. The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/]. --- Source: https://aaif.live/newsletters/mlopscommunity/2025-03-20-proxy-problems-pill-popping-and-python-s-limits