--- title: "Claude vs. Pikachu: What It Taught Us About LLMs" newsletter: "MLOps Community" date: 2025-03-27 source: https://aaif.live/newsletters/mlopscommunity/2025-03-27-claude-vs-pikachu-what-it-taught-us-about-llms --- # Claude vs. Pikachu: What It Taught Us About LLMs *Plus, ML Confessions, responsible builds, reproducible builds, secure builds, and Hidden Gems.* *MLOps Community — Agentic AI Foundation, 2025-03-27* Should’ve seen the reaction from the gym bros when I told them what I'm benching [https://go.mlops.community/gupn3l]. ## Claude Plays Pokémon - A Conversation with the Creator All work and no play apparently applies to models too. David built Claude Plays Pokémon as a way to test long-horizon coherence and agent behaviour. It uses screenshots, metadata, and a simple toolset to move through the game – and became a surprisingly useful benchmark as models improved. We also got into when fine-tuning is actually useful, and why prompting or RAG usually get you further, faster: * Prompting covers most needs if pushed properly * RAG adds knowledge without complexity * Fine-tuning is costly and rarely worth it unless precision or scale demand it Click below to listen and prevent your model having a Jack Nicholson Shining moment. Video [https://go.mlops.community/f9bk42] || Spotify [https://go.mlops.community/z8lqa4] || Apple [https://go.mlops.community/d105pb] [https://go.mlops.community/d105pb](https://go.mlops.community/d105pb) [https://go.mlops.community/f9bk42](https://go.mlops.community/f9bk42) [https://go.mlops.community/z8lqa4](https://go.mlops.community/z8lqa4) ## Building Trust Through Technology: Responsible AI in Practice I appreciate 'responsible' might not be the first word you associate with me, but don't let that put you off this episode. Allegra talked about how engineers are often expected to carry the weight of building 'responsible AI' systems, despite a lack of clear definitions or leadership support. Having clarity on intent, involving diverse perspectives early, and codifying guiding principles into technical requirements can make things more concrete. We also spent time on “perspective density” – the importance of having more varied voices in the room: * It shapes better questions * Highlights unseen risks * Leads to more robust systems Grab a coffee, hit play, and drink responsibly. Video [https://go.mlops.community/w09b3s] || Spotify [https://go.mlops.community/k6pi3r] || Apple [https://go.mlops.community/sg6kak] For more on this, check out DataCamp's [https://go.mlops.community/DataCamp]* Responsible AI Practices course. *To show how responsible I am, I'll point out this is an affiliate link and clicking helps support the newsletter. [https://go.mlops.community/sg6kak](https://go.mlops.community/sg6kak) [https://go.mlops.community/w09b3s](https://go.mlops.community/w09b3s) [https://go.mlops.community/k6pi3r](https://go.mlops.community/k6pi3r) ## Job of the Week [https://go.mlops.community/173b9c](https://go.mlops.community/173b9c) [https://go.mlops.community/MLConfess](https://go.mlops.community/MLConfess) ## More Automation + More Reproducibility = MLOps Python Package v4.1.0 Like that half-court buzzer beater you somehow sank in high school, some things just aren’t reproducible. Your build doesn’t have to be one of them. v4.1.0 of the MLOps Python Package improves reproducibility and automation with a switch from PyInvoke to Just, offering cleaner task definitions for linting, testing, packaging, and more. Builds are now fully deterministic using constraints.txt compiled with uv , locking dependency versions to avoid CI inconsistencies. The new packaging flow uses just to: * Generate constraints.txt with exact hashes * Build wheels with --require-hashes enforcement * Ensure repeatable, secure installs across environments Other updates include Gemini Code Assist for PR review and automatic GitHub ruleset enforcement. Give it a read to make sure your build's a slam dunk every time. Read it here [https://go.mlops.community/1ubqhw] [Read it here](https://go.mlops.community/1ubqhw) ## Hidden Gems ## Is Your Chatbot Secure? Your $1 Chevrolet shows you know how to spot an insecure chatbot when you see one – but how do you make sure your own chatbot’s not next? This post explains how to build secure RAG systems by combining Realm’s permission-aware data connectors with ApertureDB’s graph-vector database. It tackles key enterprise challenges like syncing fast-changing permissions and enforcing fine-grained access control at query time. Rather than relying on flat ACLs, ApertureDB models access paths as graph relationships: * Users, groups, and files are nodes, with edges showing permissions and membership * Access checks traverse the graph to validate whether a user can retrieve a chunk * Updates are efficient, needing only local edge changes when permissions shift Have a read to make sure your chatbot isn't the next newsletter punchline. Read it here [https://go.mlops.community/8s3r2v] [Read it here](https://go.mlops.community/8s3r2v) ## HERE TO HELP Before you go, here are three ways I can help - just hit reply: * Curated intros to other community members * What problems are you dealing with? Let me help you find the best solutions through my network * Looking to augment your staff for an MLOps or AI project? I got you covered Thanks for reading, catch you next time! Interested in partnering with us? Get in touch: partners@mlops.community Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin]. The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/]. --- Source: https://aaif.live/newsletters/mlopscommunity/2025-03-27-claude-vs-pikachu-what-it-taught-us-about-llms