--- title: "Count Outcomes, Not Tokens" newsletter: "User Community" date: 2026-09-17 source: https://aaif.live/newsletters/usercommunity/2026-09-17-count-outcomes-not-tokens --- # Count Outcomes, Not Tokens *Plus… persistent memory, coding-agent decay, and least-privilege MCP* *User Community — Agentic AI Foundation, 2026-09-17* Depending on when you stop for your coffee break to read this, we might already be live from Amsterdam. We’re broadcasting AGNTCon + MCPCon Europe for 36 hours straight - streaming the talks, wandering the expo floor, grabbing speakers and builders for chats, keeping things going after the conference closes, and seeing what happens through the night. Come hang out with us [https://www.youtube.com/watch?v=LbsBVem-VcE] over the next couple of days! ## Gems ## What’s happening across the chapters ## LOCAL ORGANIZERS Three more AAIF chapters held their first meetups last week, with Seoul, Toronto and Colombo [https://www.linkedin.com/posts/onwuka-ijeoma_this-week-at-the-agentic-ai-foundation-we-ugcPost-7504212810755874816-QX0Y/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAEecljcBOYrbIWFw_V8Cfdg-WggzChLgFAA] all bringing new local communities together. Toronto covered evals for Kubernetes agents, enterprise control planes and model routing, including one Jira example that cut input tokens from 13,514 to 1,025. Colombo ranged from MCP auth and graph-based grounding to WebMCP voice agents and a tool-design approach that cut one MCP server by 40%. Chennai [https://www.linkedin.com/posts/suryaprakash-lakshmanan-068a7684_agenticai-opensearch-chennaitech-activity-7504966032747442176-4Bw3?utm_source=share&utm_medium=member_desktop&rcm=ACoAAEecljcBOYrbIWFw_V8Cfdg-WggzChLgFAA] also kicked off its chapter with a strong production focus, including hybrid retrieval, permission-aware search, OpenSearch observability and tracing the full prompt lifecycle with Jaeger and Langfuse. There’s more taking shape too: first events are on the way in Shenzhen [https://luma.com/tvnwtdq6] and Bengaluru [https://luma.com/igokwrsx], while Agentic Tokyo #2 [https://www.linkedin.com/posts/satoshi-ito_agentictokyo-aaif-agenticai-activity-7505198758192869376-D82Q?utm_source=share&utm_medium=member_desktop&rcm=ACoAAEecljcBOYrbIWFw_V8Cfdg-WggzChLgFAA] brought the local community together with AAIF visitors for another round of agents, open source and shared lessons. ## Verification and recovery ## LUNCH AND LEARN Last week, Eric Bigelow from Goodfire joined us to look at uncertainty in LLM reasoning, including the points where a reasoning chain can suddenly change course and how to find them with far fewer samples. Read the session notes [https://learn.mlops.community/wp-content/uploads/2026/09/Lunch-and-Learn-Session-24-asset.pdf] This week, Tanmay Sah and Dolly Sah join us on September 18 at 09:00 PDT / 18:00 CET to talk about verification and recovery in coding agents, from the tradeoffs of checking an agent’s work to recovering after a harmful or incorrect change. Join us [https://home.mlops.community/home/events/coding-agents-lunch-and-learn-session-25-verification-and-recovery-jqvl0d239q?agenda_day=6a9844b73e654eb248880888&agenda_track=6a9844b83e654eb24888089d&agenda_stage=6a9844b73e654eb24888088d&agenda_filter_view=stage&agenda_view=list] ## Building the next voice agents ## FREE VIRTUAL EVENT What does it take to move voice AI from a slick demo to something fast, reliable, and ready for production? Join us September 16 for 90 minutes on voice-agent architecture, infrastructure, and open standards, including UNMUTE, a new MIT-licensed standard for voice agents, plus a live look at how semantic caching, CDNs, and edge infrastructure can cut latency in voice pipelines. September 16 · 08:30 PDT / 17:30 CEST JOIN LIVE [https://home.mlops.community/home/events/voice-agent-virtual-event-bnzc6hiff8] ## 36 hours live from Amsterdam ## AGNTCON + MCPCON EUROPE TO ADD ## The first official MCP certification is live ## NEW FROM AAIF AAIF and Linux Foundation Education have released the Model Context Protocol Associate (MCPA), a vendor-neutral certification based on the 2026-07-28 MCP spec. Candidates answer questions on client-server interactions and the tool invocation lifecycle, with 24% of the exam focused on security and governance. See exam details [https://training.linuxfoundation.org/certification/model-context-protocol-associate-mcpa/] ## Prompt Injection as Role Confusion ## READING GROUP This month’s reading group looked at research showing that models can infer who is speaking from writing style rather than reliably following role tags. With forged reasoning written in a model’s own style, attack success rose from under 4% to over 80% on several of the six models tested. Lucas Pavanelli walked through the ICML 2026 paper, before Sparsh Jain took the discussion into reasoning-trace extraction and agent red-teaming. Read the session notes and checklist [https://learn.mlops.community/wp-content/uploads/2026/09/Reading-Group-Promp-Injection-as-Role-Confusion-asset.pdf] ## Why cost per million tokens is a useless KPI Agentic AI can turn predictable usage into runaway spend: each retry may trigger more model calls, RAG lookups, vector retrieval, memory, and egress. That makes cost per million tokens a weak KPI, because identical token volumes can represent very different workloads and value. * Cost models are shifting toward adoption rate and cost per user, with separate baselines for engineering, product, and internal use cases. * Real-time gateways can cap retries, route models, apply fallbacks, and enforce cost and security policies before loops spiral. * Value needs separate metrics, from DORA-style delivery measures to hours saved or support workload reduced. The better unit is cost tied to a specific persona, workload, and outcome. [https://podcasts.apple.com/gb/podcast/why-cost-per-million-tokens-is-a-useless-kpi/id1505372978?i=1000789572820](https://podcasts.apple.com/gb/podcast/why-cost-per-million-tokens-is-a-useless-kpi/id1505372978?i=1000789572820) [https://home.mlops.community/home/videos/why-cost-per-million-tokens-is-a-useless-kpi-2026-09-14](https://home.mlops.community/home/videos/why-cost-per-million-tokens-is-a-useless-kpi-2026-09-14) [https://open.spotify.com/episode/0YN5yciatlDBewmLqgfRD2?si=kPZ8qGkRTOG_65vwvh964w](https://open.spotify.com/episode/0YN5yciatlDBewmLqgfRD2?si=kPZ8qGkRTOG_65vwvh964w) ## Same door. Very different walk in. An AI coding agent reported every unit test passed - because the tests were mocked so thoroughly that nothing meaningful ran. That failure frames a broader problem: agents can retrieve facts without preserving the context and relationships needed to use them safely. * Nexus stores semantic and episodic memory together, combining graph and vector search with multimodal data. * Each memory commit carries who, what, when, why, and how, while lineage and query-time permissions remain separate trust requirements. * Open gaps include recency-aware updates, permission enforcement, GraphRAG, hybrid search, and propagating corrected context across agents. The architecture shows why persistent agent memory needs more than embeddings: relationships, provenance, governance, and update semantics all matter. [Read the blog](https://mlops.community/blog/same-door-very-different-walk-in) ## The Winchester Mystery House Problem in AI development A workflow that cost $1 per 1,000 records at 90% accuracy reached 95% accuracy while costing less, after an optimizer moved 75% of the work out of the model and into code. That result sits inside a broader discussion about when agentic behavior should become a defined workflow. * Models increasingly inherit assumptions from the coding harnesses they were trained around, which can make custom harnesses harder to build. * Repeated agent tasks can often be “crystallized” into smaller models or conventional code once the process is understood. * DSPy separates task definitions from implementation, allowing prompts, models, and even harness code to be optimized without redefining the task. The practical direction is toward identifying which tasks still need open-ended agents and which are mature enough to become cheaper, more reliable workflows. [https://podcasts.apple.com/us/podcast/the-winchester-mystery-house-problem-in-ai-development/id1505372978?i=1000785551986](https://podcasts.apple.com/us/podcast/the-winchester-mystery-house-problem-in-ai-development/id1505372978?i=1000785551986) [https://home.mlops.community/home/videos/the-winchester-mystery-house-problem-in-ai-development](https://home.mlops.community/home/videos/the-winchester-mystery-house-problem-in-ai-development) [https://open.spotify.com/episode/4IjjWOjSq8XQPAXLq6ApRQ?si=TnlufikXRKqxS8ZHSpyYCQ](https://open.spotify.com/episode/4IjjWOjSq8XQPAXLq6ApRQ?si=TnlufikXRKqxS8ZHSpyYCQ) ## Dex still thinks long-running coding agents stink Coding agents can ace bounded tasks and still wreck a codebase over time. In SlopCodeBench, every tested model accumulated defects across successive requirements, while a lightly supervised internal experiment ended with 30,000–40,000 lines of code being archived. * Across six challenges and 30 checkpoints, Fable and Sol each managed 10 strict passes, or 33.3%. * The failure mode is compounding “slop”: agents inherit weak patterns from earlier code and turn them into precedent. * Human review remains the bottleneck because architecture and “taste” failures are harder to verify than syntax, tests, or task completion. The key challenge is building stronger verifiers that can catch quality decay before it becomes architecture. [Read the blog](https://aaif.io/blog/dex-still-thinks-long-running-coding-agents-stink) ## From supervised shells to purpose-built MCP tools A client asked an MCP server to run shell_exec and got an unknown-tool error - exactly the boundary a general-purpose shell cannot provide. The test used a Tesla T4 to show how a purpose-built GPU server can narrow agent access without pretending the tool schema solves everything. * Four typed GPU operations exposed inventory, metrics, summaries, and process data; invalid selectors and unregistered tools were rejected. * Under CUDA load, utilization reached 100%, while NVML and PyTorch reported different memory totals because they measure different things. * Least privilege still depends on process identity, device visibility, environment, transport, and logging. The practical lesson is to narrow the MCP contract, the runtime process, and the evidence trail together. [Read the blog](https://aaif.io/blog/from-supervised-shells-to-purpose-built-mcp-tools) ## IN-PERSON EVENTS ## Come and connect * AGNTCon + MCPCon Europe [https://events.linuxfoundation.org/agntcon-mcpcon-europe/] - September 17 * Luxembourg [https://luma.com/4ciica7u] - September 17 * Singapore [https://luma.com/gspnb7f6] - September 17 * Munich [https://luma.com/fkwvz7p3] - September 22 * Shenzhen [https://luma.com/tvnwtdq6] - September 23 * Amsterdam [https://luma.com/tavily-7u0e] - September 25 * Bengaluru [https://luma.com/igokwrsx] - September 26 Find your city here [https://aaif.io/events?tab=community], or start a chapter if there isn't one yet. ## VIRTUAL EVENTS ## Join from anywhere * 36 Hours Live from AGNTCon + MCPCon Europe [https://www.youtube.com/watch?v=LbsBVem-VcE] - September 17 - 18 * Lunch and Learn Session 24 [https://home.mlops.community/home/events/coding-agents-lunch-and-learn-session-25-verification-and-recovery-jqvl0d239q?agenda_day=6a9844b73e654eb248880888&agenda_track=6a9844b83e654eb24888089d&agenda_stage=6a9844b73e654eb24888088d&agenda_filter_view=stage&agenda_view=list] - September 18 --- Source: https://aaif.live/newsletters/usercommunity/2026-09-17-count-outcomes-not-tokens