--- title: "Voice Agents, Hard Conversations" newsletter: "MLOps Community" date: 2026-05-07 source: https://aaif.live/newsletters/mlopscommunity/2026-05-07-voice-agents-hard-conversations --- # Voice Agents, Hard Conversations *MLOps Community — Agentic AI Foundation, 2026-05-07* As the saying goes, let 'em cook [https://techcrunch.com/2026/05/05/marc-lore-says-that-ai-will-soon-enable-anyone-open-a-restaurant/]. ## Tier Pressure Most production agents cap at Level 1 because nobody can extract what the Level 2 humans know. What's the real bottleneck: Models or Tacit Knowledge? [MODELS](https://gatewaze.mlops.community/offer/surveys/?sid=yesno&question=Most+production+agents+cap+at+Level+1+because+nobody+can+extract+what+the+Level+2+humans+know.What%27s+the+real+bottleneck%3A+Models+or+Tacit+Knowledge%3F&y=MODELS&n=KNOWLEDGE&oneclick=true&accept=true) [KNOWLEDGE](https://gatewaze.mlops.community/offer/surveys/?sid=yesno&question=Most+production+agents+cap+at+Level+1+because+nobody+can+extract+what+the+Level+2+humans+know.What%27s+the+real+bottleneck%3A+Models+or+Tacit+Knowledge%3F&y=MODELS&n=KNOWLEDGE&oneclick=true&accept=true) ## Specs Appeal A tough week for agents needing a protocol spec to do basic work. ## Stop Building Goldfish Agents Without persistent memory, every interaction with your agent starts from zero. Users repeat themselves, preferences vanish between sessions, and there's no way to build the kind of context that makes an agent useful over time. Over four hours, you'll build a Python agent with persistent memory using Oracle AI Database and LangChain, implementing the full pipeline: extract, store, retrieve, inject, forget. You'll cover structured memory, vector-based semantic recall, and lightweight forgetting strategies so context stays useful instead of noisy. Two hands-on coding sessions bookend a masterclass keynote from Colin Francis (LangChain) on memory engineering for production agents. You'll leave with a working notebook, a starter repo, and patterns you can apply to your own systems. Register here [Register here](https://luma.com/p8d635rz) [https://luma.com/p8d635rz](https://luma.com/p8d635rz) ## Context Engineering for Coding Agents - Fausto's Amsterdam workshop Fausto's workshop focused on the part of agent systems you can control: what gets injected into the context window, when it gets loaded, and what should live somewhere else. You are not training the model on a daily basis, so the engineering work shifts to managing context, memory, tools, and retrieval. His rule of thumb was to keep context usage under 25%, regardless of whether you are working with a 200k or 1M token model. Past that, things get slower, more expensive, and more error-prone, with cleaner options than letting the context window bloat. The lens for the rest of the talk was a brain analogy. Attention is finite, memory shapes attention, and knowledge does not get embedded in a vacuum. An agent will only notice the right things if you have given it the right priors. From there, Fausto split context into three categories. Deterministic context includes CLAUDE.md [http://CLAUDE.md], project rules, hooks on lifecycle events, auto-memory, and scheduled loops. His point on rules was especially practical: coding conventions belong in path-scoped or file-extension-scoped rules, not dumped into CLAUDE.md [http://CLAUDE.md]. Human context covers chat turns, slash commands, and references. Probabilistic context covers sub-agents, retrieval, MCPs, skills, and observers. Sub-agents are useful because they do not inherit CLAUDE.md [http://CLAUDE.md], memory, or the default system prompt, which makes them better suited for specific non-coding tasks. Skills are markdown plus optional scripts, and can also wrap calls to other models when Claude cannot handle the job. Fausto's example was routing native video analysis through Gemini. Two practical tips came out of this section. Turn on deferred tool loading, a single flag that reduces what gets injected at session start. And lean toward project scope over user scope for skills and MCPs, so the agent has the full descriptions it needs to choose well at runtime. The second half made the case for treating long-term memory as a folder of markdown files rather than defaulting to a vector store, inspired by Karpathy's wiki memory idea. The structure is an index plus raw source files, processed summaries, and a policy that decides what to ingest and retrieve. Concepts that recur get weighted up. Concepts that go unused decay over time. An observer agent watches the session and either pulls relevant knowledge into the active context or pushes new findings into the wiki. To show the difference in practice, Fausto ran two Claude Code sessions on the same cellular automata task. Same model, same skills, same sub-agents, same CLAUDE.md [http://CLAUDE.md]. The only difference was that one had a populated wiki and the other did not. Under a five-minute timer, the wiki run pulled the concepts it needed and produced a working visualization. The default run fell back on parametric memory and live search, then ran out of time. Watch the video [https://youtu.be/jXtnhyro-QE] || View the slides [https://presentation-astro-xi.vercel.app/present?start=title] ## Voice Agent Use Cases Voice agents do not fail because the voice sounds bad. They fail when silence, transcription errors, bad turn-taking, and slow tools make the whole system feel broken. * Voice adds failure modes chat does not have, including background noise, accents, diarization, and awkward pauses that make users hang up. * The best systems often use multiple models, with smaller foreground models masking latency while larger models or slower APIs work in the background. * Cascaded architectures still matter because enterprises need control, fallbacks, and the ability to swap components when quality or reliability breaks. The hard part is making all those moving pieces feel like one natural conversation. [https://podcasts.apple.com/gb/podcast/voice-agent-use-cases/id1505372978?i=1000765639638](https://podcasts.apple.com/gb/podcast/voice-agent-use-cases/id1505372978?i=1000765639638) [https://home.mlops.community/home/videos/voice-agent-use-cases](https://home.mlops.community/home/videos/voice-agent-use-cases) [https://open.spotify.com/episode/1OEBtmJicDV2Uyux1Jq09l?si=66b1a29e9941496d](https://open.spotify.com/episode/1OEBtmJicDV2Uyux1Jq09l?si=66b1a29e9941496d) ## Hallucinations in LLMs Are Not a Bug in the Data Hallucination may be less like a model going blank and more like a model turning away from the right answer. * The blog argues that, during factual errors, the residual stream does not lose information. It follows a different direction through representation space while traveling a similar distance. * A proposed “commitment ratio” shows probability mass being pushed away from the correct token, rather than simply failing to accumulate. * The practical claim is that hallucination monitors should be domain-specific, since geometric signatures degrade across tasks and knowledge areas. The uncomfortable bit is that fluent wrong answers may be a structural outcome of next-token prediction, not a data-cleaning failure. [Read the blog](https://home.mlops.community/home/blogs/hallucinations-in-llms-are-not-a-bug-in-the-data) ## IN PERSON EVENTS * Amsterdam [https://luma.com/l8tnzkqx] - May 7 * New York [https://luma.com/h8muplvs] - May 11 * San Francisco [https://luma.com/kejjqoj0] - May 12 * San Francisco [https://luma.com/p8d635rz] - May 15 * Paris [https://luma.com/a2swgs9x] - May 27 * London [https://luma.com/9k4ohvyj] - May 28 ## VIRTUAL EVENTS * Coding Agents Lunch & Learn [https://home.mlops.community/home/events/coding-agents-lunch-and-learn-session-10-from-claude-design-to-code-vdpmyurnow?agenda_day=69f8ca374494382082de73fe&agenda_track=69f8ca384494382082de7415&agenda_stage=69f8ca374494382082de7404&agenda_filter_view=stage&agenda_view=list] - May 8 * Roundtable: Architecting Modern AI Systems [https://home.mlops.community/home/events/architecting-modern-ai-systems-platforms-agents-and-integration-tg5b4fngqb?agenda_day=69ea4c63bdbee16b0254d5e2&agenda_track=69ea4c64bdbee16b0254d618&agenda_stage=69ea4c63bdbee16b0254d5e6&agenda_filter_view=stage&agenda_view=list] - May 27 ## Compliant [https://forms.gle/8EDvXGizxyFVKfwy8](https://forms.gle/8EDvXGizxyFVKfwy8) --- Source: https://aaif.live/newsletters/mlopscommunity/2026-05-07-voice-agents-hard-conversations