---
title: "Agent A/B Tests: Crash or Cash"
newsletter: "MLOps Community"
date: 2025-09-11
source: https://aaif.live/newsletters/mlopscommunity/2025-09-11-agent-a-b-tests-crash-or-cash
---

# Agent A/B Tests: Crash or Cash

*Plus new tricks to bound hallucinations, dm-cache performance hacks, and OpenAI’s hidden reasoning shift*

*MLOps Community — Agentic AI Foundation, 2025-09-11*

Powerful insight from Altman [https://go.mlops.community/intro11sep] that may shake up your aunt’s ‘positive vibes only’ groups.

## The Memory Trap

Memory isn't a feature, it's lock-in. Users don't stay for your clever prompts - they stay because switching means losing their history.

What do you think?

## Hidden Gems

## Job of the Week

[https://go.mlops.community/jobsnl](https://go.mlops.community/jobsnl)

## Before Building AI Agents Watch This (Deep Agent Expertise)

Nishikant’s team shipped a slick shopping agent - and the A/B test tanked. The recovery hinged on context, search, and disciplined evals.

 * Context engineering over model swaps: pull real-time promos, opening hours, payment options, and user history into the prompt via pipelines.
 * Hybrid search: keyword for exact matches, semantic for intent, with LLM-led query understanding up front and LLM re-ranking on user context.
 * Evals and UI: track carts/conversion first, then LLM-as-judge, plus labeling parties; pair chat with timely UI widgets.

Do this, and the next A/B lifts instead of craters.

Video [https://go.mlops.community/pnd11sep] || Spotify [https://go.mlops.community/snd11sep] || Apple [https://go.mlops.community/and11sep]


LLM EVALUATION: PRACTICAL TIPS AT BOOKING.COM

When your LLM is hallucinating and costs are piling up, you need a way to measure what’s really happening. Booking.com shares a year of lessons building their Judge-LLM framework for large-scale evaluation.

 * Building a “golden dataset” that mirrors production, with strict annotation protocols to ensure reliability.
 * Iteratively engineering prompts to make strong models judge weaker ones, enabling scalable monitoring.
 * Balancing accuracy with cost by deploying lighter judge models for production tracking.

A practical blueprint for anyone serious about dependable LLM evaluation.

Read the blog [https://go.mlops.community/blog11sep]


IN PERSON EVENTS

 * Frankfurt [https://luma.com/lntqh2kq] - September 11
 * Lisbon [https://luma.com/bszaz8e3] - September 11
 * London [https://lu.ma/fc1cm6v5] - September 18
 * Austin [https://lu.ma/rnxflhkz] - September 18
 * Seattle [https://luma.com/g42ppkok] - September 25
 * Denver [https://luma.com/kebs46ij] - September 25
 * Miami [https://luma.com/49ev1qh5] - September 25

[Video](https://go.mlops.community/pnd11sep)

## ROC Curves Don’t Make Phone Calls

[https://go.mlops.community/MLConfess](https://go.mlops.community/MLConfess)

## Making the hard stuff simpler

Working on something tricky or planning ahead? Here’s how we can help - just hit reply:

 * Custom workshops tailored to your company’s needs
 * Hiring? I know some quality folks looking for a new adventure
 * Want to connect with someone tackling similar problems? I can introduce you

Thanks for reading, catch you next time!

---
Source: https://aaif.live/newsletters/mlopscommunity/2025-09-11-agent-a-b-tests-crash-or-cash