---
title: "Agent-Safe Evals Made Practical"
newsletter: "MLOps Community"
date: 2025-09-16
source: https://aaif.live/newsletters/mlopscommunity/2025-09-16-agent-safe-evals-made-practical
---

# Agent-Safe Evals Made Practical

*Inspect AI: UK AISI’s Framework for Serious LLM Evaluations*

*MLOps Community — Agentic AI Foundation, 2025-09-16*

How do you do Evals?

 * Building custom evals in-house
 * Using an open-source framework (e.g., Ragas, TruLens)
 * Using a commercial platform (e.g., Weights & Biases, Arize)
 * Mostly manual testing/prompt checks
 * Vibes

VOTE HERE [https://go.mlops.community/16SepVote]

## by

TUESDAY TOOL REVIEW: INSPECT AI

by Stephen Oladele [https://www.linkedin.com/in/stephenoladele/]

https://inspect.aisi.org.uk/

Inspect AI [https://inspect.aisi.org.uk/] is an open-source Python framework from the UK AI Security Institute (AISI) for building and running reproducible LLM evaluations.

It ships opinionated primitives (dataset → Task → Solver → Scorer), multi-turn/agent workflows with tools, sandboxed execution (Docker built-in, optional Kubernetes/Proxmox adapters), a VS Code log viewer, and a web-based Inspect View.

Install is one-liner: pip install inspect-ai (inspect-ai 0.3.130 as of Sep 7, 2025; MIT-licensed; Python ≥ 3.10).

⏩ TL;DR

 * Pick Inspect AI when you need reproducible, instrumented evals that include tool use/agents, sandboxing, and multi-provider model support.
 * It’s heavier than EleutherAI/im-evaluation-harness, but the logs, scoring options (regex → model-graded), and security/approval workflows pay off in teams.
 * Skip it for quick, single-prompt accuracy checks or tiny bake-offs.
 * ⭐ Verdict: 4.3/5 for teams running real eval programs (benchmarks, regressions, safety tests).


WHAT IS INSPECT AI (IN 2025)?

At its core, Inspect is a declarative framework for evaluating AI systems via composable “tasks.” A task might check:

 * whether a model refuses dangerous requests,
 * whether an agent leaks system instructions,
 * or whether an AI-to-AI convo converges on a correct answer.

Inspect handles:

✅ Prompt construction

✅ Response evaluation

✅ Score aggregation

✅ Logging (JSONL, Postgres, Dash UI)

✅ Isolation (via K8s or VMs)

Runs are explicit and inspectable. Everything is typed, reproducible, and introspectable, which is a must for frontier model auditing.

Backed by: UK AI Safety Institute [https://www.aisi.gov.uk/] (the team driving UK government safety standards for frontier AI).

Active user base: Adopted by METR, Apollo Research, other government AISIs, and major safety labs.


KEY FEATURES OF INSPECT AI

Here are all the core components the eval framework comes with:

 1. Tasks/Datasets/Solvers/Scorers: Tasks bring together datasets (load data), solvers that elicit behavior (single/multi-step), and scorers (score outputs).
 2. Models layer: One interface [https://inspect.aisi.org.uk/providers.html] over OpenAI, Anthropic, Google, Groq, Mistral, xAI (Grok), AWS Bedrock/AI Inference, Azure AI, Together, Cloudflare, Goodfire, plus local vLLM/Ollama/llama-cpp.
 3. Agents: Built-in ReAct, multi-agent composition [https://inspect.aisi.org.uk/agents.html], external agent bridge (e.g., AutoGen/LangChain).
 4. Tools: Built-ins (bash, python, text-edit, web_search, web_browser, computer) + MCP/Custom tools [https://inspect.aisi.org.uk/tools-standard.html].
 5. Sandboxing and safety: Run untrusted code and browsers in Docker [https://inspect.aisi.org.uk/sandboxing.html#sec-docker-configuration], K8s [https://k8s-sandbox.aisi.org.uk/] (pods per sample), or Proxmox [https://github.com/UKGovernmentBEIS/inspect_proxmox_sandbox] with sandbox extensions [https://inspect.aisi.org.uk/sandboxing.html] and optional domain/network controls; Tool Approval (human-in-the-loop or policy-based gating).
 6. Dev Experience: Inspect View [https://marketplace.visualstudio.com/items?itemName=ukaisi.inspect-ai] (web log viewer for basic observability) and VS Code extension for run/debug/tune.
 7. Scale Knobs: Caching, batch mode, parallelism, eval-set slicing, retry/resume.
 8. Evals Registry: Dozens of canonical evals [https://inspect.aisi.org.uk/eval-sets.html] (reasoning, coding, agent tasks, and cybersecurity).

Bonus: AISI publishes add-on packages like inspect-cyber for agentic cyber evals (see PyPI [https://pypi.org/project/inspect-cyber/]).


HOW INSPECT AI WORKS (UNDER THE HOOD)

 1. Define Tasks: JSON/CSV/HF datasets feed Task objects; mix single-turn Q&A, coding, multi-modal, or open-tool agent prompts.
 2. Elicit solutions: inspect_ai.solver provides chain-of-thought, self-critique, tool-use, and MCP calls.
 3. Execute securely: Choose sandbox back-ends: process jail, Docker, or the community K8s sandbox [https://github.com/UKGovernmentBEIS/inspect_k8s_sandbox] for scale and isolation.
 4. Score and aggregate: Out-of-the-box model-graded QA, F1, pass@k, statistical bootstrap, plus custom metrics.
 5. Analyze: Transform logs into evals_df, samples_df, events_df for Pandas-like slicing or Inspect Viz dashboards.

Each Task bundles:

 * a prompt template (e.g., a jailbreak or math question),
 * a model runner (e.g., OpenAI, HuggingFace, Ollama, local endpoint),
 * a scoring function (numeric, boolean, class).

Example CLI command:

Shell inspect eval my_eval.py --model openai/gpt-4o --limit 100


QUICK SPIN-UP (HELLO WORLD EXAMPLE)

1. Install and pick a model provider


Shell pip install inspect-ai openai export OPENAI_API_KEY=your-key

2. Hello-World eval (exact match + single call)

Python from inspect_ai import Task, task from inspect_ai.dataset import Sample from inspect_ai.solver import generate from inspect_ai.scorer import exact @task def hello_world(): return Task( dataset=[Sample(input="Just reply with Hello World", target="Hello World")], solver=[generate()], scorer=exact(), ) Shell inspect eval hello.py --model openai/gpt-4o # You can also write a Parquet log you view live with the GUI # and can also slice in Pandas for basic monitoring using the # commands: # inspect eval theory.py --model openai/gpt-4o --log_dir runs/hello-world # inspect view runs/hello-world # live GUI

3. Model-graded scoring (use an LLM to judge correctness):

Python from inspect_ai.scorer import model_graded_fact # ... Task(dataset=..., solver=[...], # scorer=model_graded_fact())

Examples for multi-choice (HellaSwag), GSM8K few-shot math, and custom math equivalence scorers are in the docs tutorial [https://inspect.aisi.org.uk/tutorial.html].


WHY MLOPS AND AI SAFETY TEAMS SHOULD CARE

 1. Single CLI for multi-vendor evals: No more bespoke scripts per provider.
 2. Agent and tool safety: Isolated K8s pods or VM sandbox plugins let you run ReAct or Auto-GPT-style agents in locked pods. Includes policy limits for tokens [https://inspect.aisi.org.uk/reference/inspect_ai.util.html], clock- and working-time [https://inspect.aisi.org.uk/reference/inspect_ai.html] (CPU time spent inside model/tool calls), messages [https://inspect.aisi.org.uk/options.html], and sandbox parallelism/retries.
 3. Model-graded scoring: A unified scoring library [https://inspect.aisi.org.uk/reference/] with bootstrap CIs and pass/fail gates that is vastly cheaper than human eval yet richer than Exact Match [https://huggingface.co/spaces/evaluate-metric/exact_match].
 4. Rich observability: Inspect View + Inspect Viz render per-step traces, costs, and scorer heatmaps. You can also log every prompt, response, and token count for audit.
 5. Ecosystem: k8s-sandbox, Hugging Face dataset bridges, SWE-Bench datasets, active Slack community [https://join.slack.com/t/inspectcommunity/shared_invite/zt-38v1m03v9-XxQg2QLocyAZLKBemqHqEg], etc.

I found out that you can also add solvers, scorers, or entire toolchains via the extensions registry [https://inspect.aisi.org.uk/extensions.html].


GOTCHAS AND WORKAROUNDS FOR INSPECT AI (OPEN SOURCE)


COMMUNITY PULSE

“We use https://github.com/UKGovernmentBEIS/inspect_ai, which is great and comes with model-graded scorers. We also built support for reporting the results back to Langtrace AI, which we use for observability.” — Karthik Kalyanaraman, Slack thread [https://mlops-community.slack.com/archives/C04T55KFV8S/p1720015554729939?thread_ts=1720014591.699139&cid=C04T55KFV8S] (July 2025)

“Inspect also has its own very active Slack community [https://join.slack.com/t/inspectcommunity/shared_invite/zt-38v1m03v9-XxQg2QLocyAZLKBemqHqEg], and the Inspect userbase includes other safety research organisations (other AISIs, Apollo, METR) as well as some of the frontier labs.” — Jason Gwartz (Head of Platform, UK AISI), Slack thread [https://mlops-community.slack.com/archives/C010A328X38/p1752767404547629?thread_ts=1752685284.759859&cid=C010A328X38] (July 2025)


REAL-WORLD USE CASE: AGENTIC SAFETY RED-TEAM

 1. Dataset: 500 complex multi-step jailbreak prompts.
 2. Solver chain: ReAct agent → tool-calling → self-critique.
 3. Sandbox: k8s_sandbox spins a pod per sample, capturing file/HTTP output.
 4. Scorer: model-graded policy checker + custom regex.
 5. Outcome: Run 500 × 4 models overnight; Inspect Viz dashboard highlights policy breaks with stack traces.

The agent is allowed to browse/execute code inside a K8s sandbox with domain allowlists while completing capture-the-flag tasks; this reduces blast radius if the model tries risky actions. (See Cybench guide [https://ukgovernmentbeis.github.io/inspect_evals/evals/cybersecurity/cybench/].)


HOW INSPECT AI STACKS UP AGAINST OTHER EVAL TOOLS (2025)


FINAL VERDICT: “PRODUCTION-GRADE EVALS & AGENT SANDBOXES, BATTERIES INCLUDED.”


RATING: ⭐⭐⭐⭐☆ (4.3/5)

If your org treats evals as production infrastructure (with agents, tools, and safety constraints), Inspect AI is the most complete OSS choice today.

The learning curve and runtime overhead are real, but the auditability, sandboxing, and scorer depth justify it for serious teams.

✅ Ship it if…

 * You run recurring evals/regressions across multiple vendors or models (cloud + local).
 * You need to evaluate agents using tools with isolation (Docker/k8s) and approval policies.
 * You want LLM-graded or domain-specific scoring beyond Exact Match.
 * Regulatory or SOC2 demands tamper-proof logs and stats.

❌Hold off if…

 * You’re doing a one-week bake-off on static benchmarks. Lighter harnesses will be faster.
 * You don’t plan to test tool use/agents (Inspect’s agent tasks can be spendy), and simple text-in/text-out scoring suffices.
 * You can’t support even Docker-level isolation yet (agentic evals will be risky).
 * You dislike declarative DSLs (lm-eval-harness may feel lighter).


STARTER COMMANDS YOU’LL ACTUALLY USE

 * Run an eval: inspect eval my_eval.py --model anthropic/claude-3-5-sonnet-20240620 --limit 100 --log-dir ./logs
 * View logs: inspect view (web UI) or open the Inspect panel in VS Code.
 * Enable sandbox: --sandbox docker or --sandbox k8s:config.yml (domain/network policy via K8s).
 * Repro tips: set --seed, use caching, and pin model versions.


RESOURCES TO LEARN MORE

 * Docs [https://inspect.aisi.org.uk/]/Getting Started/Install.
 * Model Providers [https://inspect.aisi.org.uk/providers.html] (OpenAI, Anthropic, Google, xAI, Mistral, HF, Azure, AWS, …).
 * Tutorials [https://inspect.aisi.org.uk/tutorial.html] and Examples (Hello World, GSM8K, HellaSwag, Math, Tools, CTF).
 * Sandboxing [https://inspect.aisi.org.uk/sandboxing.html] (Docker built-in; Kubernetes/Proxmox adapters).
 * Logs and Viewers [https://inspect.aisi.org.uk/log-viewer.html] (VS Code + Web).
 * Inspect Evals [https://github.com/UKGovernmentBEIS/inspect_evals] collection (incl. SWE-bench variants).
 * PyPI [https://pypi.org/project/inspect-ai/] (latest version, license).

Love this review? Share it in MLOps Slack or forward to your EvalOps lead, or share on X with #TuesdayToolReview and tag @mlopscommunity

Interested in partnering with us? Get in touch: partners@mlops.community

Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin].

Thanks again to Stephen Oladele [https://www.linkedin.com/in/stephenoladele/], the writer of The Neural Blueprint [https://neurlcreators.substack.com/],for his contribution.

---
Source: https://aaif.live/newsletters/mlopscommunity/2025-09-16-agent-safe-evals-made-practical