---
title: "Tool Tuesday #5: Build Voice Agents with OpenAI Agent SDK"
newsletter: "MLOps Community"
date: 2025-04-08
source: https://aaif.live/newsletters/mlopscommunity/2025-04-08-tool-tuesday-5-build-voice-agents-with-openai-agent-sdk
---

# Tool Tuesday #5: Build Voice Agents with OpenAI Agent SDK

*MLOps Community — Agentic AI Foundation, 2025-04-08*

Your morning pick-me-up just got an upgrade.

The Vibe Mug [https://go.mlops.community/VibeMug]: whatever you drink, whenever you're vibing.

## Voice Agents

TOOL TUESDAY #5: BUILD VOICE AGENTS WITH OPENAI AGENT SDK

What if your next AI agent could talk?

This week’s Tool Tuesday looks at how to build Voice Agents using the OpenAI Agent SDK [https://openai.github.io/openai-agents-python/]—blending LLM reasoning with real-time speech capabilities.

Forget typing. Voice is the future of human-computer interaction, and OpenAI’s latest voice models (like gpt-4o-transcribe [https://platform.openai.com/docs/models/gpt-4o-transcribe] and gpt-4o-mini-tts [https://platform.openai.com/docs/models/gpt-4o-mini-tts]) make it easier than ever to go full duplex.

Let’s break it down 👇

🧠 What is a Voice Agent?

A Voice Agent is a modular AI system that:

 1. Accepts speech input (via STT),
 2. Processes it through an LLM agent (using the OpenAI Agent SDK),
 3. Responds with synthesized speech (via TTS).

Think of it as a voice-powered GPT chatbot, but with tool use, handoffs, and real-time feedback loops. The modular architecture means you can swap components (like Whisper, GPT-4o-transcribe, GPT-4o-tts) effortlessly to upgrade your agent.

Agent SDK + Voice = Magic

The OpenAI Agent SDK provides three key components:

 * Agents: LLMs with specific instructions and tools
 * Tools: Callable functions like get_time_in_timezone()
 * Runner: Manages the agent loop (handoffs, tools, etc.)

To add voice, we use two new concepts:

 * Voice Workflow: Handles transcription ➡️ text ➡️ LLM ➡️ response
 * Voice Pipeline: Adds STT and TTS layers around the workflow

🔗 Quickstart [https://openai.github.io/openai-agents-python/voice/quickstart/]

🔧 Setting Up Your Voice Agent

Here's a quick, actionable breakdown of building your voice agent:

Step 1: Installation

Install OpenAI's Agent SDK with voice components:

Python pip install 'openai-agents[voice]' sounddevice

Step 2: Creating a Tool

Build a simple tool—for instance, retrieving the current time in any timezone:

Python from datetime import datetime from zoneinfo import ZoneInfo from agents import function_tool @function_tool def get_time_in_timezone(timezone: str) -> str: try: tz = ZoneInfo(timezone) current_time = datetime.now(tz) return current_time.strftime("%Y-%m-%d %H:%M:%S %Z%z") except ValueError: return "Invalid time zone. Please provide a valid name."

Step 3: Defining Agents

Create agents for tasks and specific languages (like Spanish):

Python from agents import Agent from agents.extensions.handoff_prompt import prompt_with_handoff_instructions spanish_agent = Agent( name="Spanish", handoff_description="Handles interactions in Spanish.", instructions=prompt_with_handoff_instructions( "Speak politely and concisely in Spanish." ), model="gpt-4o-mini", ) agent = Agent( name="Assistant", instructions=prompt_with_handoff_instructions( "Speak politely and concisely. Hand off Spanish interactions." ), model="gpt-4o-mini", handoffs=[spanish_agent], tools=[get_time_in_timezone], )

⚙️ Integrating the Voice Workflow and Pipeline

Workflow: Integrates transcription, agent processing, and response generation.

Pipeline: Chains STT, Workflow, and TTS together, orchestrating a seamless voice interaction.

Here's the simplified workflow implementation:

Python ▼ from agents import Runner from agents.voice import VoiceWorkflowBase, VoiceWorkflowHelper from collections.abc import AsyncIterator class VoiceAgentWorkflow(VoiceWorkflowBase): def __init__(self, on_start): """ Args: on_start: A callback that is called when the workflow starts. The transcription is passed in as an argument. """ self._input_history = [] self._current_agent = agent self._on_start = on_start async def run(self, transcription: str) -> AsyncIterator[str]: self._on_start(transcription) self._input_history.append({"role": "user", "content": transcription}) result = Runner.run_streamed(self._current_agent, self._input_history) async for chunk in VoiceWorkflowHelper.stream_text_from(result): yield chunk self._input_history = result.to_input_list() self._current_agent = result.last_agent

🎧 Building a Real-Time Voice Application

Integrate everything into a real-time, interactive voice app using sounddevice:

Main Components:

 * Audio capturing and playback
 * Voice Pipeline to process input and generate spoken responses

Here's how you put it together in your main application (main.py):

Python import asyncio, numpy as np, sounddevice as sd from agents.voice import StreamedAudioInput, VoicePipeline from workflow import VoiceAgentWorkflow CHUNK_LENGTH_S = 0.05 # 50ms SAMPLE_RATE = 24000 FORMAT = np.int16 CHANNELS = 1 class RealtimeCLIApp: def __init__(self): self.should_send_audio = asyncio.Event() self.pipeline = VoicePipeline( workflow=VoiceAgentWorkflow(on_start=self._on_transcription), stt_model="gpt-4o-transcribe", tts_model="gpt-4o-mini-tts" ) self._audio_input = StreamedAudioInput() self.audio_player = sd.OutputStream(samplerate=SAMPLE_RATE, channels=CHANNELS, dtype=FORMAT) def _on_transcription(self, transcription: str): print("User:", transcription) async def start_voice_pipeline(self): self.audio_player.start() result = await self.pipeline.run(self._audio_input) async for event in result.stream(): if event.type == "voice_stream_event_audio": self.audio_player.write(event.data) async def send_mic_audio(self): stream = sd.InputStream(channels=CHANNELS, samplerate=SAMPLE_RATE, dtype="int16") stream.start() read_size = int(SAMPLE_RATE * CHUNK_LENGTH_S) try: while True: await self.should_send_audio.wait() data, _ = stream.read(read_size) await self._audio_input.add_audio(data) except KeyboardInterrupt: stream.stop() async def run(self): print("Press 'K' to start/stop recording, 'Q' to quit.") asyncio.create_task(self.start_voice_pipeline()) asyncio.create_task(self.send_mic_audio()) while True: cmd = (await asyncio.to_thread(input)).strip().lower() if cmd == 'q': break elif cmd == 'k': if self.should_send_audio.is_set(): self.should_send_audio.clear() print("Recording stopped.") else: self.should_send_audio.set() print("Recording started.") if __name__ == "__main__": asyncio.run(RealtimeCLIApp().run())

🚀 Test Your Voice Agent!

Set your OpenAI API key:

Unset export OPENAI_API_KEY="sk-..."

Launch your app:

Unset python main.py

Then:

 * Press K to start recording your voice.
 * Speak your query.
 * Listen to the agent's voice response.

🎯 Example interaction:

You: "What’s the time in Tokyo?"
Agent (Voice): "The current time in Tokyo is 14:37 JST."

See the complete code on GitHub [https://github.com/Neurl-LLC/OpenAI_Voice_AI_agent].

Real-World Use Cases

This Voice Agent framework can serve as the foundation for:

 * AI customer support reps
 * Booking assistants
 * In-car voice interfaces
 * Voice-first developer tools

With modular workflows and custom tools, it’s a great launchpad for any speech-based application.

Start building today! Want the full step-by-step tutorial? Stephen is publishing the full guide on The Neural Blueprint [https://neurlcreators.substack.com/] Thursday so subscribe there.

That's it for Tool Tuesday! Tag us in your voice agent projects or the #agents [https://mlops-community.slack.com/archives/C0861BQ65A7] channel—we’d love to see what you build. 🛠️

## HERE TO HELP

Before you go, here are three ways I can help - just hit reply:

 * Curated intros to other community members
 * What problems are you dealing with? Let me help you find the best solutions through my network
 * Looking to augment your staff for an MLOps or AI project? I got you covered

Thanks for reading, catch you next time!

Interested in partnering with us? Get in touch: partners@mlops.community

Thanks for reading. See you in Slack [https://go.mlops.community/slack], YouTube [https://www.youtube.com/channel/UCG6qpjVnBTTT8wLGBygANOQ?view_as=subscriber], and podcast [https://home.mlops.community/public/content/] land. Oh yeah, and we are also on X [https://twitter.com/mlopscommunity] and LinkedIn [https://go.mlops.community/linkedin].

The MLOps Community newsletter is edited by Jessica Rudd [https://www.linkedin.com/in/jmrudd/].

Shoutout to Ranuga Disansa [https://go.mlops.community/i83caq] for his contributions.

---
Source: https://aaif.live/newsletters/mlopscommunity/2025-04-08-tool-tuesday-5-build-voice-agents-with-openai-agent-sdk