Someone asks for “a voice AI for our booking line” as if that’s a single feature. It isn’t. It’s five pieces stacked on top of each other, and getting any one of them wrong shows up immediately: a caller has to repeat themselves, or a booking that got confirmed out loud never makes it into the calendar. We build these lines with AI agents on the OpenAI Realtime API, so here’s what’s inside one.
Audio transport comes first
Before any word gets understood, audio has to travel from a phone or a browser to a model and back fast enough that the exchange feels like a conversation instead of a walkie-talkie handoff. The Realtime API supports three ways to move that audio: WebRTC for browser and mobile clients that capture and play audio directly, WebSocket for servers that already have raw audio from a telephony or media pipeline, and SIP for connecting a phone number straight into a realtime session.
A line that answers an inbound phone call needs SIP. A “talk to us” widget on a website needs WebRTC. Picking the wrong one early doesn’t just cost latency, it usually means building a bridge between two systems that didn’t need to exist.
The model: speech-to-speech or chained
There are two ways to wire the model itself, and OpenAI’s voice agent guide draws the line clearly. Speech-to-speech has the model handle live audio input and output directly, which is what gives you natural barge-in and the lowest latency to first audio. A chained pipeline splits the same job into speech-to-text, a reasoning step, then text-to-speech, and earns its keep when you want a durable transcript, deterministic logic between steps, or reuse of an existing text agent.
For a booking line, most of the value sits in the second half of the call: checking availability, writing a record, confirming a time back to the caller. That’s exactly the kind of step where you might want a plain-text transcript to run a policy check against before the agent commits to anything out loud, which is a real argument for the chained approach even with its extra beat of latency.
If you go speech-to-speech, the current model built for it is gpt-realtime-2.1, which added reasoning to speech-to-speech workflows. OpenAI recommends starting reasoning effort low for production voice agents, trading reasoning depth for the response speed a live caller expects.
Interruption handling lives at this layer too, through voice activity detection. Server VAD switches on when it hears speech and off after a stretch of silence. Semantic VAD instead runs a classifier over the words spoken so far to judge whether the caller is done, which cuts down on the model talking over someone mid-sentence. Turning that on is one setting in the session config:
{
"type": "realtime",
"model": "gpt-realtime-2.1",
"audio": {
"input": {
"turn_detection": {
"type": "semantic_vad",
"eagerness": "low"
}
}
}
}Tool calls do the actual work
The model on the call doesn’t book anything by itself. It calls tools, the same mechanism a text-based agent uses: check a calendar, look up an account, write a booking record. That’s what AI integration work usually means in practice, wiring a model up to the systems that hold the real data. When a caller hears “let me check that for you,” that pause is a tool call running in the background.
Guardrails catch the call before it goes wrong
Guardrails run validation on user input before the agent acts, and on the agent’s output before it goes back to the caller. A tripwire inside a guardrail halts execution immediately rather than letting a bad response reach the speaker. On a booking line, an input guardrail can stop a caller from talking the model into quoting a price it isn’t authorized to quote. An output guardrail can catch the model committing to a time slot before that commitment gets checked against what’s free.
Handoff to a human
None of this happens automatically. The chained architecture is the one OpenAI points to for approval-heavy flows precisely because it gives you a place to store the transcript and run a policy check before the agent responds, and that’s also the natural point to insert a “transfer to a person” branch. On a booking line, that branch gets used when a caller asks for a manager, when a guardrail trips, or when the agent runs out of confidence. Whichever it is, the call needs a clean path to a real person, with the transcript already attached so nobody has to repeat themselves from “hi, I called earlier.”
What we build, and what we weigh with a client
The five pieces don’t change: transport, model, tool calls, guardrails, and a handoff path. What changes case by case is which side of each decision fits the call you’re building. Speech-to-speech versus chained depends on how much of the value is in the conversation versus the record it produces afterward. WebRTC versus SIP depends on whether the caller opens a browser or dials a number. Which actions get a tripwire depends on what’s reversible if the model gets it wrong.
A voice agent that nails the model but skips the guardrails and the handoff path isn’t a finished booking line. It’s a demo with a phone number attached. If you’re weighing whether a phone or intake line is worth building this way, tell us what the line needs to do and we’ll walk through which of these five pieces actually matter for your case, and which ones you can skip. For the rest of what falls under AI, the same rule applies: the interesting engineering is never the model call, it’s everything built around it.