How we built a real-time Urdu voice agent on OpenAI's Realtime API

Graana.com is a Pakistani property portal. With our agency team, I worked on a voice assistant for it: a visitor asks in Urdu for, say, a five-bedroom house in G-10, Islamabad, and the assistant searches live listings and investment plans, then answers in Urdu while matching properties appear on screen.

I built mainly the voice console, the React interface in the browser; a teammate built the Node.js relay and the tools.

The shape of the system

1. A React voice console records the microphone, plays the assistant's audio and renders results.
2. A Node.js relay holds the OpenAI API key and forwards events between the browser and the Realtime API over WebSockets.
3. Tools the model can call: one searches Graana's listings, the other a catalogue of investment plans.

The model is speech-to-speech: audio goes in, audio comes out, and a response can include a function call. There's no speech-to-text, text model and text-to-speech chain to build, but the parts around the model still decide whether it feels fast.

We built on the beta interface; the generally available API renamed events (response.audio.delta became response.output_audio.delta) and restructured session settings, so pin versions and distrust old tutorials. OpenAI's current Realtime guide uses gpt-realtime-2.1 in its examples.

Why a server relay

OpenAI recommends WebRTC for browser clients, with a short-lived client secret minted by your server, and WebSockets for server-to-server connections. A relay is the other valid pattern. It costs a network hop and buys control:

• Keys. The real API key never reaches the browser.
• Auth and limits. You decide who can open a session and for how long. Every open session can spend money.
• Session config. Instructions and tools are set server-side, so nobody can send their own session.update and rewrite your prompt.
• Logging. Every event passes through one place.
• Tool execution. Tools that need secrets or private data can run on the server.

Our relay is thin: it holds the key and forwards events. The tools run in the browser console, which suits this product because the listings are public and every result is also drawn on screen. A tool touching private data would belong on the server.

A simplified sketch of a relay with those guard rails (authenticate, SESSION and log are placeholders; no reconnects or rate limits):

import { WebSocketServer, WebSocket } from "ws";

const ALLOWED = new Set([
  "input_audio_buffer.append", "input_audio_buffer.commit",
  "conversation.item.create", "conversation.item.truncate",
  "response.create", "response.cancel",
]);

new WebSocketServer({ port: 8081 }).on("connection", async (browser, req) => {
  const user = await authenticate(req);
  if (!user) return browser.close(1008, "unauthorized");

  const openai = new WebSocket("wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1", {
    headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` },
  });
  const pending = [];

  openai.on("open", () => {
    openai.send(JSON.stringify({ type: "session.update", session: SESSION }));
    pending.splice(0).forEach((msg) => openai.send(msg));
  });

  browser.on("message", (data) => {
    const event = JSON.parse(data);
    if (!ALLOWED.has(event.type)) return; // browser can't change the session
    const msg = JSON.stringify(event);
    openai.readyState === WebSocket.OPEN ? openai.send(msg) : pending.push(msg);
  });

  openai.on("message", (data) => {
    const event = JSON.parse(data);
    if (!event.type.endsWith(".delta")) log(user.id, event.type);
    browser.send(data.toString());
  });

  browser.on("close", () => openai.close());
  openai.on("close", () => browser.close());
});

Streaming audio and turn-taking

The console records 16-bit PCM at 24 kHz and streams it to the relay in small chunks. Replies come back as audio deltas and play as they arrive.

The harder question is when the user has finished. The turn detection guide describes two automatic modes: server_vad ends a turn after a stretch of silence, and semantic_vad uses a classifier to judge whether the user has finished their thought.

We turned automatic detection off and used push-to-talk: the visitor holds a button while speaking, and releasing it commits the audio and requests a response. It feels less natural than an open microphone, but rules out false turns from background noise or other voices in the room.

When the visitor presses the button while the assistant is talking, the console:

1. Stops local playback immediately.
2. Cancels the in-progress response.
3. Truncates the assistant's message to the audio actually heard, with conversation.item.truncate and the playback position.

Step 3 is easy to miss. Without it, the history holds the full reply, and later answers can build on words the user never heard. Per the conversations guide, WebRTC tracks playback and truncates for you; over WebSockets the client owns playback, so it must report it.

Tool calls that drive the screen

The two tools, get_property and get_investment, have JSON schemas with enums for what Graana's search understands: rent or sale, house, flat or plot, marla or kanal, prices in rupees. What mattered:

• Required fields encode conversation rules. City is required for a property search; its description tells the model to ask for it first.
• Normalise in the description. "If the user states years, convert to months before calling." The API never has to guess.
• Return less to the model than you show the user. The console fills the property grid with up to 50 listings, but the model gets one line back, such as "Properties available: 12". The screen carries the detail; the voice narrates. Replies stay short and the history stays small. The trade-off: the model can't describe a specific listing unless you pass it the details.
• Errors are tool output. If the property search fails, the model gets the error message and can say so, instead of leaving the visitor in silence.

Latency budget

What users feel is the gap between finishing speaking and hearing the first word back. It's a sum of stages:

A tool round trip sits in the middle of the user's wait, so a slow search API is directly audible. Timestamp each stage: one end-to-end number says something is slow, per-stage numbers say what.

Urdu

• Pin the language and name the confusable one. Our instructions say to reply strictly in Urdu, explicitly not Hindi or English; everyday spoken Urdu and Hindi overlap heavily.
• Speak Urdu, call tools in English. Tool parameters must be in English, so "kiraye par ghar" (a house for rent) becomes purpose: "rent", subtype: "house".
• Test the hard tokens. Sectors like G-10 and F-7, prices in lakh and crore, sizes in marla and kanal: one misheard token silently changes the search. Build a test list early.
• Keep a transcript. We enabled input transcription to log what the visitor said. It comes from a separate model, so it isn't exactly what the voice model understood.

Cost

Pricing is per token, audio included. OpenAI's latency and cost guide puts user audio at roughly one token per 100 ms and assistant audio at one per 50 ms, and notes that the entire conversation is sent for each response, so later turns cost more. What helps:

• Short spoken replies and short tool results; the one-line result pays off twice.
• Stable instructions and tools within a session, so cached input can be reused.
• Ending idle sessions; with an open microphone, noise can trigger responses you pay for.
• Trying a mini model only after the full model works; the guide warns mini models can be weaker at function calling.

Failure modes to plan for

• Dropped sockets. The conversation lives in the session; reconnecting starts fresh unless you replay context.
• Invented details. A model that only knows a count can still make up specifics. Tell it not to describe listings beyond what the tool returned.
• Hanging tools. Put a timeout on every tool call and return a clear message when it fires.

Conclusion

A speech-to-speech model handles the audio. The product is everything around it: a relay that owns keys and session config, interruption handling that keeps history honest, tool schemas that encode your rules, and a screen that carries the detail so the voice can stay brief. For Urdu, write the test list of local names, numbers and units before you tune anything else.