Engine & deployment
Voice engine
How the platform's own engine holds a live conversation.
The voice engine is the service behind the own engine. It is a separate process from the API: it takes a call's audio over a WebSocket, holds the conversation, and reports what happened back to the platform.
The pipeline
caller audio -> speech detection -> end of turn -> speech-to-text -> language model (streamed)
-> sentences -> text-to-speech (streamed) -> paced playback at 8 kHzInside the engine, audio is 16-bit mono PCM at 8 kHz. Each telephony protocol is converted at the edge.
Where audio comes from
| Source | Protocol |
|---|---|
| Twilio | Media Streams (μ-law, 8 kHz) |
| Plivo | Audio Streams (μ-law, or linear PCM) |
| Exotel | Voicebot stream (16-bit PCM, 8 kHz) |
| A browser tab | The engine's own protocol for the browser voice test |
Hearing the caller
- Speech detection combines a voice-activity detector with energy and spectral checks, so hiss, hum, clicks and tones are not taken for speech.
- End of turn is a silence timer that adapts: shorter after what looks like a complete sentence, longer after a single word. If the caller resumes before the agent is audible, the two parts are merged into one turn.
- Barge-in is "pause, then decide". When the caller speaks over the agent, playback pauses at once. When the transcript arrives, a long enough utterance cancels the agent's reply; a short one ("ok") lets it resume.
- Echo guard: the engine compares incoming audio and its transcript with what it is playing, so that the agent's own voice coming back down the line does not interrupt it.
- A high-pass filter and a noise gate are on by default. Spectral noise suppression is available and off by default.
Speaking
Replies are streamed from the language model, cut into sentences, and each sentence is sent to text-to-speech as soon as it is complete. Audio is played slightly ahead of real time so that an interruption can cut it quickly. An optional ambient background track can be mixed in.
What the engine does during a call
| Feature | Behaviour |
|---|---|
| Function tools | Offered to the model as tools. The engine makes the HTTP request itself and returns the result to the model. |
| Graph agents | Walked node by node; rule edges cost nothing, intent edges take one short model request. |
| Knowledge base | One search per caller turn, on the caller's words, added to the model's context. |
| Keypad | Digits become a caller turn, or are collected by a tool or a graph node. Sensitive entries are redacted and silenced in the recording. |
| Transfer | The engine announces the handover and ends its stream; the telephony side connects the person. |
| Voicemail | Off unless the agent turns it on (task_config.voicemail: true). Then, on outbound calls, a beep, a long uninterrupted greeting or a known phrase marks an answering machine. |
| Redaction | Card, account and identity numbers and codes are masked before the model, the logs and the platform see them. Pattern-based. |
| AI disclosure | A configured line is spoken right after the welcome message. |
| Recording | The engine records the call (mono, both sides mixed) and uploads it to the platform. |
When something fails
If a model server errors, times out or answers nothing, the error is logged to the call and the agent speaks a fallback line. That line is spoken through text-to-speech as well, so when the text-to-speech server is the one that failed the caller hears silence; there is no automatic switch to a second speech-to-text or text-to-speech server yet. After several failures in a row the engine ends the call with the reason backend_failure.
Response time
The time from the caller stopping to the agent's first audio is the sum of four things: the end-of-turn wait, speech-to-text, the model's first sentence, and the first audio from text-to-speech. Each turn's breakdown is logged and returned with the call's result.
In the development setup (in-house speech and language models on the same network, a short sentence per turn) this measured between about 1.1 and 1.2 seconds, of which 0.6 seconds was the end-of-turn wait. These figures were measured on Gemma 4 E4B, October 2026; 26B results to follow.
Note: That is one measurement on one setup, not a guarantee. Your figure depends on your model servers, their load and the network between them and the engine.
Capacity
One engine process takes MAX_CALLS calls at once (8 by default) and caps the requests in flight to each model server. A call beyond the limit is refused before it is answered. On shutdown the engine stops taking calls and gives the ones in progress a grace period to finish.
Limitations
- Speech detection tells voices from noise, not the caller from other voices. A television or a second person is speech.
- End of turn is a silence timer. A caller who pauses for longer mid-sentence is answered early.
- There is no echo cancellation, only the echo guard. It was tuned on simulated echo.
- Speech-to-text is not streaming: the utterance is uploaded when the caller stops, so there are no partial transcripts.
- Recordings are mono and follow the engine's view of time.
- Sentences are synthesized one after another; a text-to-speech server slower than real time leaves audible gaps.
- Graph event edges other than keypad events are not delivered during a live call.
- The engine has carried a small number of real calls, on Exotel. It has not been load-tested with real calls, and has not yet carried a real Twilio or Plivo call.