# How do voice AI assistants work?

> How voice AI assistants work: speech recognition, a language model and text-to-speech, versus newer speech-to-speech models, and why latency matters.

URL: https://www.jalkarna.in/based/learn/how-voice-ai-assistants-work/  
Last updated: 2026-11-16

A voice AI assistant listens to your speech, works out what you mean, decides what to say, and speaks the answer back. It does this either through a chain of separate components (speech recognition, a language model, text-to-speech) or through a single speech-to-speech model that handles audio end to end.

## The classic pipeline: three steps

The traditional design splits the work into stages:

1. **Speech-to-text (STT).** Audio from your microphone is transcribed into text. This step also has to decide when you've finished talking.
2. **Language model.** The transcript goes to a [large language model](https://www.jalkarna.in/based/learn/what-is-a-large-language-model/), which writes a text reply using the conversation so far.
3. **Text-to-speech (TTS).** The reply is converted to audio with a synthetic voice. See [what is text-to-speech](https://www.jalkarna.in/based/learn/what-is-text-to-speech/).

The strength of this design is that each part can be swapped independently. Its weakness is that every stage adds delay, and information that isn't text gets lost. Tone, emphasis, laughter and hesitation don't survive transcription.

## Speech-to-speech models

A newer approach uses a model that takes audio in and produces audio out directly. There's no mandatory text step in the middle, so the model can respond to how something was said as well as what was said, and it can start speaking sooner. The tradeoff is less modular control: you can't easily inspect or swap an intermediate transcript.

## Why latency decides whether it feels natural

Human conversation has very short gaps between turns. If an assistant takes a long pause before answering, the exchange feels like a walkie-talkie rather than a conversation. Latency comes from several places: detecting the end of your turn, running the model, generating audio, and sending audio over the network. Streaming helps, because the assistant can begin speaking before the whole reply is finished.

## What is turn-taking and interruption handling?

Two people rarely wait for perfect silence before speaking. A good voice assistant needs **voice activity detection** to know when someone is talking, and **barge-in** support so you can interrupt it mid-sentence. When you talk over it, it should stop playing audio and listen. In a group setting there's an extra problem: several people can speak at once, so the system needs a rule for who it listens to.

## How does voice AI work on Discord?

A Discord bot can't use your phone's microphone directly. It joins a voice channel as a participant and receives a separate audio stream per speaker over Discord's voice connection, typically compressed with the [Opus codec](https://en.wikipedia.org/wiki/Opus_(audio_format)). The bot decodes those streams, feeds them to its speech system, and sends its own audio back into the channel. Discord allows a bot to be connected to only one voice channel per server at a time.

## Common limitations

- **Mistakes in noisy rooms.** Background noise and overlapping speakers reduce accuracy.
- **Accents and code-switching.** Recognition can degrade when a speaker mixes languages or uses less-common accents.
- **Made-up answers.** Like any language model, a voice assistant can state wrong things confidently; see [what are AI hallucinations](https://www.jalkarna.in/based/learn/what-are-ai-hallucinations/).
- **Privacy.** Always check whether audio is recorded or stored.

## Doing this with Based

Based's `/join` command starts a real-time spoken conversation in your voice channel. It uses a native speech-to-speech AI rather than a separate speech-to-text then text-to-speech pipeline. There are two modes: **Group** (talks with everyone, one speaker at a time) and **Focused** (only the person who started the session). Members can talk over Based to cut a reply short, `/hush` stops the current reply, and `/leave` ends the session. `/voiceconfig` sets the language and mode; "auto" follows the language members use. No voice recordings or transcripts are saved. Every slash command also works with the "." prefix, such as `.join`. See the [voice docs](https://www.jalkarna.in/based/docs/voice/) and the [voice chat feature page](https://www.jalkarna.in/based/features/voice-chat/).

## FAQ

### What are the main parts of a voice assistant?

A classic assistant has speech-to-text, a language model, and text-to-speech. Newer speech-to-speech systems fold these into one model that works on audio directly.

### Why is there a delay before a voice assistant answers?

Latency adds up from detecting the end of your turn, running the model, generating audio, and network travel. Streaming the reply as it is generated reduces the wait.

### Can you interrupt a voice AI while it is talking?

Many can, through a feature often called barge-in. Based lets members talk over it to cut a reply short, and the behaviour is configurable.

### Does Based record voice conversations?

No. Based does not save voice recordings or transcripts.

## Related
- [What is text-to-speech (TTS)?](https://www.jalkarna.in/based/learn/what-is-text-to-speech/)
- [What is a large language model (LLM)?](https://www.jalkarna.in/based/learn/what-is-a-large-language-model/)
- [What is an AI chatbot?](https://www.jalkarna.in/based/learn/what-is-an-ai-chatbot/)
- [What are AI hallucinations?](https://www.jalkarna.in/based/learn/what-are-ai-hallucinations/)
- [How do music bots work on Discord?](https://www.jalkarna.in/based/learn/how-music-bots-work-on-discord/)
