Voice and media
How do voice AI assistants work?
How voice AI assistants work: speech recognition, a language model and text-to-speech, versus newer speech-to-speech models, and why latency matters.
A voice AI assistant listens to your speech, works out what you mean, decides what to say, and speaks the answer back. It does this either through a chain of separate components (speech recognition, a language model, text-to-speech) or through a single speech-to-speech model that handles audio end to end.
The classic pipeline: three steps
The traditional design splits the work into stages:
- Speech-to-text (STT). Audio from your microphone is transcribed into text. This step also has to decide when you've finished talking.
- Language model. The transcript goes to a large language model, which writes a text reply using the conversation so far.
- Text-to-speech (TTS). The reply is converted to audio with a synthetic voice. See what is text-to-speech.
The strength of this design is that each part can be swapped independently. Its weakness is that every stage adds delay, and information that isn't text gets lost. Tone, emphasis, laughter and hesitation don't survive transcription.
Speech-to-speech models
A newer approach uses a model that takes audio in and produces audio out directly. There's no mandatory text step in the middle, so the model can respond to how something was said as well as what was said, and it can start speaking sooner. The tradeoff is less modular control: you can't easily inspect or swap an intermediate transcript.
Why latency decides whether it feels natural
Human conversation has very short gaps between turns. If an assistant takes a long pause before answering, the exchange feels like a walkie-talkie rather than a conversation. Latency comes from several places: detecting the end of your turn, running the model, generating audio, and sending audio over the network. Streaming helps, because the assistant can begin speaking before the whole reply is finished.
What is turn-taking and interruption handling?
Two people rarely wait for perfect silence before speaking. A good voice assistant needs voice activity detection to know when someone is talking, and barge-in support so you can interrupt it mid-sentence. When you talk over it, it should stop playing audio and listen. In a group setting there's an extra problem: several people can speak at once, so the system needs a rule for who it listens to.
How does voice AI work on Discord?
A Discord bot can't use your phone's microphone directly. It joins a voice channel as a participant and receives a separate audio stream per speaker over Discord's voice connection, typically compressed with the Opus codec). The bot decodes those streams, feeds them to its speech system, and sends its own audio back into the channel. Discord allows a bot to be connected to only one voice channel per server at a time.
Common limitations
- Mistakes in noisy rooms. Background noise and overlapping speakers reduce accuracy.
- Accents and code-switching. Recognition can degrade when a speaker mixes languages or uses less-common accents.
- Made-up answers. Like any language model, a voice assistant can state wrong things confidently; see what are AI hallucinations.
- Privacy. Always check whether audio is recorded or stored.
Doing this with Based
Based's /join command starts a real-time spoken conversation in your voice channel. It uses a native speech-to-speech AI rather than a separate speech-to-text then text-to-speech pipeline. There are two modes: Group (talks with everyone, one speaker at a time) and Focused (only the person who started the session). Members can talk over Based to cut a reply short, /hush stops the current reply, and /leave ends the session. /voiceconfig sets the language and mode; "auto" follows the language members use. No voice recordings or transcripts are saved. Every slash command also works with the "." prefix, such as .join. See the voice docs and the voice chat feature page.
Frequently asked questions
What are the main parts of a voice assistant?
A classic assistant has speech-to-text, a language model, and text-to-speech. Newer speech-to-speech systems fold these into one model that works on audio directly.
Why is there a delay before a voice assistant answers?
Latency adds up from detecting the end of your turn, running the model, generating audio, and network travel. Streaming the reply as it is generated reduces the wait.
Can you interrupt a voice AI while it is talking?
Many can, through a feature often called barge-in. Based lets members talk over it to cut a reply short, and the behaviour is configurable.
Does Based record voice conversations?
No. Based does not save voice recordings or transcripts.
Keep learning
- Voice and media
What is text-to-speech (TTS)?
Text-to-speech (TTS) turns written text into spoken audio. Learn how TTS works, how neural voices differ from older methods, and where it's used.
- AI basics
What is a large language model (LLM)?
A large language model is an AI trained on huge amounts of text to predict and generate language. Learn how LLMs work, what they do well and where they fail.
- AI basics
What is an AI chatbot?
An AI chatbot is software you talk to in everyday language, powered by a language model. Learn how it works, how it differs from older bots, and its limits.
- AI basics
What are AI hallucinations?
AI hallucinations are confident but false or invented answers from a model. Learn why they happen, how to spot them, and how to reduce them.
- Voice and media
How do music bots work on Discord?
How Discord music bots work: joining voice, streaming audio from sources, managing queues, and why many bots avoid YouTube. Includes common fixes.
Try it in your server
Based is an AI Discord bot in 300+ servers. Every slash command also works with the "." prefix.