The Speech-to-Text article covered converting spoken audio into text. This final article in the phase covers the reverse direction: text-to-speech, or TTS.
The simple definition
Text-to-speech is the technology that converts written text into spoken audio. Recall directly from the OCR article’s Kurzweil Reading Machine story — text-to-speech was the third and final technology Kurzweil’s team had to invent from scratch, alongside omni-font OCR and flatbed scanning, to actually let a machine read printed text aloud to a blind user. Without this piece, the recognized text from OCR would have had nowhere to go — a genuinely necessary final step, not an optional add-on.
Why generating natural-sounding speech is a genuinely hard problem
Recall from the Speech-to-Text article’s discussion of audio as a continuous, sequential signal. Generating that signal is, if anything, an even harder problem than recognizing it — a system doesn’t just need to produce some audio matching the text, it needs to get pacing, intonation, emphasis, and natural pauses right, all the subtle qualities that separate a flat, robotic reading from something that sounds genuinely human. Early text-to-speech systems, including Kurzweil’s original 1976 machine, produced audibly synthetic, mechanical-sounding speech — recognizable and functional, but clearly not human.
flowchart LR
A[Written text] --> B[Convert to a sequence of sound-unit representations]
B --> C[Model predicts natural pacing, pitch, emphasis]
C --> D[Generated audio waveform]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of the difference between a beginner reading a script aloud in a flat, word-by-word monotone, versus a skilled voice actor who naturally varies their pacing, emphasis, and tone to bring the same written words to life. Both are technically “reading the same text,” but only one sounds genuinely natural.
Where this breaks down: A voice actor draws on genuine emotional interpretation and performance skill. Modern text-to-speech systems achieve their increasingly natural-sounding results through the same diffusion and autoregressive generation techniques covered throughout the Generative Models phase, applied to audio waveforms — a learned, statistical model of how natural speech patterns typically sound, not a performer’s genuine emotional read of the material.
Why modern TTS became good enough to enable a genuine, serious problem
This is worth stating directly, since it’s the real reason this technology deserves careful, honest treatment. Recent text-to-speech systems can now clone a specific person’s voice from just a short sample of their real speech — sometimes just seconds of audio, often pulled from public videos or social media — producing entirely new, synthetic speech that sounds convincingly like that real person saying words they never actually said. This capability directly enables the deepfake concerns first raised in the GAN article, now applied specifically to voice.
The real story: a $25 million heist built on exactly this capability
This deserves to be told in full, because it’s a genuine, documented, and genuinely alarming illustration of what this technology can already do. In February 2024, a finance employee at the Hong Kong office of a multinational firm received what appeared to be a video conference invitation from the company’s UK-based Chief Financial Officer.
During the call, the CFO and several other senior colleagues the employee recognized instructed him to transfer funds for a confidential transaction. Every single person on that call except the victim was fake — scammers had used publicly available video and audio of the real executives to generate convincing deepfake video and cloned voices, reading from a script, in real time.
Believing the call to be genuine, the employee made 15 separate transfers totaling roughly HK25.6 million USD — before the fraud was discovered. Hong Kong police described it as one of the first known cases of this specific kind of multi-person, real-time deepfake fraud, and investigators noted the scammers deliberately kept the fake executives’ dialogue brief and scripted, avoiding extended, spontaneous conversation that might have exposed inconsistencies.
A concrete example, layered
For a simple beginner example: an audiobook app that converts an e-book’s written text into a natural-sounding narrated audio version is running text-to-speech, letting a reader listen rather than read. For a production example: real voice-cloning products from companies like ElevenLabs, and OpenAI’s own published text-to-speech API, are used legitimately for audiobook narration, accessibility tools, and video localization — the exact same underlying capability documented in the Hong Kong fraud case, deployed instead for real, consented, beneficial purposes, which is precisely why responsible providers increasingly require proof of consent before cloning a specific person’s voice.
Why this connects directly to the AI Safety and Guardrails articles
It’s worth tying this back explicitly to concerns raised earlier in this glossary. Recall from the Guardrails article that practical, deployed safeguards exist specifically to catch what a model’s own training can’t fully prevent. Voice cloning’s real, demonstrated fraud risk is exactly why major providers now build in safeguards — requiring the speaker’s own consent recording before cloning their voice, watermarking generated audio, and restricting voice-cloning access — a direct, practical response to the genuine, documented harm the Hong Kong case demonstrated.
Five real, genuinely different approaches, with their actual published results
This is worth grounding concretely, since text-to-speech has gone through several genuinely distinct technical generations, each with real, measured results. WaveNet (DeepMind, 2016) pioneered generating raw audio waveforms directly, sample by sample, using a deep autoregressive network — a genuine leap in naturalness over older, robotic-sounding systems, though computationally expensive to run.
Tacotron 2 (Google, 2018) took a two-stage approach — a sequence-to-sequence network converts text into an intermediate mel spectrogram (a visual representation of sound frequencies over time), and a modified WaveNet then converts that spectrogram into actual audio; its published result was genuinely striking, achieving a Mean Opinion Score (a standard human-rated naturalness metric) nearly matching real human speech recordings.
FastSpeech addressed a real limitation both of the above shared — slow, one-sample-at-a-time autoregressive generation — by predicting an entire spectrogram in parallel rather than sequentially; its published finding was a genuinely dramatic 270x speedup in spectrogram generation compared to autoregressive Transformer-based TTS, with comparable quality.
Amazon Polly’s Neural TTS engine took the sequence-to-spectrogram-plus-vocoder approach into a real, commercial product, explicitly marketed as replacing older “concatenative” synthesis — literally splicing together pre-recorded speech fragments — with a fully learned neural approach.
ElevenLabs, a more recent, specialized voice-cloning company, focuses specifically on zero-shot voice cloning from very short audio samples — the exact capability underlying both the legitimate accessibility tools and the real Hong Kong fraud case covered above, a genuine illustration of how the same core technical approach serves entirely different, real-world outcomes depending on how it’s deployed.
flowchart LR
A[WaveNet: raw waveform, sample by sample] --> F[Major naturalness leap, but slow]
B[Tacotron 2: text to spectrogram, then vocoder] --> G[MOS nearly matching human speech]
C[FastSpeech: parallel, non-autoregressive] --> H[270x faster than autoregressive TTS]
D[Amazon Polly NTTS: commercial seq2seq + vocoder] --> I[Replaced older concatenative splicing]
E[ElevenLabs: zero-shot voice cloning] --> J[Seconds of audio needed to clone a voice]Follow text into a waveform
written text
↓ normalize abbreviations, numbers, dates, and currency
pronunciation and sound-unit representation
↓ predict timing, pitch, stress, and audio features
spectrogram or another learned audio representation
↓ vocoder or audio decoder
playable waveformText normalization is easy to overlook. Consider:
Input: “Dr. Rao arrives at 6:05 with ₹1,250.”
Possible spoken form:
“Doctor Rao arrives at six oh five with
one thousand two hundred fifty rupees.”The system must choose the correct meaning of abbreviations and numbers before making a natural voice. A phoneme is a basic speech sound, while a spectrogram is a numerical picture of frequencies changing through time. A vocoder or audio decoder turns the learned sound representation into a waveform that a speaker can play.
Ordinary TTS versus voice cloning
| Ordinary TTS | Voice cloning |
|---|---|
| Uses an available synthetic voice | Tries to reproduce a particular person’s voice |
| Requires text to speak | Requires text plus a voice reference or trained voice |
| Main question is natural pronunciation | Adds identity, consent, impersonation, and fraud risks |
Current production examples
OpenAI’s GPT-4o Mini TTS accepts text and outputs audio. Its official model page documents a maximum input of 2,000 tokens, showing why a long book would normally be divided into smaller sections before speech generation.
Google Cloud Text-to-Speech is another real service used for accessibility, navigation, announcements, language learning, and voice applications. Streaming systems can begin returning audio before an entire long response is generated, reducing the delay a listener experiences.
Applications should disclose synthetic speech and must not treat possession of a short voice recording as automatic consent to clone that person. Sources: official OpenAI documentation for GPT-4o Mini TTS and Google Cloud Text-to-Speech.
Common misconception
A frequent beginner assumption: that a cloned voice is easy for anyone to identify as fake once you know to listen for it. The Hong Kong case’s own details argue directly against this — trained, professional finance staff, actively suspicious enough to question the situation, were still convinced by the combination of a familiar voice and a familiar face together, precisely because the scammers deliberately limited the interaction’s length and spontaneity to avoid revealing the kind of subtle inconsistencies that might otherwise give away a synthetic voice.
Closing out this phase
This article completes the Multimodal AI phase, and it’s worth tracing the full arc it built: Multimodal AI named the broad field of understanding across data types, built on the shared-representation architecture of a Multimodal Model. Vision-Language Model (VLM) covered the vision-plus-text combination, powered by the patch-based Vision Encoder, itself one specific technique within the much older field of Computer Vision.
Image Generation tied this phase’s understanding focus back to the Generative Models phase’s creation focus, before OCR, Speech-to-Text, and text-to-speech closed the phase with three specific, historically grounded technologies — all three, remarkably, traceable back to one inventor’s 1976 machine, built to solve one blind man’s very real, human problem, and all three now capable of real, documented good and real, documented harm in equal measure.
In one sentence
Text-to-speech converts written text into spoken audio, completing the technology trio Kurzweil first combined in 1976, and while its modern, voice-cloning form powers genuine accessibility and creative tools, the real, documented $25 million Hong Kong deepfake heist stands as a stark, concrete reminder of exactly what this capability can also be used to do.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed