Speech-to-Text

The OCR article covered converting visual text into usable characters. This article covers the direct audio equivalent: speech-to-text, also called automatic speech recognition, or ASR.

The simple definition

Speech-to-text is the technology that converts spoken audio into written text. Recall from the OCR article’s core task — mapping a raw signal (pixels) to text. Speech-to-text solves the same fundamental problem, but starting from a genuinely different raw signal: a continuous audio waveform rather than a static image, requiring the model to track meaning as it unfolds across time rather than across space.

Why audio is a genuinely different, arguably harder signal than an image

Recall from the Sequence article’s emphasis on order carrying meaning. An image, as covered in the Vision Encoder article, can be chopped into a fixed grid of patches, all existing simultaneously. Audio has no such simultaneity — it’s an inherently sequential, continuous signal, unfolding one instant at a time, where the exact same sound can mean different things depending on the words before and after it, background noise, accent, and speaking speed all interfering with the raw signal in ways a static image’s pixels never do.

flowchart LR
    A[Continuous audio waveform] --> B[Split into short time segments]
    B --> C[Convert segments into vectors]
    C --> D[Transformer processes the sequence]
    D --> E[Output: written transcription]

The real story: the same inventor, tackling the mirror-image problem

Recall directly from the OCR article’s Kurzweil Reading Machine story. Solving OCR alone wasn’t enough to build a machine that could read aloud to a blind user — Kurzweil’s team also had to build a working text-to-speech synthesizer, the reverse-direction technology covered in the very next article, to actually voice the recognized text.

Speech-to-text, historically, developed along a related but separate track — recognizing spoken words rather than generating them — and Ray Kurzweil himself is also credited with developing the first commercially marketed large-vocabulary speech recognition system, making him a genuine, singular figure across nearly every major technology covered throughout this phase.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a skilled court stenographer, typing out spoken testimony in real time, correctly distinguishing between similar-sounding words using context — knowing “there,” “their,” and “they’re” apart not from the sound alone, which is often identical, but from the surrounding sentence’s actual meaning.

Where this breaks down: A stenographer draws on genuine, real-time linguistic understanding built over years of training. A modern speech-to-text system achieves the same disambiguation through the same statistical, attention-based mechanism covered throughout the Transformers phase — learned patterns connecting sound to likely word sequences, not a stenographer’s genuine, moment-to-moment comprehension of the testimony’s actual content.

The real, well-documented breakthrough that opened this technology up to everyone

This deserves to be told with its actual, published numbers, since it’s a genuine, consequential moment for the field. In September 2022, OpenAI released Whisper, a speech recognition model trained on 680,000 hours of audio collected from the internet — including 117,000 hours specifically covering 96 languages other than English — and, critically, released it as fully open-source, free for anyone to download, inspect, and build on.

In published zero-shot evaluations, Whisper made roughly 55% fewer errors on average than Wav2Vec, a prior widely used baseline model.

Because it was trained on real, messy, “weakly labeled” audio scraped from across the internet, rather than carefully curated, narrow benchmark datasets, Whisper proved genuinely robust to accents, background noise, and technical jargon in a way many previous systems struggled with — and OpenAI later released a hosted Whisper API, priced at just $0.006 per minute of audio, making production-grade speech recognition accessible to any developer at minimal cost.

A concrete example, layered

For a simple beginner example: a voice memo app that automatically transcribes a recorded conversation into readable, searchable text is running speech-to-text on the raw audio, converting the spoken words into a written record in real time. For a production example: Whisper, referenced throughout this article, is used in real, widely deployed applications ranging from generating automatic subtitles and closed captions for video content, to Spotify’s real, published partnership with OpenAI for AI-powered voice translation of podcasts — a genuine, production use case built directly on this exact open-source model.

Why speech-to-text still has real, honest, documented limitations

It’s worth being direct about a genuine, acknowledged weakness here, not presenting the technology as fully solved. OpenAI’s own documentation for Whisper explicitly cautions that the model can “hallucinate” — producing words in a transcript that were never actually spoken, a direct echo of the exact hallucination problem covered throughout the RAG & Retrieval phase, here applied to audio rather than text generation.

This happens partly because the model was trained on noisy, imperfect internet audio and learned some tendency to predict plausible-sounding words even when the actual audio signal is unclear or silent, a genuine, acknowledged trade-off of training on such a large, unfiltered real-world dataset rather than a smaller, cleaner one.

Real, differentiated approaches beyond Whisper, with their own findings

It’s worth naming a few more real, published systems, since speech-to-text approaches genuinely differ in strategy, not just in scale.

wav2vec 2.0 (Meta, 2020) took a self-supervised approach distinct from Whisper’s supervised training on labeled internet audio — it first learns general audio representations from large amounts of unlabeled speech, echoing the Self-Supervised Learning article’s core technique, then fine-tunes on a comparatively small amount of labeled data; its published finding was that this approach could match prior state-of-the-art results using 100 times less labeled data.

Google Speech-to-Text and Amazon Transcribe, both real, widely deployed commercial services, differ from Whisper mainly in their production integration — offering real-time streaming transcription, speaker diarization (distinguishing different speakers automatically), and deep integration with each company’s broader cloud infrastructure, trading Whisper’s research-grade openness for enterprise support and service-level guarantees.

Deepgram, a real, specialized speech-AI company, built its own end-to-end deep learning architecture specifically optimized for low-latency, real-time transcription in production call-center and voice-agent settings, an area where Whisper’s original design was not specifically optimized.

Follow speech into a transcript

microphone waveform
       ↓ divide audio into short time segments
audio features or tokens
       ↓ speech recognition model
partial transcript: “Please send...”
       ↓ later audio supplies more context
final transcript: “Please send the report by Friday.”

A live-caption application values low delay because words must appear while someone speaks. A meeting archive may accept more delay in exchange for timestamps, punctuation, speaker diarization, and revisions based on later audio.

Speaker diarization answers “Who spoke when?” It labels different speakers; it is separate from recognizing what each person said.

Measure errors with Word Error Rate

Reference:  Send the report on Friday
Prediction: Send a report on Friday

1 substituted word ÷ 5 reference words = 20% WER

WER counts substituted, deleted, and inserted words relative to the reference transcript. Lower is better, but one overall number can hide poor performance for accents, noisy rooms, names, numbers, or a particular language.

Current production examples

OpenAI’s GPT Transcribe supports completed audio files, streamed file transcripts, and committed turns in Realtime sessions. Its documentation lists keyword hints and multiple language hints for specialist terminology, multilingual audio, and code-switching. The documented service price is $0.0045 per audio minute at the time of this article.

Google Cloud Speech-to-Text is another production service used for captions, call analysis, and voice interfaces. In both systems, transcription produces text; a separate language-model or application step may summarize, answer questions, or take actions from that transcript.

Sources: official OpenAI documentation for GPT Transcribe and Google Cloud Speech-to-Text.

Common misconception

A frequent beginner assumption: that speech-to-text systems work by matching sounds to a fixed, one-to-one dictionary of word-pronunciations, similar to a lookup table. As the Whisper discussion above showed, modern systems are considerably more sophisticated — using the same sequence-processing, attention-based architecture covered throughout the Transformers phase to consider surrounding context, not just isolated sound-to-word matching, which is exactly why they can correctly distinguish homophones and handle genuinely messy, accented, or noisy real-world audio.

Where this fits in what comes next

You now understand converting spoken audio into text, and the real, open-source breakthrough that made this technology broadly accessible. The final article in this phase, Text-to-Speech, covers the reverse direction — converting written text back into spoken audio, closing the loop this entire phase has traced from Kurzweil’s original 1976 machine through to today’s real, documented concerns about voice cloning and fraud.

In one sentence

Speech-to-text converts spoken audio into written text by processing sound as a continuous sequence rather than a static image, and OpenAI’s 2022 release of Whisper — trained on 680,000 hours of real internet audio and given away fully open-source — genuinely democratized production-grade speech recognition for developers everywhere, honest hallucination limitations included.

Related Terms

• OCR
• Sequence
• Text-to-Speech
• Hallucination
• Attention