Speech and audio

Overview

Speech and audio machine learning covers automatic speech recognition (ASR), text-to-speech (TTS), speaker and sound identification, and the newer speech-to-speech models that listen and talk without text in between. The models are the same transformers and convolutional networks as elsewhere on this site, applied to a signal that arrives at 16,000 samples a second and has to be handled in time. This hub is for engineers adding voice to a product or processing recorded audio at scale, and it concentrates on what differs from text and images: the signal, the alignment between audio and words, and latency.

It matters because audio is where the demo-to-production gap is widest. A model that transcribes a podcast perfectly can fail on a phone call with a fan in the background, and a system with two seconds of delay is unusable for conversation however accurate it is.

Key ideas

Audio as a signal: frames and spectrograms

Speech models work at 16 kHz mono, so the first step is always resampling, and feeding 44.1 kHz or 8 kHz audio to a 16 kHz model without resampling is the most common silent failure. The waveform is then cut into overlapping frames, typically 25 ms long every 10 ms, and each frame is converted to a log-mel spectrogram: the energy in 80 or 128 frequency bands spaced the way the ear spaces them. The result is an image-like array of 100 columns per second that CNNs and transformers can read. Neural audio codecs such as EnCodec go further and turn audio into discrete tokens, which is what lets a language model generate speech directly.

Speech recognition: from frames to words

Two designs dominate. Encoder-decoder models read the whole spectrogram and generate text token by token with attention, which is how Whisper works. Whisper large-v3 (1.55 B parameters, 30 s input windows, MIT licence, November 2023) transcribes and translates 99 languages and is the default for offline, batch transcription. Connectionist temporal classification (CTC) models instead emit one output per frame, including a blank, and collapse repeats, so they need no decoder and align naturally with time. The transducer (RNN-T) adds a small prediction network to CTC and is the standard design for streaming on phones. Conformer encoders, which interleave convolution and attention, sit under most of these. NVIDIA’s Parakeet-TDT 0.6B v2 (600 M parameters, CC-BY-4.0, May 2025) is an English-only example that is both fast and accurate on long files.

Word error rate (WER) is the metric: substitutions, deletions and insertions divided by reference words. It depends heavily on text normalisation (casing, punctuation, numbers as digits or words), so two systems can only be compared with the same normaliser, and a clean-read-speech WER of 3 percent says nothing about a noisy call centre. Speaker diarisation (who spoke when) and voice activity detection are separate models around the recogniser.

Text-to-speech and voice

A TTS system turns text into a mel spectrogram and then into a waveform, either in two stages (an acoustic model such as Tacotron 2 and a vocoder) or end to end, as VITS does with a variational model and adversarial training. The current wave treats speech as tokens from a neural codec and generates them with a language model, which gives natural prosody and voice cloning from a few seconds of reference audio. Kokoro-82M (82 M parameters, Apache-2.0, January 2025) runs well on CPU, and Piper (MIT) runs on a Raspberry Pi. The metrics that matter in a product are time to first audio and real-time factor, since a user hears the delay before the quality. Voice cloning raises consent and licence questions that belong in the design review, see the safety hub.

Audio embeddings and streaming

Self-supervised encoders such as wav2vec 2.0 and HuBERT learn representations from unlabelled audio that transfer to recognition, emotion and speaker tasks with little labelled data, in the same way that pretrained backbones serve vision. CLAP pairs an audio encoder with a text encoder for text-to-audio retrieval and zero-shot sound classification, and speaker embeddings turn a voice into a vector for verification. All of them feed the similarity search machinery unchanged.

Streaming is an engineering discipline of its own. A streaming model sees audio in chunks with a fixed lookahead, so attention must be limited to a window, partial results are emitted and later revised, and an endpointing rule decides when the user has finished speaking. Speech-to-speech models such as Moshi (7 B parameters, CC-BY-4.0 weights, September 2024) fold recognition, a language model and synthesis into one full-duplex network with about 200 ms of latency, and the hosted realtime APIs from OpenAI and Google do the same behind a websocket. The trade-off is latency against accuracy, and the only honest measurement is end to end, microphone to speaker, on the target hardware.

Where to start

There is no audio lesson in the beginner track at launch, so the path runs through the general lessons. Your first neural network in PyTorch gives you the training loop, Embeddings and similarity search covers the representation that audio embeddings plug into, and From notebook to service covers the serving side where streaming lives. The deep learning hub covers the training choices audio models share. An intermediate track on audio is planned.

Tools and platforms

Common mistakes

• Skipping resampling. A 44.1 kHz file read as 16 kHz plays at the wrong speed and transcribes as nonsense.
• Transcribing silence. Whisper in particular invents plausible text on empty or music-only segments, so run a voice activity detector first.
• Comparing WER figures computed with different text normalisers, or on clean read speech when the product sees phone calls.
• Chunking a stream at fixed seconds without overlap, so that words on the boundary are lost or doubled.
• Running an offline 30 s model as if it were streaming and blaming the hardware for the latency.
• Cloning a voice without documented consent, or using a TTS model whose licence forbids the product’s use.

Going deeper

• Radford et al., the Whisper paper on speech recognition via large-scale weak supervision, 2022.
• Baevski et al., wav2vec 2.0, 2020, and Gulati et al., Conformer, 2020.
• Graves, Sequence Transduction with Recurrent Neural Networks, 2012, the transducer paper.
• Kim, Kong and Son, Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech, 2021, the VITS paper.
• Défossez et al., High Fidelity Neural Audio Compression, 2022, the EnCodec paper, and Moshi: a speech-text foundation model for real-time dialogue, 2024.
• The torchaudio documentation and the NeMo ASR documentation.