Audio generation is the use of artificial intelligence to create new sound, such as spoken words, music, ambient noise and sound effects. A user can type a sentence and hear it read in a natural voice, describe a jazz tune and receive a short piece, or ask for the sound of rain on a tin roof. The field is part of generative AI and includes more specific tasks such as speech generation, music generation and text-to-speech. It relies on deep learning models trained on large collections of recordings.
History of audio generation
Sound synthesis has a long history. Electronic instruments of the twentieth century, the Moog synthesizer and software such as Csound, produced sound from mathematical rules. Speech synthesis began with formant and concatenative methods, which joined recorded fragments of speech. In the 1990s and 2000s, statistical models improved the quality of synthetic voices but they still sounded robotic. A major change came in 2016 with DeepMind’s WaveNet, a neural network that produced raw audio sample by sample and sounded far more natural. Later models such as Tacotron and WaveGlow improved speed. Neural audio codecs, including SoundStream and EnCodec, converted sound into discrete tokens, which let language-model techniques generate audio. Systems such as AudioLM, MusicLM, AudioGen, Stable Audio and many commercial tools followed, covering speech, music and effects.
How audio generation works
A digital audio signal is a long sequence of numbers. CD-quality sound has 44,100 samples every second, so a minute contains more than 2.6 million values. Generating this directly is hard, because structure appears at many time scales, from the vibration of a single note to the shape of an entire song. Practical systems therefore use several steps.
- Representation: audio is turned into a more compact form, either a spectrogram, an image-like map of frequencies over time, or a sequence of discrete tokens from a neural codec.
- Conditioning: a text prompt, lyrics, a melody, a speaker sample or another sound is encoded as guidance for the model.
- Generation: a transformer, diffusion model or autoregressive network produces the compact representation. See diffusion model and attention mechanism.
- Decoding: a vocoder or codec decoder converts the representation back to a waveform that can be played.
Training data may include speech corpora, music libraries, field recordings and paired text descriptions.
Main types of audio generation
- Speech generation: converts text into spoken language, with control over voice, accent, emotion and speed. See text-to-speech.
- Music generation: composes melodies, harmonies, instrumentation and sometimes vocals. See music generation.
- Sound effects and ambience: creates footsteps, weather, machinery and environments for games and film.
- Voice conversion and cloning: changes one voice into another or imitates a speaker from a short sample.
- Audio editing and enhancement: removes noise, restores old recordings, separates instruments and extends or fills missing audio.
Applications
Audio generation is used in audiobooks and podcasts, virtual assistants and call centers, navigation and accessibility tools for people who are blind or cannot speak, language learning and dubbing of videos into other languages. Game developers generate adaptive music and effects. Filmmakers create temporary soundtracks and sound design. Musicians use it to sketch ideas, create stems and experiment with styles. Companies produce jingles and background music cheaply, and researchers use synthetic audio to train and test speech recognition systems.
Evaluation
Quality is judged by naturalness, clarity, similarity to a target speaker or style, musical coherence and match to the prompt. Listening tests with human raters, often reported as mean opinion scores, remain the standard. Automatic measures compare the statistics of generated and real audio, but they do not capture everything a listener notices, such as emotion or musical taste.
Limitations
- Long-range structure: generated songs may lose their form or repeat oddly over several minutes.
- Artifacts: buzzing, echo or metallic sounds can appear, especially in noisy or complex scenes.
- Control: it is hard to specify exact timing, lyrics, emotion and mixing details through a text prompt.
- Data bias: models perform best on languages, accents and genres that are well represented in training data.
- Cost: high-quality generation needs significant computing power.
Risks and ethics
Voice cloning can reproduce a person’s voice from a few seconds of recording. Criminals have used cloned voices in phone scams that imitate relatives or executives, and synthetic audio can spread false claims. See deepfake. Musicians and voice actors worry about unauthorized imitation of their voices and styles, and about training on copyrighted recordings without payment. Responses include consent requirements for voice cloning, watermarks such as inaudible signals embedded in generated audio, detection tools, platform policies and new laws. Some jurisdictions now protect a person’s voice as part of their right of publicity, and several music industry disputes are in court.
A worked example
A small history podcast wants a calm narrator voice and a short theme tune. The producer writes the script, selects a licensed text-to-speech voice and generates the narration. She then prompts a music model for an unhurried acoustic theme, tries four versions and chooses one. She checks the license terms for commercial use, discloses that the narrator voice is synthetic in the show notes and has a human listener review pronunciations of foreign place names before release.
Audio generation compared with audio recognition
It helps to separate generation from recognition. Recognition systems, such as speech-to-text, listen to sound and produce text or labels, while generation systems start from text, a description or another signal and produce sound. Many modern products combine both. A voice assistant first transcribes the user’s request, uses a large language model to decide on a reply and then speaks the reply with a text-to-speech model. Real-time translation tools chain recognition, translation and synthesis so that a speaker’s words come out in another language within seconds, sometimes in a voice that resembles the original speaker.
Future directions
Research is moving toward models that handle speech, music and effects in one system, produce longer and better structured pieces, respond to a creator’s edits in real time and run efficiently on phones. Better consent tools and reliable watermarking are also priorities, because wider adoption depends on public trust that listeners can tell real recordings from synthetic ones.
FAQs
What is the difference between audio generation and speech generation? Audio generation is the broad field covering all sound. Speech generation is the part that creates spoken language.
Can AI copy a person’s voice? Yes, with a short sample it can produce a close imitation, so consent and safeguards are important.
Can I use AI-generated music commercially? It depends on the tool’s license and local copyright law, so read the terms before publishing.
More in Generative AI
Key facts
- Audio is a waveform sampled thousands of times per second, so generating it directly is demanding, and many models work on compressed audio tokens or spectrograms.1
- WaveNet, published by DeepMind in 2016, was an early neural model that produced natural-sounding speech sample by sample.2
- Voice cloning can imitate real people, which creates risks of fraud and impersonation.3
Related concepts
In the news
Quick checkWhy is generating raw audio difficult for AI?Show answer
Audio contains tens of thousands of samples per second and patterns on both tiny and long time scales, so models usually compress it into tokens or spectrograms first.
Sources
- Defossez et al., High Fidelity Neural Audio Compression, https://arxiv.org/abs/2210.13438
- van den Oord et al., WaveNet: A Generative Model for Raw Audio, https://arxiv.org/abs/1609.03499
- U.S. Federal Trade Commission, Preventing the Harms of AI-enabled Voice Cloning, https://www.ftc.gov/policy/advocacy-research/tech-at-ftc/2024/11/preventing-harms-ai-enabled-voice-cloning
Editorially reviewed by Specialty Digest Editorial TeamLast reviewed November 17, 2026Researched and drafted with AI assistanceReport an issue
