We’ve all had that moment on a phone call: the instant an AI voice sounds a little too stiff or synthetic, our brain checks out.

If you’re building voice AI—whether it’s a real estate assistant calling leads or a support agent resolving tickets—that text-to-speech (TTS) engine isn’t just a technical detail. It’s the entire front door to your product. It’s what makes a caller stick around because they feel heard, rather than hanging up on another clunky automated system.

The problem, though, is that most popular TTS benchmarks and flashy demos are obsessed with English. But if you’re building products for users in India, an English-focused model barely covers the surface. Real-world conversations routinely jump between Hindi, English, and regional languages like Telugu—often within the exact same call.

Before committing to a voice model for our production AI agent, we needed straight answers: Which open-source TTS systems can actually deliver natural, warm, and clear speech across all three languages? And do they maintain that quality consistently, or do they crumble the moment they switch scripts?

To find out, we ran our own hands-on evaluation. In this article, we’re opening up our internal testing sheet to share what worked, what flopped, and where the tech still falls short.

Why Hindi, English, and Telugu TTS Is a Hard Problem

Building a single TTS evaluation that spans Hindi, English, and Telugu is harder than it looks, for a few structural reasons.

Script and phoneme diversity is the first issue. Hindi and Telugu (a Dravidian, syllabic script) have phoneme inventories and prosodic patterns that are quite different from English and from each other. A model trained predominantly on English data often mishandles retroflex consonants, aspirated stops, or vowel length distinctions that are meaningful in Hindi and Telugu.

Training data imbalance compounds this. Most open-source TTS checkpoints are trained on English-dominant datasets. Hindi and, especially, Telugu, have far less high-quality, permissively licensed training audio available, which shows up directly in pronunciation accuracy and naturalness.

Prosody and expressiveness expectations also differ by language. A “neutral, professional” tone in English doesn’t map one-to-one onto what sounds natural in Hindi or Telugu. Emphasis patterns, pacing, and intonation contours are language-specific, so a model can sound fluent in one language and stilted in another even if the underlying architecture is identical.

Finally, license fragmentation limits real-world choices. Many of the strongest-sounding open models carry non-commercial or research-only licenses (CC-BY-NC, CPML-style terms), which rules them out for production use regardless of how well they score.

Because of this, we couldn’t just run one English benchmark and assume the results would transfer. We ran three separate evaluation passes—one per language—and only then looked at what was consistent across all three.

Models Evaluated

All models below are open-source or freely accessible checkpoints, evaluated on a consistent Google Colab (T4 GPU) setup. The specific model set differed slightly by language, reflecting which checkpoints are actually available and functional for each language today.

The primary evaluation set (rubric-scored) included AI4Bharat F5, four Supertonic voice presets (F1–F4), AI4Bharat, AI4Bharat Male, Kokoro, Chatterbox, Coqui TTS, MMS-TTS, SeamlessM4T, and Suno Bark.

A note on language scope: This set in our source data was not explicitly language-tagged. Based on the evaluation context, this batch corresponds most closely to our Hindi-track testing, but we’re flagging this as an assumption rather than a confirmed label, since the underlying spreadsheet did not carry a language column for this table. We’d rather be upfront about that gap than assert a language we can’t verify from the data itself.

The English track (ranked, no numeric scores captured) had Supertonic first, Parler-TTS second, Coqui TTS third, and Chatterbox fourth.

The Telugu track (ranked, no numeric scores captured) had IndicF5 TTS first, Facebook MMS-TTS second, SeamlessM4T third, and Parler-TTS fourth.

Several supertonic entries in the primary set represent different female voice presets within the Supertonic model rather than different underlying models—we kept them separate because voice presence had a measurable effect on perceived quality in our listening tests.

Below is a representative snippet for each model as it was actually loaded and run during evaluation. These are simplified for readability but reflect the real install/inference pattern used in each notebook.

IndicF5 (AI4Bharat)

IndicF5 is an Indic-focused text-to-speech model designed for Indian languages. In our evaluation, it performed particularly well on Telugu, ranking first in the Telugu track, and it also reflects the strong performance we observed from Indic-specialized models.

# pip install transformers==4.49.0 soundfile torch

from transformers import AutoModel
import soundfile as sf

model = AutoModel.from_pretrained(
    "ai4bharat/IndicF5",
    trust_remote_code=True
)

audio = model(
    "आपका स्वागत है, यह प्रॉपर्टी शहर के बीचोंबीच स्थित है।",
    ref_audio_path="ref_female.wav",
    ref_text="Reference transcript matching ref_audio_path",
    speed=0.7
)

sf.write(
    "indicf5_out.wav",
    audio / max(abs(audio)),
    samplerate=24000
)

Kokoro

Kokoro is a lightweight TTS model that was included in our primary evaluation set. It achieved an overall score of 4.33, with strong naturalness and reasonably good clarity and expressiveness.

# pip install kokoro-tts

from kokoro import KPipeline

pipeline = KPipeline(lang_code="a")

generator = pipeline(
    "Let's schedule your property tour for this weekend."
)

for i, (gs, ps, audio) in enumerate(generator):
    pipeline.save(audio, f"kokoro_out_{i}.wav")

Chatterbox

Chatterbox is a modern TTS model designed for expressive speech generation and supports voice-related conditioning capabilities. It scored 4.00 overall in our primary evaluation, performing consistently across naturalness, expressiveness, and clarity.

# pip install chatterbox-tts

from chatterbox.tts import ChatterboxTTS

model = ChatterboxTTS.from_pretrained(device="cuda")

wav = model.generate(
    "यह घर तीन बेडरूम और दो बाथरूम के साथ आता है।",
    cfg_weight=0.5,
    exaggeration=0.6,
)

model.save(wav, "chatterbox_out.wav")

# Sentence-level splitting and trim_silence()
# were applied for Hindi to avoid double-pausing artifacts.

Coqui TTS (XTTS-v2)

Coqui’s XTTS-v2 is a multilingual TTS system that supports speaker conditioning through reference audio. It ranked third in our English comparison and scored 3.00 overall in the primary evaluation.

# pip install TTS

from TTS.api import TTS

tts = TTS(
    "tts_models/multilingual/multi-dataset/xtts_v2"
).to("cuda")

tts.tts_to_file(
    text="This listing includes a private garden and two parking spots.",
    speaker_wav="ref_female.wav",
    language="en",
    file_path="coqui_out.wav",
)

MMS-TTS (Meta, per-language checkpoints)

MMS-TTS provides separate checkpoints for different languages, making it useful for multilingual evaluation. In our results, its performance varied significantly by language: it ranked second for Telugu, while the MMS-TTS checkpoint in the primary evaluation scored 2.00 overall, mainly due to poor clarity.

# pip install transformers scipy

from transformers import VitsModel, AutoTokenizer
import torch
import scipy.io.wavfile

model = VitsModel.from_pretrained(
    "facebook/mms-tts-tel"
)

tokenizer = AutoTokenizer.from_pretrained(
    "facebook/mms-tts-tel"
)

inputs = tokenizer(
    "ఈ ఇల్లు మూడు బెడ్రూమ్‌లతో ఉంది.",
    return_tensors="pt"
)

with torch.no_grad():
    output = model(**inputs).waveform

scipy.io.wavfile.write(
    "mms_tts_out.wav",
    rate=model.config.sampling_rate,
    data=output.numpy().squeeze()
)

SeamlessM4T-v2

SeamlessM4T-v2 is a broad multilingual speech and translation model designed to handle multiple languages within a unified architecture. Despite its wide language coverage, it scored only 1.33 overall in our primary evaluation and ranked third in the Telugu comparison.

# pip install transformers sentencepiece

from transformers import SeamlessM4Tv2Model, AutoProcessor
import torch
import scipy.io.wavfile

processor = AutoProcessor.from_pretrained(
    "facebook/seamless-m4t-v2-large"
)

model = SeamlessM4Tv2Model.from_pretrained(
    "facebook/seamless-m4t-v2-large"
)

inputs = processor(
    text="This property is close to the metro station.",
    src_lang="eng",
    return_tensors="pt"
)

audio = model.generate(
    **inputs,
    tgt_lang="eng"
)[0].cpu().numpy().squeeze()

scipy.io.wavfile.write(
    "seamlessm4t_out.wav",
    rate=16000,
    data=audio
)

Suno Bark

Suno Bark is a generative audio model capable of producing expressive speech and other audio styles. It received the lowest score in our primary evaluation, 1.00, scoring 1.0 across naturalness, expressiveness, and clarity.

# pip install git+https://github.com/suno-ai/bark.git

from bark import generate_audio, SAMPLE_RATE
import scipy.io.wavfile

audio_array = generate_audio(
    "Welcome home! Let me show you around this beautiful property.",
    history_prompt="v2/en_speaker_9",
)

scipy.io.wavfile.write(
    "bark_out.wav",
    rate=SAMPLE_RATE,
    data=audio_array
)

Parler-TTS

Parler-TTS allows the desired voice characteristics to be described through natural-language prompts, giving users control over attributes such as tone, pace, and speaking style. It ranked second in our English track and fourth in the Telugu track.

# pip install git+https://github.com/huggingface/parler-tts.git

from parler_tts import ParlerTTSForConditionalGeneration
from transformers import AutoTokenizer
import torch
import soundfile as sf

model = ParlerTTSForConditionalGeneration.from_pretrained(
    "ai4bharat/indic-parler-tts"
).to("cuda")

tokenizer = AutoTokenizer.from_pretrained(
    "ai4bharat/indic-parler-tts"
)

desc_tokenizer = AutoTokenizer.from_pretrained(
    model.config.text_encoder._name_or_path
)

description = (
    "A clear, energetic female voice, "
    "moderate pace, warm tone."
)

prompt = "ఈ ఇల్లు నగర కేంద్రానికి దగ్గరగా ఉంది."

torch.manual_seed(42)

input_ids = desc_tokenizer(
    description,
    return_tensors="pt"
).input_ids.to("cuda")

prompt_ids = tokenizer(
    prompt,
    return_tensors="pt"
).input_ids.to("cuda")

generation = model.generate(
    input_ids=input_ids,
    prompt_input_ids=prompt_ids
)

sf.write(
    "parler_out.wav",
    generation.cpu().numpy().squeeze(),
    model.config.sampling_rate
)

Evaluation Methodology

Every model was run through the same seven-stage notebook pipeline: install, imports, model load, generation, save, playback, and export. This consistency mattered—differences in inference settings (sample rate, chunking, normalization) can distort a benchmark as much as differences in the underlying model.

For this evaluation round, our primary signal was structured human listening evaluation, scored against a fixed rubric. Automated objective metrics such as Word Error Rate (WER), Real-Time Factor (RTF), and inference latency were part of our broader benchmarking process for other model comparisons, but were not captured in this specific results file—so we report them here only qualitatively, and flag them as a gap rather than backfilling numbers we don’t have.

Test Scenarios

Each model generated speech from a fixed set of representative sentences relevant to a real estate voice-agent context—property descriptions, scheduling language, and persuasive/conversational phrasing—rendered through each model’s available voice or speaker preset. Audio was reviewed by ear across multiple listening passes rather than a single one-shot generation, to reduce the risk of a lucky or unlucky sample skewing the score.

Objective Evaluation Metrics—What They Measure

For context, in TTS benchmarking more broadly, objective metrics typically include Word Error Rate (WER)—how accurately an ASR system can transcribe the generated speech back to the original text, used as a proxy for intelligibility; Real-Time Factor (RTF)—generation time relative to audio duration, used as a proxy for production latency; and speaker similarity—for voice cloning scenarios, how close the generated voice embedding is to the reference speaker’s embedding.

These metrics matter for production readiness, but this particular evaluation round did not log them. We’re calling that out explicitly rather than presenting placeholder numbers as if they were measured.

Subjective Human Evaluation

Objective metrics tell you whether a transcription matches; they don’t tell you whether a voice sounds pleasant, energetic, or trustworthy on a sales call. For a persuasive, customer-facing voice agent, that gap matters a lot—a model can have low WER and still sound flat, robotic, or fatiguing over a multi-minute call.

We scored each model in the primary set on a 1–5 scale across three dimensions: naturalness (does the speech sound human rather than synthetic?), tonality and expressiveness (does the voice carry appropriate emphasis, warmth, and energy?), and clarity and intelligibility (is the speech easy to understand without straining?). The Overall Score is the mean of these three dimensions. For the English and Telugu tracks, evaluation was conducted as a direct comparative ranking rather than a three-dimension rubric, so no per-dimension scores are available there—only relative rank.

Detailed Benchmark Results

Primary Evaluation Set

ai4bharatf5 was the standout performer, scoring a perfect 5.0 on naturalness, tonality/expressiveness, and clarity, for a perfect overall score of 5.00—the only model in this set to do so. supertonic_f1 came next at 4.67 overall (5.0 naturalness, 4.0 expressiveness, 5.0 clarity), losing half a point only on expressiveness.

There’s then a step down to a 4.0–4.33 band. ai4bharat and kokoro both scored 4.33 overall (5.0 naturalness, 4.0 expressiveness, 4.0 clarity for each), and chatterbox scored a flat 4.0 across all three dimensions.

The middle of the pack sits around 3.0. supertonic_f3 scored 3.33 overall (3.0 naturalness, 4.0 expressiveness, 3.0 clarity), while supertonic_f2 and coqui_tts both scored an even 3.0 across all three dimensions.

Below that, quality drops off noticeably. ai4bharat_male scored 2.67 overall (3.0 naturalness, 3.0 expressiveness, 2.0 clarity), and supertonic_f4 scored 2.33 overall (2.0 naturalness, 2.0 expressiveness, 3.0 clarity). mms_tts scored 2.0 overall, with an unusual profile—3.0 on naturalness but only 2.0 on expressiveness and just 1.0 on clarity, meaning it sounded relatively human but was genuinely hard to understand.

The bottom of the set was seamlessm4t, at 1.33 overall (2.0 naturalness, 1.0 expressiveness, 1.0 clarity), and suno_bark, which scored the minimum 1.0 across every dimension.

What this actually means: the gap between the top two models (5.00 and 4.67) and the next band (4.0–4.33) is real but modest—all five are plausibly usable. The gap below an overall score of about 3.0 is where things get more serious: every model in that range has at least one dimension scoring 2 or lower, which in practice meant audible pronunciation issues or flat, fatiguing delivery.

One detail worth calling out: the four supertonic_f* entries are the same underlying model evaluated with different voice presets, and their overall scores range from 2.33 to 4.67—a spread of 2.34 points. That’s a larger gap than the distance between many different models in this set, which tells us voice preset selection was, in our tests, at least as consequential as model choice.

English Track

Supertonic led the English ranking, which is consistent with its strong showing in the primary set (as supertonic_f1). Parler-TTS and Coqui TTS followed in second and third place, with Chatterbox ranking last in this specific English comparison—worth noting since Chatterbox scored a respectable 4.0 overall in the primary set above. This is a reminder that rank position depends on the comparison set: Chatterbox looked strong against suno_bark and seamlessm4t, but weaker against this particular English lineup.

Telugu Track

IndicF5, an AI4Bharat model built specifically for Indic languages, ranked first for Telugu—consistent with the broader pattern in our results that Indic-focused training data tends to produce better outcomes than general multilingual models for regional Indian languages. MMS-TTS placed second, ahead of both SeamlessM4T and Parler-TTS for Telugu specifically, even though in other contexts MMS-TTS scored lower (as seen in the primary set, where mms_tts landed near the bottom of that ranking). This divergence is a useful signal on its own: model performance is not just architecture-dependent, it’s language-dependent, and a ranking from one language should not be assumed to transfer to another.

Cross-Language Comparison and Consistency

Putting the three result sets side by side, a few patterns hold up.

AI4Bharat-family models perform disproportionately well on Indic languages. ai4bharatf5 topped the primary set, and IndicF5 topped the Telugu track. This is consistent with the expectation that models trained with Indic phonetic and prosodic data outperform general-purpose multilingual models on Hindi and Telugu.

Supertonic is a strong, consistent English/general performer, ranking first in the English track and scoring highly (as supertonic_f1) in the primary evaluation—but its quality is highly sensitive to voice preset, as shown by the spread across supertonic_f1 through supertonic_f4.

No single model dominates across all three language tracks. The model that wins for Telugu (IndicF5) is not the same model that wins for English (Supertonic), and the strongest performer in the primary set (ai4bharatf5) is a distinct entry from both. This means a production system spanning all three languages likely needs a per-language model selection strategy rather than a single universal model—at least with the checkpoints available today.

General-purpose multilingual models underperformed language-specialized ones. seamlessm4t scored low in the primary set and ranked third of four for Telugu; suno_bark was the lowest performer we tested in any track. Broad multilingual coverage did not translate into strong performance on any single language in our tests.

What About Voice Cloning

Voice cloning was not part of this evaluation round. None of the result sets in our source data include speaker-similarity scoring or reference-voice comparisons, so we are not reporting on cloning quality here. Several of the evaluated model families (including Coqui and Chatterbox-style architectures) do support voice cloning in general, and this is a natural candidate for a future benchmarking pass—but we don’t have measured data to report today, and we’d rather leave this section short than speculate.

Major Observations and Engineering Insights

A few things stood out to us as we ran this evaluation.

Voice preset matters as much as model choice, in some cases more. The supertonic_f* spread (2.33 to 4.67) was wider than the gap between several distinct models. Teams benchmarking TTS should not treat “the model” as a single fixed quality level—preset and speaker selection needs its own evaluation pass.

Language-specialized checkpoints reliably outperform general multilingual ones for Hindi and Telugu in our tests. This has practical implications for model selection: default to Indic-trained checkpoints (like AI4Bharat’s models) as your starting point for these languages, rather than assuming a globally popular multilingual model will generalize well.

Ranking position is comparison-set-dependent. Chatterbox’s relative position changed depending on which other models it was compared against (4.0 overall score in the primary set vs. last place in the English-specific ranking). Absolute scores, where available, are more informative than rank alone.

The evaluation is currently rubric- and listening-based, not fully instrumented. We don’t yet have WER, RTF, or latency numbers tied to these specific results, which limits how confidently we can speak to production readiness on infrastructure grounds (as opposed to perceptual quality).

Trade-offs to Consider

Quality versus consistency: the highest-scoring model (ai4bharatf5) scored perfectly across all dimensions, suggesting more consistent quality than models with an uneven profile (like one that’s strong on naturalness but weak on clarity).

Naturalness versus clarity: these did not always move together—mms_tts scored 3.0 on naturalness but only 1.0 on clarity, showing that a voice can sound relatively human while still being hard to understand.

Voice and preset variability: multiple presets of the same model varied by up to 2.34 points in overall score, a larger effect than most cross-model gaps in this dataset.

Based purely on what this data shows: for Hindi (or the primary evaluation track), ai4bharatf5 and supertonic_f1 are the strongest starting candidates, both scoring above 4.5 overall. For English, Supertonic is the top-ranked option, followed by Parler-TTS. For Telugu, IndicF5 is the top-ranked option, with MMS-TTS as a reasonable second candidate.

License compatibility must be checked independently of these rankings. A model’s perceptual quality score says nothing about whether its license permits commercial deployment—that verification needs to happen model-by-model before any shortlist becomes a final decision.

Because voice presets had a large effect on perceived quality, any shortlisted model should be re-evaluated across its available presets before final selection, not just at a single default setting.

Across Hindi, English, and Telugu, the clearest and most consistent finding in this round of evaluation is that no single open-source TTS model wins everywhere. ai4bharatf5 led the primary evaluation set with a perfect score, Supertonic led the English track, and IndicF5 led the Telugu track—three different models, three different language contexts. Indic-specialized checkpoints outperformed general multilingual models on Hindi and Telugu, while Supertonic held up well for English and, depending on preset, competitively elsewhere.

For a production voice agent that needs to operate across all three languages, this points toward a per-language model selection strategy, informed by both these perceptual scores and—once available—objective latency and accuracy metrics. We see this benchmark as a solid first checkpoint rather than a final verdict: the next round, with instrumented objective metrics and a confirmed voice-cloning evaluation, will tell us a lot more about what’s actually deployable.

Based on the benchmark data, ai4bharatf5 is the top performer for Hindi, scoring a perfect 5.00 overall across naturalness, expressiveness, and clarity; Supertonic is the top performer for English, ranking first in that track; and IndicF5 TTS is the top performer for Telugu, also ranking first in its track. No single model wins across all three languages, so the best choice is language-specific—and license compatibility for commercial use still needs to be verified separately for each of these picks before production deployment.

Resources

The notebooks used for benchmarking and the generated audio recordings are available here: https://drive.google.com/drive/folders/14_xE6GNMjTjVz2_HJlKqf5lNl5JQL1rc?usp=drive_link

These resources can be used to explore the models, reproduce the experiments, and listen to the generated samples.

Deploying It for Enterprise Use-Cases

NextNeural is a fully built Sovereign AI platform that has been built for advanced Voice AI-based use-cases like the above. The platform, powered by advanced neural models, allows businesses to integrate environment classification in the voice models running there. If you are a developer working in a company, and need this as part of a comprehensive API, feel free to reach out.