Case Study: 10K hours of Bengali Speech at Scale
Case Studies
Abstract
Bengali is spoken by more than 280 million people but accounts for less than 0.1% of web text, leaving advanced speech models short on real, conversational audio data. Current public datasets mostly feature short clips, lack detailed metadata, and do not cover a wide mix of topics and dialects. Additionally, they fail to capture the clear differences in pronunciation, vocabulary, and accent between Indian and Bangladeshi Bengali.
Poseidon partnered with a global Voice AI company to collect and transcribe 10,000 hours of single and multi-speaker Bengali audio. This dataset is far more challenging for speech models because it captures unscripted conversations, natural talking speeds, overlapping speech, and deep regional dialects. Fine-tuning Whisper on this data reduced the character error rate (CER) by about 32%, which closely approaches the performance of leading commercial APIs. This study documents the dataset, explains the challenges that set it apart from public data, and shows the resulting improvements in model performance.
1. The Challenge
This project set out to build a Bengali speech corpus at a scale and difficulty public data cannot reach: 10,000 hours of single-speaker and multi-speaker audio with ground-truth transcripts, collected from verified native speakers.
Bengali is hard for reasons that compound:
- Scarcity: 280M+ speakers, but under 0.1% of web content. Public corpora are short, clean, and generic.
- Dialect and orthography: Multiple dialects with distinct vocabularies and competing spelling conventions; India (West Bengal) and Bangladesh Bengali diverge in vocabulary and usage.
- Code-switching: Native speakers routinely mix English into Bengali speech, especially in technical and financial domains.
- No agreed ground truth: Even frontier LLMs disagree on the "correct" grammar, spelling, and punctuation for many constructions (see Appendix A.2).
The core problem is not model capacity. It is data. Capturing the full complexity of the language requires sourcing from native speakers and transcribing with humans in the loop.
2. How We Built the Dataset
The data was collected from verified native speaking contributors, with metadata captured at the source (gender, age, region, dialect). The corpus splits into single-speaker audio, where contributors read ground-truth scripts, and multi-speaker audio, where contributors hold natural conversations that are then diarized and transcribed by human linguists.
Transcript generation is difficult because no single model produces grammatically accurate Bengali text. To reliably produce scripts at scale, we use a model-assisted, human-in-the-loop pipeline. Candidate transcripts are scored with an automated transcript-quality metric, leveraging an LLM-as-judge architecture. Native-speaker linguists review and correct the transcripts, and that feedback is used for improving transcripts. Transcript error rates decline batch over batch as the human feedback tunes our pipeline. Full methodology, metric definitions, and the LLM-disagreement study are in Appendix A.
3. Poseidon Bengali Data Is Harder Than Public Data
Models trained only on "easy" public data (short clips, clean studio recordings, generic scripts) fail on real, messy, domain-specific speech. Poseidon's Bengali data is measurably more challenging across four axes: longer clip lengths, higher code-switching rates, greater lexical diversity, and more varied acoustic environments.
Figure 1: Paired distributions, Poseidon vs public datasets (clip length, code-switching rate, lexical diversity, acoustic environment).
4. Topic Coverage
Beyond difficulty, the dataset is broad. Poseidon transcripts span a wide range of real-world topics rather than the narrow, generic scripts typical of read-speech corpora.
Figure 2: Transcript topic diversity and coverage.
5. Fine-Tuning on Poseidon Data Moves the Needle
To test whether the data improves models, we ran a controlled experiment: fine-tune Whisper Large V3 on Poseidon's Bengali data, and benchmark against India's top sovereign model and a top closed-source model.
After fine-tuning on Poseidon’s internal training dataset, Whisper's CER on test data drops by 32%, out-performing every OSS model, overtaking top sovereign API and closing most of the gap to commercial API’s. This number is reported on Poseidon’s internal test dataset (8 hours, 1k samples of Bengali), which are audio clips the model never saw during training (see Figure 3). As such, training on the right data produces a large, measurable gain.
Figure 3: Character error rate, Whisper vs Poseidon-fine-tuned Whisper vs top sovereign AI vs strongest commercial API on Poseidon’s test dataset, which the models never saw before.
6. Where the Gains Come From (WER Across Test Sets)
Aggregate numbers can hide where improvement comes from. Evaluating word error rate (WER) across three test sets, a purely public set, Poseidon's test set, and a balanced set, shows WER improves for both open- and closed-source models, with the largest gains on the Poseidon distribution, where difficulty is highest.
Figure 4: WER across public / Poseidon / balanced test sets, open- and closed-source models.
7. Qualitative Analysis
Aggregate metrics are necessary but not sufficient. The clearest evidence is in specific, hard cases. These are representative examples.
Technical tokens (the "ions" case). For the English sentence "The pH level is indicated by the amount of H ions of the hydrogen present in the tested chemical," the Poseidon-trained model preserves the technical tokens correctly (h, আয়ন = "ion"). strongest commercial API mis-hears আয়ন as আয়রন ("iron") and drops the h.
Figure 5: English / Bangla ground truth / strongest commercial API / Poseidon.
Code-switched financial speech. On dense English-in-Bengali financial speech (fixed deposit, mutual fund, SIP), the Poseidon Bengali ASR model ties strongest commercial API on WER, while top sovereign API transliterates the English terms into Bengali script, inflating WER on code-switched clips.
Figure 6: Side-by-side transcript table, English / Bangla ground truth / strongest commercial API / Poseidon / top sovereign API.
8. Generalizing Beyond One Language: Poseidon SONAR
The Bengali dataset and results demonstrate an approach Poseidon has generalized across languages. The question for any model builder evaluating training data is the same: how do I know a dataset, model, or test set is good on a new language, a new model, or a new distribution? Poseidon SONAR is our evaluation benchmark for exactly this: bring any model and any data, and evaluate it across many metrics and subgroups (gender, region, dialect).
Figure 7: Poseidon SONAR eval bench https://sonar.psdn.ai/
9. Conclusion
This case study is hard evidence that Poseidon data is novel, genuinely difficult, and able to move the needle on performance of frontier ASR models. Our aim is not to compete with model builders. It is to demonstrate, with measured results, the value of data sourced from native speakers and verified by humans. The same pipeline: collection, fine-tuning, and evaluation, generalizes to any low-resource language.
Appendix A: Data Collection and Transcript QA Methodology
A.1 Collection and contributor verification
Audio is sourced from verified native speakers, with speaker metadata (gender, age, region, dialect) captured at the point of collection. Single-speaker sessions use ground-truth scripts; multi-speaker sessions capture natural conversation, which is diarized and transcribed by human linguists.
A.2 Why transcript generation is hard: frontier LLMs rarely agree
Bengali has no single, model-agreed ground truth. In an internal study of 160 transcripts evaluated by three frontier models (GPT-5.2, Claude Opus 4.5, Gemini 3 Pro), the models flagged 2,407 errors combined, but all three models agreed on only 13.5% of them, and 53.8% were flagged by a single model. Against native-linguist annotations, the best individual model (Gemini 3 Pro) recovered just 30.8% of errors, and all three combined reached 40.8%. Disagreements included false positives (for example, flagging কি as a misspelling of কী, which is grammatically distinct), missed possessive case markers, formality bias, and inconsistent West Bengal vs Dhaka vocabulary standards. This is the evidence base for a human-in-the-loop pipeline.
Figure 8: Inter-model disagreement on Bengali transcripts (agreement rate and per-model error recall vs human linguists)
A.3 Human-in-the-loop review
Native-speaker linguists review and correct model-assisted drafts. Poseidon’s Annotation Platform enforces a standard error taxonomy and consistent conventions so that contributors and reviewers resolve the same ambiguities the same way.
Figure 9: Poseidon Annotation Platform
A.4 Transcript quality metric and LLM-as-judge
We score transcript quality with the Poseidon Score (PSDN), a composite from 0.0 to 1.0 combining word accuracy, character accuracy, and semantic similarity, alongside an LLM-as-judge. Evaluation must control for format artifacts: in multi-speaker audio, silence and turn-taking depress raw scores even when transcription is accurate (single-speaker baseline 0.86 vs dual-speaker 0.71, a gap human reviewers did not perceive).
A.5 Error rate declines across batches
As the pipeline matures, transcript error rate falls batch over batch, evidence that the metric, the taxonomy, and the human review loop compound into durable quality gains.
Figure 10: transcript error rate by batch over time.