AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Telugu Spoken QA Benchmark Shows Gains—and Evaluation Gaps

Spoken question answering (SQA) asks a system to answer a spoken question directly, but resources for low-resource languages such as Telugu remain scarce. A new benchmark, VākQA, provides 2,001 human-verified Telugu factoid question-answer pairs across six domains, with 2.53 hours of speech audio and bilingual transcriptions. The authors call it the first Telugu SQA benchmark.

The benchmark was built from real spoken Telugu quiz-style YouTube videos, not translated or synthesized speech. Pyannote voice activity detection segments non-silent regions into 7-second chunks with 2-second overlap. A fine-tuned Seamless-large-v2 Telugu ASR model, trained on roughly 900 hours of speech data, transcribes each chunk; Gemini extracts question-answer pairs verbatim. Word-level timestamps from Whisper-timestamped with IndicWhisper, followed by fuzzy string matching (Levenshtein ratio at least 85%), align the QA text to audio. Five annotators verified and translated them into English.

The evaluation compares Gemini-2.5-Flash, which accepts speech or text, with open-weight text-only models: Gemma-3 (4B, 12B, 27B), Llama-3.1, Hex-1, Sarvam-m, and Qwen-3-4B. No open-weight Telugu speech LLM was available, so those models were not tested on raw speech. Inputs include direct speech, Telugu ASR text, cascaded ASR→machine translation into English, and oracle text. Scoring uses human 1–5 ratings, Exact Match, F1, BLASER-2.0, and an LLM judge.

The paper validates automatic evaluation. Gemini-as-a-judge correlates best with human ratings (Spearman ρ = 0.86), but it is non-uniform: more lenient for low-quality answers and stricter for high-quality ones. Open-weight judges penalize correct Telugu answers differing in surface form. A reference answer “March twenty-three” and candidate “March 23” were rated 5 by Gemini and 1 by Gemma-3-12B. Switching judges from Gemini to Gemma-3-12B lowered scores for 46.23% of 2,001 Gemini QA answers, improved 21.14%, and left 32.63% unchanged.

With oracle Telugu text, Gemini scores 3.63/5. Open-weight models trail: Gemma-27B 2.55, Gemma-12B 2.01, Sarvam-m 1.98, Gemma-4B 1.43; several others score near or below 1.5. Direct speech reduces Gemini to 3.28. Pairwise, speech degrades performance on 21.2% of questions, improves 13.1%, and leaves 65.7% unchanged relative to oracle Telugu text. Oracle English text scores 3.52; switching from Telugu to English degrades 18.8%, improves 16.5%, and leaves 64.7% unchanged. Cascaded pipelines compound errors: oracle text→Seamless MT scores 2.74; full ASR→MT configurations range from 2.44 to 2.80.

Asked in oracle Telugu text for Telangana’s state fruit, Gemini answers mango. Given the same question as speech, it answers Bathukamma, a state festival, apparently confusing “pandu” (fruit) with “panduga” (festival). The authors attribute this to acoustic confusion causing semantic drift.

Limitations remain. Faithful transcript-based translation can introduce scope ambiguity: Telugu “mana,” implying India, becomes ambiguous in English, leading Gemini to answer Sputnik instead of Aryabhata for India’s first satellite. Gemini’s non-uniform strictness limits fine-grained comparisons, and some Science and Geography reference answers use English transliterations whose scoring consistency is untested.

The broader significance: proprietary models lead; native Telugu text preserves cultural and scope cues; speech or cascaded ASR→MT pipelines add distinct errors. The results do not show that current models solve the task; they establish a benchmark and expose evaluation and pipeline bottlenecks for low-resource spoken QA.