HinglishBench v0.1GSM8K parallel set, 82 pairs

HinglishBench: does code-mixed Hinglish cost open models?

Parallel English / romanized-Hinglish reasoning evaluation of Beam-1.0-501B-A23B, Gemma-3-27B-it and Qwen3-32B. 82 GSM8K test items, each with a verified Hinglish translation. Zero-shot, single run, fair chain-of-thought prompt.

Leaderboard

Model	EN acc	Hinglish acc	Gap (HI-EN)	Median token tax (HI/EN)

Token tax = per-item ratio of completion tokens (Hinglish / English), median over 82 pairs. Beam runs at reasoning_effort=low; medium run: observed tax 1.39x (single run).

Headline findings

Beam scores highest on Hinglish of the three tested (95.1%, one item ahead of Gemma's 93.9%) despite English-predominant training - but a Hinglish answer costs a median ~1.5x tokens.
The tax is not universal. Gemma-3-27B scores Hinglish slightly above English (93.9% vs 91.5%) at no token cost; Qwen3-32B loses 7.3 points of accuracy on Hinglish.
The two prompt variants we tested don't remove the tax. A "think in English" system prompt changed accuracy by 0 and tokens by +10% on 30 items. reasoning_effort=medium didn't remove it in our run either (observed 1.39x vs 1.47x; single runs, no variance estimate).

Per-item explorer

Methodology, dataset and code: see the linked dataset card and repo. Beam outputs via HF Inference Providers (Baseten). Translations generated by Beam itself, number-verified automatically; 18 of 100 sampled items excluded for failing verification. Single run per cell - observed differences are reported without claiming statistical significance.