AI Papers Reader

Personalized digests of latest AI research

View on GitHub

A New Translation Benchmark Puts Hand-Crafted Rules on Every Hard Case

Machine-translation systems have become strong enough that many standard benchmarks are near saturation. Yet the researchers behind the new Last Translation Benchmark argue that evaluation itself is a weak link: automatic metrics can be gamed or misaligned with human judgment, while traditional human scoring is expensive, noisy, and hard to reproduce. Their response is a crowd-sourced stress test built around failures that still trip up leading systems.

The Last Translation Benchmark (LTB) is a rolling collection of hard-to-translate inputs. LTBv1 includes 3,456 accepted examples in 109 languages, spanning text, images, audio, and video. For every example, a contributor supplies a human reference translation and at least one “verification rule” that names a concrete error an evaluator should look for. An LLM then acts as a rule checker; a candidate must pass all rules to count as correct. Submissions are accepted only if most of the automatic systems shown to contributors fail them while the human reference passes, and each submission is reviewed by a contributor fluent in the language pair.

A simple English-to-Czech example illustrates the mechanism. The source sentence “The two new nurses share the 100m men’s world record” forces Czech to mark gender on the word for nurse. Google Translate, Gemini 3.1 Pro, and GPT-5.6 Sol all produced a feminine Czech form of “nurses,” although the context indicates the athletes are men. The benchmark’s rule catches exactly that mistake: the translation should not use “sestra” or a similar feminine term. The human translation, using the masculine form “zdravotníci” for medical workers, passed.

The first results show how much room remains. On the 911-example text-only evaluation subset, none of 29 systems came close to a high score; the best, Gemini 3.1 Pro, passed 41.9 percent of examples. More revealing is what happens when LLMs are told what to watch for. The researchers measured an average pass rate of 7.2 percent for LLMs translating without rules. With human-written rules included in the translation prompt, the average jumped to 89.8 percent; with LLM-generated synthetic rules, it reached only 12.9 percent. The authors interpret this gap as evidence that modern LLMs can often avoid an error once it is named, but still struggle to infer the hidden difficulty on their own.

Generic LLM-as-a-judge scoring, by contrast, tended to rate such translations as “good,” and judge-based rankings showed more self-preference. In a small-scale human study, 22 fluent annotators who evaluated 317 examples placed human translations at the top, an ordering that agreed with rule-based verification better than with generic judges or conventional metrics.

The authors caution that LTB is a stress test, not a measure of everyday translation quality. It is intentionally biased toward rare and tricky examples, and the verification rules focus on specific failure modes rather than every aspect of quality. Passing more examples would mean progress on hard cases, not proof that translation is solved. That distinction, in their view, is exactly why the benchmark is needed: progress becomes visible in the failures, not just in averaged scores.