A Statistical Check for Swapping Human Annotators with AI
Researchers increasingly ask large language models (LLMs) to label data that people once labeled, from rating chatbot answers to flagging hateful content. The shortcut saves time and money, but it raises a basic question: when is an LLM’s label good enough to replace a human’s? A new paper from researchers at the Technion, the University of Haifa, and Google Research proposes a statistical test for that decision and reports that replacement passed in about half of the cases they examined.
Two existing tests, the Alternative Annotator Test (ALT-TEST) and the Statistical Turing Test (STT), assume that the same annotators labeled every item. Many real projects do not work that way. Crowdworkers choose their own tasks and drop out, and many datasets omit annotator identities. The authors report that 11 of their 20 datasets did not meet this requirement.
Their method, ALTRA-TEST, works with whatever annotations exist. It repeatedly resamples items, draws three human labels for each, and swaps one of them for the LLM’s label. This yields two panels: an all-human panel and a mixed human-LLM panel. Both are scored with Krippendorff’s alpha, a standard measure of inter-annotator agreement (IAA) that corrects for chance. The LLM passes if the mixed panel’s agreement is statistically non-inferior to the human panel’s, meaning it is not meaningfully lower. A small loss is tolerated, scaled by how much humans already disagree, so the same rule applies to objective and subjective tasks alike.
The paper gives a concrete illustration using DICES-350, a conversational-safety dataset whose annotators fall into 13 demographic groups. Across the whole pool, four LLMs passed when using their best prompting technique: Gemma-3-12B, Gemini-3.1 Pro, GPT-5.6-Terra, and GPT-5.6-Luna. When the test required passing within every group, only Gemma-3-12B succeeded; the other three failed for the Latinx group. The authors argue that pooled testing averages out such misalignments.
In simulations where the true label is known, ALTRA-TEST accepted more genuinely better LLMs than ALT-TEST and STT at the same false-acceptance rate. With 200 items and the default margin constant of 0.1, the better-model pass rate was 0.98, while the worse-model pass rate stayed below 0.05. Performance was reasonable at smaller samples too, with a better-model pass rate of 0.88 at 100 items.
On real data, the authors tested 18 LLMs and five prompting techniques across 20 datasets spanning text, image, audio, and video. Under chain-of-thought prompting, 71 of 201 judge-dataset pairs (35.3%) passed. Allowing each judge its best technique raised this to 95 of 201 (47.3%). Gemini-3.1 Pro passed 16 of 20 datasets. For passing judges, the median minimum sample needed was 111 items. On CIFAR-10H, which includes gold labels, eight of nine LLM-prompt combinations at least as accurate as humans passed, and none of the 36 less accurate combinations did.
The authors stress several caveats. The verdict is relative to the observed human pool: in simulations, a random LLM began passing when human agreement fell below about 0.1, so they recommend testing only when IAA is at least 0.2. The test also cannot detect errors that LLMs share with human annotators, and it evaluates one label per item rather than the full distribution of opinions. Outcomes varied widely by dataset, which the authors take to mean that LLM annotators should be validated task by task.
The work offers a statistically grounded check for a decision many teams currently make informally. It does not, however, establish that an LLM is correct; it establishes only that the LLM preserves the agreement humans already share, and only within the datasets tested.
Chat about this paper
To chat about this paper, you'll need a free Gemini API key from Google AI Studio.
Your API key will be stored securely in your browser's local storage.