AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Red-Team Safety Tests: New Math Shows When a Clean Sheet Counts

AI developers increasingly point to red-team evaluations as evidence that a model is safe. A new paper argues such tests are neither worthless nor universally decisive: their evidential value is a calculable quantity.

Bandana Kaur of APIsec Research Labs defines the “evidential ceiling” as the largest factor by which one test result can move the odds between elevated and acceptable harm. For a zero-harm benchmark, the evidence is a likelihood ratio. With elevated harm at 0.1% and acceptable harm at half that, 520 prompts produce a ratio of 0.77—barely evidence. At a 1% harm rate, the same benchmark is stronger: a two-fold update needs 138 prompts, a ten-fold update 458, and false certification below 5% needs 299. AdvBench, with 520 prompts, clears all three.

Kaur’s central result is a boundary. For any fixed budget, a minimum harm rate exists above which a finite benchmark can certify safety to a stated standard; below it, no feasible passive benchmark can. With 100,000 prompts, a fifty-percent improvement target, and an evidentiary threshold of 0.5, the boundary is about 1.4 harms per 100,000. For rare, catastrophic harms, the paper concludes, large benchmarks cannot carry certification; other evidence must.

One counterintuitive finding: at 520 prompts, a clean sheet and one reproduced failure carry equal evidence when the harm rate is about 0.133%. Above that, absence is stronger—at 1%, a clean result carries 1.4 times the evidence of one observed harm. Below it, a single failure dominates: at 0.01%, one observed harm carries about 25.7 times as much evidence as zero harms.

Auditing eight suites, Kaur reports that at a 1% harm rate, power to detect a fifty-percent reduction ranges from 14.9% for XSTest (250 prompts) to 59.3% for SafetyBench’s 2,100-prompt Chinese subset; AdvBench has 22.6%, HarmBench 18.7%. None reaches 80%, which would need about 3,679 prompts per arm. Yet power differs from certification. At 0.1% harm, AdvBench’s clean result still carries a 59.4% false-certification probability; cutting that below 5% needs more than 2,995 prompts, beyond every suite reviewed. At 1%, 299 prompts suffice, and seven of eight suites clear that bar.

An example from Anthropic’s Claude 2 model card illustrates the failure mode. It reports four harmful responses among 328 held-out prompts—legitimate at prompt level. But five responses were sampled per prompt, so 1,640 responses were scored while the numerator counted prompts. Because responses within a prompt can correlate, the per-response rate users actually experience is not recoverable.

Kaur acknowledges limits. The math assumes approximately independent trials; adaptive red-teaming could change the scaling, though she argues the natural mechanism would tighten, not loosen, the bound. Her distributional analysis compares benchmarks with ordinary queries from LMSYS-Chat-1M, so it cannot measure distance to real adversarial attack traffic.

The paper’s broader point: safety benchmarks are not uninformative. They are informative about a specific, computable set of propositions. The discipline they need, Kaur writes, is to state which claim an evaluation supports before the test is run.