AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Benchmark Links Obfuscated Message Recovery to Evidence-Based Web Risk Investigation

Platform abuse campaigns hide redirection instructions using emojis, homophones, decomposed characters and redundant symbols, then send users to services tied to pornography, fraud, gambling or illicit transactions. Existing benchmarks evaluate obfuscated text and risky webpages separately, so they miss a key governance problem: a small restoration error can change which website is investigated downstream. Researchers introduced RiskChainBench to measure that chain.

The benchmark pairs 3,600 synthetic token-text restoration inputs from 600 source sessions with 600 human-labeled local web environments. Each source has six variants; the recipes separately stress composite restoration, phonetic substitutions, entry confusables (lookalike characters in destination entries), mixed lexical and platform-token corruption, few-line entry layouts, and grapheme-safe vertical entry layouts. In the first task, a model restores the canonical message, operational intent and a ranked list of destination entries. The restoration is frozen before browsing. In the second task, the same underlying model acts as a vision-language-model-driven web agent, investigating the correctly associated website and producing an evidence-cited risk report without seeing the source message, its restoration or domain-reputation cues. Restoration and correct-routing web investigation are scored separately.

For end-to-end evaluation, the top-ranked entry from a pre-fixed primary variant acts as an offline gate. For example, if that entry does not resolve to the associated website, the gated result is counted as a failure; lower-ranked candidates cannot repair the gate, and the protocol does not route an incorrect entry to another benchmark site. Human labels decide task correctness, while a fixed multimodal evidence judge assesses faithfulness, sufficiency, completeness and consistency as coverage-conditioned diagnostics.

Across ten models, Entry Top-1 ranged from 35.19% to 95.22%. GPT-5.4 led Entry Top-1 at 95.22%, whereas GPT-5.6 SOL led full reconstruction at 73.31% and had the lowest character error rate, 1.11%. That split shows entry recovery is not the same as reconstructing the whole message. In web investigation under correct routing, decision accuracy ranged from 26.3% to 62.8%; GPT-5.6 SOL led at 62.8%, with hierarchical exact match of 61.0% and decision macro-F1 of 59.3%, while Kimi K2.5 led violation-type macro-F1 at 48.1%.

The entry gate reduced end-to-end accuracy to 16.7%–60.8%, with losses of 0.7 to 32.3 percentage points relative to the same frozen web-only results. Execution failures accounted for 31.9% of web runs, compared with just 0.9% for post-decision typing errors. The authors identify stable exploration and risk judgment, not fine-grained typing, as the principal bottlenecks.

The study has limitations. The authors note that results use one trajectory per model–website pair, several violation types are sparsely populated, and evidence diagnostics depend on successful judge coverage. The messages are synthetic, and the local sandbox is designed for safe, reproducible capability evaluation rather than estimating real-world prevalence. Rerun variance and broader risk types and languages remain for future work.

By linking restoration to active web investigation, RiskChainBench exposes cross-stage failures that separate text and webpage benchmarks can hide. It does not establish real-world moderation performance, but it offers a released benchmark, protocol and resettable sandbox for measuring where the chain breaks and for training more reliable agents.