AI Papers Reader

Personalized digests of latest AI research

View on GitHub

When AI Watches Video, Following Instructions Is Still Hard

Multimodal large language models (MLLMs) can summarize a video, locate actions, and answer questions about what they see. Real users often add requirements: “only after the ingredients are mixed,” “as a JSON object,” “in Spanish,” “under 150 words.” A new benchmark suggests these instructions are where video AI still stumbles.

The benchmark, Video-IFBench, comes from researchers at Tencent, Shanghai Jiao Tong University, and other institutions. It contains 1,500 prompts built from more than 700 public videos totaling about 49 hours. Unlike conventional video benchmarks, which grade factual accuracy, Video-IFBench scores whether models comply with constraints attached to a task. The prompts cover 32 task types and 39 constraints, split between semantic requirements (temporal scope, which people to mention) and format requirements (JSON output, response length, banned words). Instructions appear in four structures: single, multi, selection, and nested.

The team built the dataset with a semi-automatic pipeline. An MLLM extracts global descriptions and timestamped facts; another stage turns those facts into instructions and conditional branches, using true/false variants to create selection and nested prompts. Humans verify every sample. For scoring, the authors combine an LLM-as-Judge with programmatic checks. They report two metrics: TCSR, the average fraction of constraints satisfied after a model has addressed the required task, and TISR, a stricter measure requiring the task to be covered and every checklist item satisfied.

Conditional prompts expose a distinctive failure mode. In one selection example built from a cell-biology video, the instruction offered seven branches, each describing different on-screen structures with different tasks; only one branch matched the footage. Qwen3.5-397B-A17B-Think, one of the stronger open-source models tested, answered from the wrong branch, producing a log of lysosomes, mitochondria, and other structures instead of executing the active branch’s requested task. The paper identifies this as a wrong-branch error, a mistake that accuracy-only benchmarks would likely miss.

Overall, the best model, Google’s Gemini-3-Pro, reached only 76.5 TCSR and 54.5 TISR. The best open-source model, Qwen3.5-397B-A17B-Think, scored 69.6 and 46.1, trailing by 6.9 TCSR points and 8.4 TISR points. Across all models, TISR was lower than TCSR, meaning models often satisfied some constraints but not all. Nested instructions were especially hard: Gemini-3-Pro scored 53.7 TCSR and 46.0 TISR on that subset, while the best open-source model managed only 33.0 and 28.2.

Constraint load mattered too. On single-task prompts, Gemini-3-Pro’s TISR fell from 62.7% with one to four constraints to 46.9% with nine or more. Some weaker open-source models collapsed more dramatically: Qwen3-Omni-30B-A3B-Instruct dropped from 40.1% to 3.5%. The authors report that semantic constraints were consistently harder than formatting rules.

The study has limits. The authors note that swapping the LLM judge changes absolute scores, although rankings and trends stayed consistent. They also found that “thinking” variants did not always help; some InternVL3.5 models performed worse in thinking mode. The results are a diagnostic snapshot, not a proof of real-world reliability. Still, the researchers argue that instruction following should become a first-class evaluation dimension for video MLLMs—and that current models have a long way to go.