AI Papers Reader

Personalized digests of latest AI research

View on GitHub

'Vibe Working' Reality Check: New Benchmark Reveals AI Agents Still Struggle with Real Office Work

As AI coding assistants gain widespread adoption, tech companies are eager to push large language model (LLM) agents into broader professional workflows—a shift industry insiders call “vibe working.” But while an AI can easily draft a brief email or summarize a document, can it handle complex, multi-hour office projects? A new benchmark developed by researchers at Baidu, titled OmegaUse-OfficeVal, suggests that while AI agents are remarkably cheap and fast, they remain far behind human workers in producing reliable, publication-ready deliverables.

Unlike traditional AI benchmarks that test brief UI clicks or synthetic exercises, OmegaUse-OfficeVal evaluates agents on 100 authentic, long-horizon office workflows across Microsoft Word, Excel, PowerPoint, and PDF files. Derived from real workplace requests, these assignments mirror tasks typically assigned to junior staff or interns, requiring an average of 2.32 hours of human labor to complete. Crucially, the benchmark pairs every task with two economic metrics: recorded human completion time and a market price proxy (averaging $6.86 per task, with complex jobs priced up to $29).

To appreciate the benchmark’s rigor, consider a task from its treasury and accounting suite. An agent is asked to build a custom macro button in Excel that dynamically highlights a transfer slip across workbooks and triggers a print preview in one click—a job requiring sustained execution that took human annotators over five hours. In an environmental engineering scenario, an agent must recreate a detailed industrial flow diagram from a visual reference into an editable, single-slide PowerPoint presentation, adhering precisely to font choices, 0.75-point line widths, and exact visual centering.

Evaluating these digital workers presented a unique challenge. To avoid subjective human scoring or unreliable “LLM-as-judge” evaluations, the researchers created deterministic, code-based verifiers derived from detailed rubrics. These automated tools open and inspect the agent’s final files, checking basic usability (whether the file opens and remains editable) and specific requirements. Importantly, the rubrics penalize “collateral damage”—such as scrambled layouts, altered text, or broken formulas—that would force a human manager to waste time repairing the document.

The researchers tested several frontier models—including GLM-5.2, Qwen3.7-Plus, Kimi K2.6, DeepSeek-V4-Pro, and MiniMax M3—against a human baseline. The results reveal a glaring gap between speed and deliverable quality. Human annotators achieved a deliverable quality score of 27.79. GLM-5.2 led the AI models with a score of 17.91, followed by Qwen3.7-Plus at 17.51. As task length grew beyond three hours, AI performance degraded sharply, with models frequently suffering total failures due to cumulative structural errors.

Financially, the AI agents look enticing: models finished tasks in minutes for pennies (Qwen3.7-Plus averaged just $0.21 in token costs per task). However, extreme cost efficiency offers little value if the resulting spreadsheet or slide deck is corrupted or unusable. OmegaUse-OfficeVal highlights that before AI can power true “vibe working,” models must evolve beyond rapid generation to master long-horizon planning, precision layout control, and rigorous quality assurance.