AI Papers Reader

Personalized digests of latest AI research

View on GitHub

A Business-Task Benchmark Puts AI Agents' “Self-Evolution” to the Test

AI agents are being asked to handle multi-step business work, and ideally they would get better with experience. Measuring that improvement is difficult: static public benchmarks are vulnerable to data contamination, and their training/test splits often cannot show whether gains actually came from learning. In a new preprint, researchers introduce GDPevo, an “evolution-native” benchmark designed to test whether agents can learn from prior tasks and transfer that knowledge to new ones.

The benchmark, built by a fully automated pipeline, contains 240 tasks in 24 groups spanning CRM, ERP, finance, healthcare, legal, and data-analysis workflows. Each group shares a simulated business environment, with 5 training and 5 held-out test tasks. Its core mechanism, rule hybridization, decomposes workflows into atomic business rules, distributes subsets across training tasks, and recombines them in test tasks. Because the rules are company-specific and absent from general world knowledge, an agent must infer them from training and compose them at test time. A deterministic rule-based grader, translated from rubric points into code, scores every task.

A failure case shows why this matters. In court-administration task group tg018, DeepSeek-V4-Pro-Preview with Claude Code scored 48.36% on held-out tasks with no training, but 39.83% after fewshot training, in which the agent saw training questions and gold answers. The authors found that the generated skill learned local heuristics while dropping their scope conditions. In one test, the model treated a continued case with no final order as closed and chargeable, reporting nine material conflicts instead of seven; it scored 17.78% versus 40.00% for the base agent. Training had made it worse.

Still, evolution generally helped. Across four agent configurations—GPT-5.5 with Codex, Opus-4.8 with Claude Code, GLM-5.2 with Claude Code, and DeepSeek-V4-Pro-Preview with Claude Code—every supervision type beat the no-training baseline by 2.59 to 16.44 percentage points. Fewshot supervision gave the largest gains. Opus-4.8 with Claude Code rose from 50.63% to 67.07%; GPT-5.5 with Codex rose from 49.37% to 64.51% while cutting test-time cost by a relative 20.88%. With skill-generation overhead amortized, that configuration’s cost was $1.296 per task, nearly identical to the $1.294 base cost. Yet the best evolved score stayed far below a fully informed oracle ceiling of 91.6%, showing current self-evolution is far from complete.

The study also finds tradeoffs. In cross-domain transfer tests across CRM, ERP, and finance groups, fewshot skills overfit their source domain: five of six off-diagonal transfers were negative, with the worst losing 5.0 percentage points. Reflection-based supervision, which uses outcome feedback instead of gold answers, transferred more robustly—up to 6.5 percentage points—but sacrificed some within-domain specialization. The authors also report that a deliberately naive skill-creation method matched more elaborate ones, suggesting the model’s intelligence, not the harness method, drives most of the gain.

GDPevo’s broader significance is as a measurement tool: its automated pipeline can regenerate task groups to counter contamination, and deterministic graders make failures traceable. It does not prove that agents can reliably improve on the job; it offers a way to see where they fall short.