“Trainee-Bench” Reveals SOTA AI Agents Fail to Master the First Day on the Job
A new benchmark simulating the dynamic, uncertain environment of a corporate internship has exposed significant reliability flaws in state-of-the-art Multi-modal Large Language Models (MLLM) agents, challenging the perception that these systems are ready for autonomous real-world deployment.
The new evaluation framework, dubbed Trainee-Bench, was introduced by researchers from Shanghai AI Laboratory and Fudan University to bridge the gap between idealized laboratory testing and the stochastic realities of production scenarios. Unlike traditional benchmarks that test agents on static, fully observable tasks, Trainee-Bench simulates a “first day” experience where a trainee agent must continuously handle streaming tasks, explore uncertain environments, and learn from past mistakes.
The results show that even the most advanced agents struggle severely. Gemini-3-Flash, the top performer among the tested models, achieved only a 35% overall Success Rate, with performance plummeting in high-stress, complex situations.
Trainee-Bench evaluates agents across three critical competencies, which current models consistently fail to master.
The first challenge is Dynamic Task Scheduling and Multi-task Handling. Agents are hit with a stream of tasks—such as data reviewing, booking a meeting, and planning an event—all with distinct priorities and deadlines. The agent must context-switch efficiently without missing a key deadline.
The second, and perhaps most critical, challenge is Active Exploration in Novel Tasks. In Trainee-Bench, crucial information—referred to as “latent clues” like access passwords or technical manual locations—is deliberately withheld. An agent cannot simply follow initial instructions; it must actively search file systems or engage in multi-turn dialogues with non-player characters (NPCs) to acquire the necessary data. If an agent is tasked with creating an advertising campaign plan, for instance, it must proactively ask an NPC for the required user density data rather than hallucinating the information.
The final challenge involves Continuous Evolution (Learning from Previous Tasks). Trainee-Bench dynamically generates unique task instances based on logical rules, forcing agents to distill generalized strategies rather than relying on rote memorization of fixed datasets. Experiments showed that current learning mechanisms provided only marginal gains (+0.04 Checkpoint Score increase after self-evolution).
The most telling finding emerged when researchers compared autonomous agent performance against performance achieved with human guidance. On “Hard Tasks,” an autonomous agent achieved an average Checkpoint Score of 0.24. When human experts provided tiered hints—such as suggesting the correct knapsack optimization algorithm for a budget planning task—the agent’s score surged to 0.83.
This dramatic disparity highlights a critical reliability gap: current LLM agents are proficient executors when given complete information and a clear plan, but they largely fail as proactive problem-solvers capable of autonomously seeking out missing information or effectively internalizing strategic lessons to prevent repeated errors.
The findings establish a clear path forward for AI development, suggesting a pivot from optimizing isolated skills to building robust mechanisms for exploration, temporal planning, and strategic experience internalization.
Chat about this paper
To chat about this paper, you'll need a free Gemini API key from Google AI Studio.
Your API key will be stored securely in your browser's local storage.