AI Papers Reader

Personalized digests of latest AI research

View on GitHub

New Benchmark 'MEMTRACK' Tests AI Agents' Long-Term Memory in Realistic Workflows

San Francisco, CA – October 26, 2023 – Researchers have introduced MEMTRACK, a novel benchmark designed to evaluate the long-term memory and state-tracking capabilities of AI agents in complex, multi-platform enterprise environments. Unlike previous benchmarks that often focus on simple conversational scenarios, MEMTRACK simulates realistic organizational workflows, integrating asynchronous events across platforms like Slack, Linear, and Git. This approach aims to push the boundaries of AI agent development by testing their ability to handle noisy, conflicting, and cross-referencing information over extended periods.

The core of MEMTRACK lies in its meticulously crafted dataset, which comprises 47 distinct instances. Each instance presents a chronological timeline of events, interwoven with questions that require agents to access and synthesize information from various sources. These scenarios are designed to mimic real-world software development processes, forcing agents to grapple with concepts such as information acquisition, selection, and conflict resolution.

For example, an agent might need to track a software bug reported on Slack, follow its resolution through a series of Linear tickets, and even access code changes in a Git repository. The timeline might contain deliberately misleading or incomplete information, mirroring the challenges of information management in actual organizations. One such scenario could involve an agent needing to determine the root cause of a performance issue. This would require piecing together information from Slack discussions about the problem, Linear tickets detailing implementation efforts, and potentially even code reviews from Git to pinpoint the exact line of code responsible.

The MEMTRACK dataset was created through a combination of manual expert curation and a scalable agent-based synthesis approach. This ensures that the scenarios are both ecologically valid—meaning they reflect real-world situations—and challenging enough to test advanced memory capabilities. The researchers also emphasize that MEMTRACK is designed to be backend-agnostic, meaning it can be used to evaluate various memory storage and retrieval mechanisms.

Key metrics for evaluating agent performance on MEMTRACK include Correctness, Efficiency, and Redundancy. Correctness measures how accurately the agent answers questions, while Efficiency assesses how effectively it utilizes available tools without excessive calls. Redundancy focuses on the agent’s ability to avoid repeatedly fetching the same information, a crucial aspect of efficient long-term memory usage.

Experiments conducted using state-of-the-art Large Language Models (LLMs) like GPT-5 revealed significant challenges. Even the best-performing models struggled to maintain context over long horizons, handle cross-platform dependencies, and resolve contradictions effectively. Notably, the top GPT-5 model achieved only a 60% Correctness score on MEMTRACK, highlighting the need for more robust memory solutions for AI agents.

The researchers hope that MEMTRACK will serve as a crucial framework for advancing evaluation research in memory-augmented agents. By moving beyond conversational benchmarks, MEMTRACK sets the stage for developing more sophisticated, multi-agent, and multi-platform AI systems capable of tackling complex organizational tasks. The MEMTRACK dataset is publicly available for researchers to use and contribute to the field.