AI Papers Reader

Personalized digests of latest AI research

View on GitHub

When LLM Agents Keep Memory in Files, Organization Pays—Up to a Point

Many deployed AI agents now store long-term memory the way a developer stores code: as a directory tree of markdown files that the agent itself reads, writes, and reorganizes. The practice has become an industry default, but its two working assumptions—that an agent can keep a growing store organized, and that organization improves answers—had not been systematically tested. In a new preprint, researchers from UIUC, UCSD, UC Merced, Adobe Research, and Texas A&M report the first systematic look. Their central finding is conditional: organization reliably cuts search costs, but it does not reliably improve correctness, and keeping a store tidy gets harder as it grows.

Rather than propose another memory system, the team formalized the one already shipping. A management agent integrates each incoming chunk of conversation or task log into a single markdown filesystem; a search agent answers queries by navigating that filesystem and citing sources; and in a second setup, an execution agent attempts household tasks while its trajectories are distilled into reusable skill files. Across long-conversation benchmarks (LoCoMo, PersonaMem, REALTALK) and the embodied ALFWorld benchmark, the researchers compared this agent-curated store against baselines including no memory, raw chunk retrieval, and verbatim session dumps.

The clearest failure mode appeared when a reorganizing agent was asked only to restructure a verbatim store. The gpt-5.4-mini model silently condensed facts as it rewrote them; adding a single “keep every fact” rule held content roughly fixed. On REALTALK, a 21-day real-message benchmark, the condensing version answered 41.2% of questions correctly versus 77.6% for the preserving version—a 36.4-percentage-point loss caused by the model’s default behavior, not by an instruction to compress.

The payoff of organization showed up in search cost, not accuracy. On the largest conversational stream, PersonaMem 128k, per-query search cost fell from 3.9 cents with a verbatim dump to 1.6 cents for a reorganized store. But no store shape won everywhere: the agent-curated store led on LoCoMo with 86.1% correctness yet collapsed to 37.5% on PersonaMem 32k, where the plain verbatim dump scored 78.1%. On skills, the best memory depended on who consumed it. With the stronger gpt-4.1 execution agent, a verbatim episode log won at 87.1%; with the weaker gpt-4.1-mini, it fell to 66.4%, while synthesized guidance from curated skills reached 76.4%—a 10-percentage-point reversal. The tool harness also mattered: replacing file functions with a bash shell reshaped the store as dramatically as swapping models, producing a 147-file store on PersonaMem 128k where file-tool stores had one or two files.

The authors caution that organization eroded as stores grew: adherence to the taxonomy contract declined for all but the strongest management agent, and no agent they measured converted organization itself into better answers. Curation also carries a real cost—building the agent-curated LoCoMo store consumed about $12.93 in model calls, versus zero for a mechanical dump. And because the experiments spanned one conversation or a 140-task chain, the study cannot say whether organization pays over months of accumulation. Its contribution, the researchers argue, is to turn a shipped-by-default assumption into a design space: memory shape is a variable that can be measured, managed, and deliberately chosen.