AI Papers Reader

Personalized digests of latest AI research

View on GitHub

When AI Web Agents Need to Remember, FocusMem Factorizes the Problem

Software agents that operate web browsers need memory at two timescales: reusable experience from finished tasks and progress within the current task. Storing entire interaction histories is token-hungry, so some systems compress screenshots and actions into compact “latent” memory tokens. That compression has three failure modes, according to researchers at Peking University and collaborators: important details can be lost; one fixed summary may not fit every decision stage; and a retrieved memory can be irrelevant or misleading.

Their new system, FocusMem, separates those responsibilities while keeping the policy—the vision-language model that chooses actions—frozen. A role-aware content basis stores episodic evidence (lessons from completed trajectories) and working evidence (progress and constraints from the current task) differently. A state-conditioned readout uses the current decision context to generate a decision-specific view of the same evidence, so one trajectory can inform different steps in different ways. A trust gate then scores each memory block and drops blocks that look irrelevant to the current step.

A concrete case from the paper shows the difference. On an MMInA-Shopping task that asked for the body color of a specific coffee mug, the Qwen3-VL-8B policy without memory searched the store and then scrolled through the results list for all 15 permitted steps, never opening the product page. The same frozen policy with FocusMem used a compressed episodic memory from a related coffee-cup task to click the first product image on step 3, and produced the correct answer, “Blue,” in five steps.

The main evaluation covered five benchmarks: MMInA-Wikipedia, MMInA-Shopping, DeepShop, WebVoyager, and Online-Mind2Web. FocusMem’s unweighted average success rate was 48.3%, versus 33.7% for the paper’s primary baseline, Action-only Fixed, which also uses dual fixed latent memory but without the factorization. The gains were 10.8, 13.5, 18.0, 17.0, and 13.9 percentage points on the five benchmarks, respectively. FocusMem also averaged higher than the strongest prior latent-memory adaptation, Mem-W-style, at 42.5%, and far above the 28.5% no-memory average, although the authors call no-memory a system-level reference rather than a controlled baseline.

A cumulative ablation on MMInA-Shopping showed each added component contributed: success rose from 41.0% with no memory to 45.5% with fixed latent memory, 50.5% after adding the content basis, 54.5% after state-conditioned readout, and 59.0% after the trust gate. The extra computation is not free: time per step increased by 1.70 times, though fewer wasted steps meant per-task time grew only 1.51 times.

The authors are careful about scope. FocusMem was tested with one frozen policy, Qwen3-VL-8B, on web benchmarks; other backbones and mobile or desktop environments remain untested. The trust diagnostic injected clearly irrelevant episodic trajectories, so subtler mismatches or stale working memory were not covered. Automated judges also carry some residual bias, even with method-blind prompts.

The broader implication is that effective latent memory is not just about compressing more history. As the researchers frame it, what matters is what is retained, what is exposed at a given decision, and whether the agent is allowed to use it. The results support that division of labor, without proving it is the only way to build memory for browser agents.