AI Papers Reader

Personalized digests of latest AI research

View on GitHub

PC Agent: Letting AI Handle Complex Digital Work While You Sleep

A team of researchers from Shanghai Jiao Tong University and Generative AI Research Lab (GAIR) have unveiled PC Agent, a groundbreaking AI system capable of performing complex digital tasks autonomously. This development significantly advances the capabilities of digital agents, moving beyond simple task execution to handling intricate workflows that require sustained effort across multiple applications and sophisticated decision-making. The paper detailing PC Agent, published on arXiv, highlights three key innovations that enable this leap: PC Tracker, a cognition completion pipeline, and a multi-agent system.

Current digital agents, while able to perform simple tasks like web searches, often struggle with complex activities such as creating presentations or editing videos. These more complex activities demand sustained interaction across multiple applications, sophisticated planning, and flexible adaptation to changing circumstances—capabilities that previous AI systems lacked. PC Agent addresses these limitations by focusing on human cognition transfer. The researchers’ central insight is that truly capable digital agents must learn not just what humans do on a computer, but also why and how they make decisions during complex work.

PC Tracker: Capturing the Cognitive Context

PC Tracker is a lightweight data collection infrastructure that efficiently records human-computer interaction trajectories. Instead of simply recording screen activity like a standard screen recorder, PC Tracker captures the full cognitive context. This includes not only the user’s actions (clicks, keystrokes, scrolling) but also the system state at each action, creating a rich dataset of events. This contextual information is crucial for understanding the why behind each action, rather than just the what.

For example, a simple task like adding a title to a PowerPoint slide involves multiple actions: opening PowerPoint, selecting the title box, typing the title, and potentially saving the file. PC Tracker doesn’t just record the sequence of clicks and keystrokes. It also captures the visual context (the screen captures at each point), providing the AI with the necessary information to understand the purpose of each action within the overall task.

Cognition Completion: Transforming Data into Cognitive Trajectories

The raw data collected by PC Tracker is then processed using a two-stage cognition completion pipeline. First, it refines the raw data, filtering out irrelevant actions and standardizing the format. Second, it employs large language models (LLMs) to complete the action semantics (adding descriptive labels and explanations) and infer the underlying human thought processes behind each action. This converts the raw interaction data into “cognitive trajectories” – a detailed representation of the user’s cognitive process during task execution.

For instance, a simple mouse click on a “save” button isn’t just a click. The cognition completion pipeline enhances it with a description such as “click the save button to save the current presentation to the desktop” and might even infer the underlying human thought: “I’ve finished creating the presentation, so I should save it now.”

Multi-Agent System: Robust Execution and Error Handling

Finally, PC Agent uses a multi-agent system comprising a planning agent and a grounding agent to execute tasks. The planning agent, trained on the cognitive trajectories, is responsible for high-level task planning and decision-making. The grounding agent, leveraging advanced visual grounding capabilities, precisely locates GUI elements and executes the actions planned by the planning agent. This system design incorporates self-validation mechanisms; for example, the grounding agent checks if a requested action (like clicking a button) is even possible given the current screen state, providing robustness and error handling.

Data Efficiency and Future Directions

The researchers demonstrate the data efficiency of their approach. PC Agent, trained on only 133 cognitive trajectories, can successfully handle complex tasks involving up to 50 steps across multiple applications. This impressive result suggests that focusing on high-quality, cognitively rich data is far more effective than simply using large amounts of raw interaction data for training AI agents. The researchers make their framework, including PC Tracker and the cognition completion methods, open-source, fostering further research and development in this promising area.

Future work will focus on improving robustness and long-term planning, enhancing the handling of complex mouse actions (like drag-and-drop), and exploring the potential of task-free data for pre-training. PC Agent represents a significant leap towards creating truly capable digital agents that can seamlessly handle complex real-world work, freeing up human time and energy for more creative and strategic tasks.