AI Papers Reader

Personalized digests of latest AI research

View on GitHub

The Self-Evolving AI Assistant: How KnowAct-GUIClaw Learns to Navigate Your Apps Like a Pro

Imagine asking a digital assistant to find a flight on a travel app, copy the details to your calendar, and text a summary to your spouse. Today’s AI agents can handle parts of this, but they are notoriously clumsy when interacting with graphical user interfaces (GUIs). They act like amnesiacs: every time they open an app, they must click through menus from scratch, often getting stuck in repetitive loops or hallucinating information as they jump between screens.

To solve this, researchers from the Harbin Institute of Technology, Shenzhen, and the Shenzhen Loop Area Institute have unveiled KnowAct-GUIClaw. This novel framework allows AI personal assistants to build a self-evolving library of “skills” and “memories” based on past successes and failures. By doing so, the AI learns to navigate phone and desktop applications with human-like efficiency, achieving state-of-the-art performance across Android, iOS, HarmonyOS, and Windows.

At the heart of the system is the philosophy “Know Deeply, Act Perfectly,” organized into a four-step loop: Know, Route, Act, and Reflect.

Divvy and Conquer: The Host and the Executor

Instead of relying on a single, massive AI model to do everything, KnowAct-GUIClaw splits the work between two distinct entities: a highly capable “host” agent (the manager) and a lightweight “GUI executor” (the hands).

The host handles the big picture, managing conversation context and user profiles. The GUI executor is only summoned when the system absolutely needs visual interaction. For example, if you ask the agent to “set an alarm one hour before tonight’s party,” the host reads your email, calculates the correct time mathematically, and hands off only the final task—setting the alarm—to the GUI executor. This division of labor slashes computing costs and prevents errors.

Concrete Intuition: How It Remembers and Shortcuts

To understand how KnowAct-GUIClaw improves on standard agents, consider two concrete examples of how it uses its self-evolving memory and skills:

  1. Learning from Mistakes (The Mastodon Example): In a test involving the social network Mastodon, the agent was tasked with creating an invitation link. Normally, an AI agent gets stuck wandering endlessly through the mobile app’s settings menu because the mobile version doesn’t support invite-link creation. However, during the “Reflect” stage of a failed run, KnowAct-GUIClaw writes a post-run lesson: “Advanced invite links require the web admin panel.” On future attempts, the agent retrieves this advisory memory, completely bypasses the mobile app, and immediately launches the web browser to successfully complete the task.

  2. Taking Shortcuts (The Price Comparison Example): If asked to compare the price of a solid-state drive on two shopping apps, JD and Taobao, a standard agent would open JD, click the search bar, type the product name, and hit enter. KnowAct-GUIClaw, however, utilizes a “shortcut” store. It validates on-device deep links to bypass lengthy navigation. It launches directly into JD’s product-results page in one step. It then navigates Taobao normally, collects both prices, and passes them back to the host via a shared “blackboard” to make the final recommendation.

Unmatched Performance

By reflecting on its actions and distilling trajectories into reusable Python-like skills, KnowAct-GUIClaw dramatically cuts down on execution steps. On the rigorous MobileWorld benchmark, KnowAct-GUIClaw achieved a 64.1% success rate, outperforming top-tier closed-source models like GPT-5.5.

Remarkably, these learned skills are transferable. When skills distilled by a highly advanced model (Kimi-2.6) were handed to a smaller, open-source model (Qwen3.5), the smaller model’s success rate shot up by 16.2%.

By allowing AI to learn from its digital footsteps, KnowAct-GUIClaw paves the way for assistants that don’t just follow orders, but actually get smarter the more we use them.