AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Bridging the Digital and Physical: AI Agents Learn to Navigate Both Worlds

Artificial intelligence agents have traditionally been confined to either the digital realm, sifting through vast amounts of online information, or the physical world, navigating and interacting with real-world environments. This separation has limited their ability to tackle tasks requiring a blend of both—like following an online recipe to cook a meal or using online maps to plan a physical journey. A new research paper introduces EMBODIED WEB AGENTS, a novel paradigm aiming to create AI agents that can seamlessly bridge these two domains.

The researchers have developed a unified simulation platform that integrates realistic 3D indoor and outdoor environments with interactive web interfaces. This platform allows AI agents to perform complex tasks that necessitate interaction with both the physical and digital worlds. To showcase the capabilities of these agents and identify current challenges, they have also released the EMBODIED WEB AGENTS Benchmark.

The benchmark features a diverse array of tasks, including:

  • Cooking: Agents must understand online recipes, identify ingredients within a simulated kitchen, and execute the cooking steps. For example, an agent might need to follow a recipe that instructs it to “slice the apple and the bread” for a sandwich.
  • Navigation: Agents are tasked with following directions obtained from online maps to reach a specific destination in a simulated urban environment.
  • Shopping: This involves tasks like browsing online stores for products, comparing prices, and then navigating to a physical store to complete a purchase. Imagine an agent needing to find the cheapest 500g of cheese online and then making its way to the correct store.
  • Traveling: Agents need to use web resources like Wikipedia to gather information about landmarks while navigating a simulated city, for instance, learning about a Gothic building’s history while walking past it.
  • Geolocation: Agents must use visual cues from their environment and potentially web searches to determine their precise location, akin to playing a sophisticated version of GeoGuessr.

Experimental results presented in the paper highlight a significant gap between the performance of current state-of-the-art AI systems and human capabilities. While agents can often perform well within either the digital or physical domain individually, they struggle with tasks that require fluid transitions and coordinated reasoning across both. The research identifies “cross-domain errors” as the primary bottleneck, where agents become trapped in repetitive loops within one domain or misinterpret instructions that span both.

The introduction of EMBODIED WEB AGENTS and its accompanying benchmark marks a crucial step towards developing more integrated and versatile AI systems. By creating a testbed for this complex, interconnected intelligence, the researchers aim to foster further advancements in AI that can more effectively understand and interact with our multifaceted world.