AI Papers Reader

Personalized digests of latest AI research

View on GitHub

New Benchmark MIRAI Challenges LLM Agents to Forecast Global Geopolitical Events

In a major push to test the reliability and skill of artificial intelligence in high-stakes diplomacy, researchers have introduced MIRAI (Multi-Information FoRecasting Agent Interface), a novel benchmark designed to evaluate Large Language Model (LLM) agents on their ability to accurately predict future international events.

While modern LLMs have shown impressive reasoning prowess, accurately forecasting complex geopolitical shifts—such as alliances, conflicts, or trade agreements—requires autonomous information gathering and deep temporal reasoning over diverse data. MIRAI addresses this gap by creating an agentic environment where LLMs must actively consult an extensive database of structured historical events and textual news articles derived from the refined GDELT database (Global Database of Events, Language, and Tone).

The core challenge for the LLM agents lies in mastering three dimensions: autonomously integrating critical information from global databases, utilizing domain-specific APIs (Application Programming Interfaces) via a code-based interface, and jointly reasoning over these diverse, time-stamped facts to predict future relations.

The Complexity of Real-World Forecasting

Unlike typical QA tasks, MIRAI requires agents to operate using a “Think, Act, Observe” strategy (known as ReAct), generating executable code to query data, analyze context, and justify predictions.

For example, an agent might be tasked with forecasting Australia’s (AUS) actions toward China (CHN) on November 18, 2023. The LLM must query historical records, which might show frequent interactions related to economic cooperation (e.g., “Cooperate economically,” code 061) or intent to negotiate (code 036). Simultaneously, the agent must browse news articles that might mention recent high-level visits, efforts to “heal old wounds,” or, conversely, continued concerns over human rights issues or aggression in the Indo-Pacific.

The research shows that successful forecasting hinges entirely on this tool-use capability. Models relying solely on their internal knowledge, such as standard chat models without tool access, underperformed dramatically.

Stronger Models Lead, But Challenges Remain

The comprehensive benchmarking revealed that international event forecasting is highly difficult, even for state-of-the-art models. The highest-performing agent, GPT-4o utilizing its full suite of APIs, achieved only a 29.6% F1 score in predicting fine-grained, second-level relations (specific actions defined by the CAMEO ontology).

Moreover, the complexity increases significantly with time horizon. Predicting events 30 or 90 days out proved substantially harder than short-term forecasts (1 or 7 days), demonstrating the difficulty LLMs have in capturing long-term temporal trends and anticipating trend shifts.

The study also highlighted a divide in performance based on the model’s action strategy. Stronger LLMs like GPT-4o gained significant advantages from the “Code Block” action, which allows the agent to write more flexible, multi-line Python code to process data and run complex analyses. Weaker models, however, struggled with this flexibility, frequently running into code execution errors.

By establishing MIRAI as a rigorous standard, researchers aim to encourage the development of more accurate, sophisticated, and trustworthy LLM agents capable of aiding policymakers and analysts in navigating the dynamics of an interconnected world.