AI Papers Reader

Personalized digests of latest AI research

View on GitHub

OneKE: A Dockerized, Multi-Agent System for Robust Knowledge Extraction

A team of researchers from Zhejiang University and Ant Group have unveiled OneKE, a novel knowledge extraction system designed to overcome limitations of existing approaches. Published in a recent preprint, OneKE leverages the power of Large Language Models (LLMs) within a sophisticated multi-agent framework, enhancing both accuracy and adaptability. The system is dockerized for easy deployment and has been open-sourced, fostering broader accessibility and collaborative development.

The core innovation of OneKE lies in its multi-agent architecture. Three distinct agents collaborate to perform the knowledge extraction task:

  1. Schema Agent: This agent acts as the system’s architect, determining the output schema—the structure of the extracted information. It can handle various data types (HTML, PDF, text) and either use pre-defined schemas or dynamically deduce the appropriate structure using the LLM, depending on user input. For example, if a user wants to extract information about chemical reactions from a scientific paper (PDF), the Schema Agent would create a schema with fields like “reactants,” “products,” and “reaction conditions.” If the user provides no specific schema instructions for news data, it generates a generic schema fitting common news articles.

  2. Extraction Agent: This is the workhorse, using the LLM and the schema provided by the Schema Agent to extract the information from the source document. OneKE’s design accommodates multiple LLMs (LLaMA, GPT-4, etc.), allowing for flexibility and leveraging the strengths of different models. It employs a case retrieval mechanism, incorporating successful past extractions into the prompt to improve performance. For instance, if the system has previously successfully extracted information from similar news articles, that knowledge is incorporated to guide the extraction from a new news article.

  3. Reflection Agent: This agent is responsible for quality control and error correction. It compares the extracted information with previously stored cases (both successful and unsuccessful extractions) to identify errors and refine the output. This iterative process allows the system to learn from its mistakes, gradually improving its performance. If the Extraction Agent misses a crucial piece of information during news extraction, the Reflection Agent can identify this gap using past examples of successful extractions and prompt the Extraction Agent to re-attempt the task.

OneKE’s architecture also incorporates a configurable knowledge base, including a Schema Repository (storing pre-defined and user-defined schemas) and a Case Repository (a database of past extractions, including both correct and incorrect examples and the reasons behind the errors). This system allows for fine-tuning the system’s performance and for easy debugging. The user can also update the knowledge base to adapt to new tasks and data types.

The authors evaluated OneKE’s performance on benchmark datasets for Named Entity Recognition (NER) and Relation Extraction (RE), demonstrating significant improvement over baseline models through incorporating case retrieval and reflection. Case studies further highlighted OneKE’s adaptability across diverse domains, including web news extraction and scientific literature analysis. This makes the system particularly appealing for various real-world applications.

OneKE represents a significant advancement in knowledge extraction by building a robust and adaptable system, combining the power of LLMs with a carefully designed multi-agent architecture and a flexible knowledge base. Its open-source nature promises to stimulate further development and widespread adoption in various fields.