AI Papers Reader

Personalized digests of latest AI research

View on GitHub

Scientists Build 'Verifiable' LLM for Medicine, Dramatically Reducing Hallucinations

A team of researchers has developed a novel biomedical search system that leverages artificial intelligence to provide answers grounded in scientific literature, with every claim verifiable by a direct citation. This new framework addresses a critical challenge facing Generative Large Language Models (LLMs): the tendency to “hallucinate” or produce confident but factually incorrect information, a risk deemed intolerable in fields like pharmaceuticals and medicine.

The system, a custom-built Retrieval-Augmented Generation (RAG) pipeline, combines an advanced search mechanism tailored for PubMed abstracts with a fine-tuned, open-source LLM. The core innovation lies in forcing the model to act as a rigorous scientific author: every statement generated in response to a medical query is followed by the relevant PubMed ID (PMID), allowing users to check the original source material instantly.

The system’s Information Retrieval (IR) component relies on a powerful hybrid search strategy, combining fast lexical (keyword) matching with deep semantic (meaning-based) understanding. This hybrid approach proved crucial for navigating the specialized terminology of biomedical literature.

For instance, if a scientist queries about “myocardial infarction,” a traditional search might miss relevant abstracts that use only the phrase “acute coronary syndrome.” The new system, prioritizing semantic understanding while maintaining fast lexical indexing, overcomes these language barriers. In evaluations against the established BioASQ dataset, the hybrid search demonstrated an absolute improvement of 23% in retrieval accuracy compared to the standard PubMed search engine, significantly boosting the quality of the source material fed to the LLM.

For the generative step, the researchers fine-tuned versions of the smaller, open-source Mistral-7B Instruct model. The goal was to train the model specifically for referenced question-answering, where it must synthesize information from up to ten provided abstracts and embed citations correctly.

Comparing the fine-tuned Mistral models (M1 and M2) against powerful proprietary models like GPT-4 Turbo showed highly encouraging results. While all models occasionally struggled with hallucinating reference numbers, the fine-tuned M2 model performed best in terms of reliability, producing only 3 instances of mistaken abstract IDs, compared to 79 instances from the M1 model and a higher number of no-reference answers in the zero-shot versions. This high citation fidelity makes the M2 model a reliable foundation for generating clinically trustworthy summaries.

By making the custom training dataset (PQAref) and the fine-tuned models publicly available, the team aims to accelerate the development of verifiable, transparent AI tools in life sciences. This new referenced RAG system promises to increase scientific productivity by delivering reliable, instantly verifiable answers, all while allowing organizations to maintain full control over the models and data internally, avoiding reliance on external third-party APIs.