AI Papers Reader

Personalized digests of latest AI research

View on GitHub

RePIC: Reinforcing Multimodal Models for Personalized Image Captioning

Recent advancements in AI have seen the rise of powerful multimodal large language models (MLLMs) capable of understanding and generating text related to images. However, a significant challenge remains in their ability to generate personalized image captions that accurately incorporate user-specific details. Existing methods often rely on supervised fine-tuning (SFT) with large datasets of captions, but this approach can be costly to curate, especially for complex scenarios like multi-concept image captioning (i.e., describing images with multiple people or objects).

A new framework, RePIC (Reinforced Post-Training for Personalized Multi-Modal Language Models), introduces a reinforcement learning (RL)-based approach to tackle this personalization problem. The researchers behind RePIC, from Seoul National University, argue that SFT methods struggle with personalized image captioning due to limitations in visual recognition and generalization.

RePIC aims to enhance two key capabilities in MLLMs for personalization:

  1. Robust Visual Recognition: The ability to consistently identify objects across various conditions like different lighting, poses, or backgrounds. This ensures captions are accurate and faithful.
  2. Consistent Personalized Generation: The ability to weave personal information, such as names, into generated captions correctly.

The RePIC framework employs three core components, all powered by RL:

  • Object Consistency Tuning (OCT): This component uses pairs of images (positive for the same object, negative for different objects) to train the model to recognize objects consistently, rewarding correct “yes/no” answers to questions like “Is visible in this picture?".
  • Visual Localization Tuning (VLT): To improve spatial understanding, VLT uses the Intersection over Union (IoU) score to reward accurate bounding box predictions for specified regions in an image. This helps the model understand object placement.
  • Identity Consistency Tuning (ICT): This component focuses on ensuring personal names are correctly incorporated. It uses positive pairs of images and associated names, rewarding captions that accurately use the provided names. For multi-concept scenarios, it rewards captions that include a proportion of the given names.

The paper demonstrates that RePIC significantly outperforms existing SFT-based personalization methods, particularly in challenging multi-concept image captioning tasks. For instance, in a scenario with four distinct concepts in an image, RePIC produces faithful and detailed captions, accurately identifying and describing each concept, while prior SFT methods struggle. The research also highlights that while SFT methods might perform well on single-concept tasks or with abundant, high-quality data, they falter in more complex, diverse, or out-of-distribution scenarios. RePIC’s RL-based approach offers greater robustness and generalization, making it a promising direction for truly personalized AI experiences.