NICO-RAG: Multimodal Hypergraph Retrieval-Augmented Generation for Understanding the Nicotine Public Health Crisis

TL;DR

This paper introduces NICO-RAG, a multimodal retrieval-augmented generation framework utilizing hypergraph-based knowledge representation to enhance understanding of the nicotine public health crisis through large-scale image and text data.

Contribution

It presents NICO-RAG, a novel multimodal RAG framework that efficiently retrieves image features and constructs hypergraph-based factual responses for public health research.

Findings

NICO-RAG performs comparably to state-of-the-art methods on over 100 questions.

The NICO dataset includes over 200,000 multimodal samples on nicotine products.

Hypergraph organization improves factual accuracy in multimodal retrieval.

Abstract

The nicotine addiction public health crisis continues to be pervasive. In this century alone, the tobacco industry has released and marketed new products in an aggressive effort to lure new and young customers for life. Such innovations and product development, namely flavored nicotine or tobacco such as nicotine pouches, have undone years of anti-tobacco campaign work. Past work is limited both in scope and in its ability to connect large-scale data points. Thus, we introduce the Nicotine Innovation Counter-Offensive (NICO) Dataset to provide public health researchers with over 200,000 multimodal samples, including images and text descriptions, on 55 tobacco and nicotine product brands. In addition, to provide public health researchers with factual connections across a large-scale dataset, we propose NICO-RAG, a retrieval-augmented generation (RAG) framework that can retrieve image features without incurring the high-cost of language models, as well as the added cost of processing image tokens with large-scale datasets such as NICO. At construction time, NICO-RAG organizes image- and text-extracted entities and relations into hypergraphs to produce as factual responses as possible. This joint multimodal knowledge representation enables NICO-RAG to retrieve images for query answering not only by visual similarity but also by the semantic similarity of image descriptions. Experimentals show that without needing to process additional tokens from images for over 100 questions, NICO-RAG performs comparably to the state-of-the-art RAG method adapted for images.