Purrfessor: A Fine-tuned Multimodal LLaVA Diet Health Chatbot

TL;DR

Purrfessor is a multimodal AI chatbot fine-tuned with nutrition data that provides personalized dietary advice, integrating visual meal analysis to improve user engagement and perception of care.

Contribution

This work introduces Purrfessor, a novel multimodal chatbot fine-tuned with food data, enhancing dietary guidance through visual analysis and interactive engagement.

Findings

Purrfessor significantly improved user perceptions of care and interest.

Fine-tuning with nutrition data enhances chatbot performance.

User feedback emphasizes the importance of responsiveness and personalization.

Abstract

This study introduces Purrfessor, an innovative AI chatbot designed to provide personalized dietary guidance through interactive, multimodal engagement. Leveraging the Large Language-and-Vision Assistant (LLaVA) model fine-tuned with food and nutrition data and a human-in-the-loop approach, Purrfessor integrates visual meal analysis with contextual advice to enhance user experience and engagement. We conducted two studies to evaluate the chatbot's performance and user experience: (a) simulation assessments and human validation were conducted to examine the performance of the fine-tuned model; (b) a 2 (Profile: Bot vs. Pet) by 3 (Model: GPT-4 vs. LLaVA vs. Fine-tuned LLaVA) experiment revealed that Purrfessor significantly enhanced users' perceptions of care ($\beta = 1.59$, $p = 0.04$) and interest ($\beta = 2.26$, $p = 0.01$) compared to the GPT-4 bot. Additionally, user interviews highlighted the importance of interaction design details, emphasizing the need for responsiveness, personalization, and guidance to improve user engagement.