EmoAssist: Emotional Assistant for Visual Impairment Community

TL;DR

This paper introduces EmoAssist, an emotional-aware multi-modal model and benchmark designed to improve assistive AI for visually impaired users by recognizing emotions and providing empathetic, actionable responses.

Contribution

It presents the first emotional intelligence benchmark for assistive LMMs and proposes EmoAssist, a model optimized for emotional understanding in VI assistance.

Findings

Significant improvements in empathy and suggestion metrics (147.8% and 89.7%)

Outperforms GPT-4o in emotional recognition and assistance

First benchmark incorporating emotional evaluation for VI assistive models

Abstract

The rapid advancement of large multi-modality models (LMMs) has significantly propelled the integration of artificial intelligence into practical applications. Visual Question Answering (VQA) systems, which can process multi-modal data including vision, text, and audio, hold great potential for assisting the Visual Impairment (VI) community in navigating complex and dynamic real-world environments. However, existing VI assistive LMMs overlook the emotional needs of VI individuals, and current benchmarks lack emotional evaluation of these LMMs. To address these gaps, this paper introduces the EmoAssist Benchmark, a comprehensive benchmark designed to evaluate the assistive performance of LMMs for the VI community. To the best of our knowledge, this is the first benchmark that incorporates emotional intelligence as a key consideration. Furthermore, we propose the EmoAssist Model, an Emotion-Assistive LMM specifically designed for the VI community. The EmoAssist Model utilizes Direct Preference Optimization (DPO) to align outputs with human emotional preferences. Experiment results demonstrate that the EmoAssist Model significantly enhances the recognition of implicit emotions and intentions of VI users, delivers empathetic responses, and provides actionable guidance. Specifically, it shows respective improvements of 147.8% and 89.7% in the Empathy and Suggestion metrics on the EmoAssist Benchmark, compared to the pre-tuning LMM, and even outperforms state-of-the-art LLMs such as GPT-4o.