POV Learning: Individual Alignment of Multimodal Models using Human Perception

TL;DR

This paper introduces POV Learning, a method that uses individual perception data to improve the alignment of multimodal models with personal human expectations, enhancing subjective predictive accuracy.

Contribution

It proposes a novel approach to individual alignment by integrating perception signals into machine learning models, demonstrated through a new dataset and a perception-guided transformer.

Findings

Exploiting perception signals improves individual predictive performance.

The perception-guided model outperforms baseline in subjective assessments.

Personalized alignment can steer AI towards individual expectations.

Abstract

Aligning machine learning systems with human expectations is mostly attempted by training with manually vetted human behavioral samples, typically explicit feedback. This is done on a population level since the context that is capturing the subjective Point-Of-View (POV) of a concrete person in a specific situational context is not retained in the data. However, we argue that alignment on an individual level can boost the subjective predictive performance for the individual user interacting with the system considerably. Since perception differs for each person, the same situation is observed differently. Consequently, the basis for decision making and the subsequent reasoning processes and observable reactions differ. We hypothesize that individual perception patterns can be used for improving the alignment on an individual level. We test this, by integrating perception information into machine learning systems and measuring their predictive performance wrt.~individual subjective assessments. For our empirical study, we collect a novel data set of multimodal stimuli and corresponding eye tracking sequences for the novel task of Perception-Guided Crossmodal Entailment and tackle it with our Perception-Guided Multimodal Transformer. Our findings suggest that exploiting individual perception signals for the machine learning of subjective human assessments provides a valuable cue for individual alignment. It does not only improve the overall predictive performance from the point-of-view of the individual user but might also contribute to steering AI systems towards every person's individual expectations and values.