Multimodal Affect Analysis for Product Feedback Assessment

TL;DR

This paper presents a multimodal affect recognition system that analyzes facial expressions, body posture, hand gestures, and voice to assess consumer reactions to products at kiosks, utilizing Kinect sensors and machine learning for real-time feedback.

Contribution

It introduces a novel multimodal affect analysis system combining visual and audio cues with machine learning, specifically support vector machines, for real-time consumer feedback assessment.

Findings

System achieves high accuracy in classifying liking or disliking.

Real-time performance demonstrated with Kinect-based sensors.

Effective multimodal integration enhances feedback assessment.

Abstract

Consumers often react expressively to products such as food samples, perfume, jewelry, sunglasses, and clothing accessories. This research discusses a multimodal affect recognition system developed to classify whether a consumer likes or dislikes a product tested at a counter or kiosk, by analyzing the consumer's facial expression, body posture, hand gestures, and voice after testing the product. A depth-capable camera and microphone system - Kinect for Windows - is utilized. An emotion identification engine has been developed to analyze the images and voice to determine affective state of the customer. The image is segmented using skin color and adaptive threshold. Face, body and hands are detected using the Haar cascade classifier. Canny edges are identified and the lip, body and hand contours are extracted using spatial filtering. Edge count and orientation around the mouth, cheeks, eyes, shoulders, fingers and the location of the edges are used as features. Classification is done by an emotion template mapping algorithm and training a classifier using support vector machines. The real-time performance, accuracy and feasibility for multimodal affect recognition in feedback assessment are evaluated.