SeLiNet: Sentiment enriched Lightweight Network for Emotion Recognition in Images

TL;DR

This paper introduces SeLiNet, a lightweight neural network designed for on-device emotion recognition in images, which effectively combines body and aesthetic features to estimate emotions and sentiments with significantly reduced model size.

Contribution

SeLiNet is a novel, compact network that jointly estimates emotions and sentiments from images, optimized for on-device deployment with minimal size and competitive accuracy.

Findings

Achieves an AP score of 27.17 on EMOTIC dataset with >85% model size reduction.

On-device implementation reaches an AP score of 26.42 with >93% size reduction.

Maintains competitive performance while significantly reducing computational requirements.

Abstract

In this paper, we propose a sentiment-enriched lightweight network SeLiNet and an end-to-end on-device pipeline for contextual emotion recognition in images. SeLiNet model consists of body feature extractor, image aesthetics feature extractor, and learning-based fusion network which jointly estimates discrete emotion and human sentiments tasks. On the EMOTIC dataset, the proposed approach achieves an Average Precision (AP) score of 27.17 in comparison to the baseline AP score of 27.38 while reducing the model size by >85%. In addition, we report an on-device AP score of 26.42 with reduction in model size by >93% when compared to the baseline.