Enhancing Multi-Modal Video Sentiment Classification Through Semi-Supervised Clustering

TL;DR

This paper proposes a semi-supervised clustering approach to improve multi-modal video sentiment classification by leveraging unlabeled data to learn meaningful representations before supervised fine-tuning.

Contribution

It introduces a novel clustering-based semi-supervised pre-training method that enhances sentiment classification accuracy in multi-modal videos with limited labeled data.

Findings

Improved sentiment classification accuracy with limited labeled data

Effective extraction of meaningful representations from multi-modal data

Enhanced understanding of underlying data structures

Abstract

Understanding emotions in videos is a challenging task. However, videos contain several modalities which make them a rich source of data for machine learning and deep learning tasks. In this work, we aim to improve video sentiment classification by focusing on two key aspects: the video itself, the accompanying text, and the acoustic features. To address the limitations of relying on large labeled datasets, we are developing a method that utilizes clustering-based semi-supervised pre-training to extract meaningful representations from the data. This pre-training step identifies patterns in the video and text data, allowing the model to learn underlying structures and relationships without requiring extensive labeled information at the outset. Once these patterns are established, we fine-tune the system in a supervised manner to classify the sentiment expressed in videos. We believe that this multi-modal approach, combining clustering with supervised fine-tuning, will lead to more accurate and insightful sentiment classification, especially in cases where labeled data is limited.