InstaIndoor and Multi-modal Deep Learning for Indoor Scene Recognition

TL;DR

This paper introduces a multi-modal deep learning approach for indoor scene recognition using social media videos, combining speech and visual data to improve classification accuracy.

Contribution

It presents a novel fusion model leveraging social media video data and introduces the InstaIndoor dataset for indoor scene recognition.

Findings

Achieves 70% accuracy and 0.7 F1-Score on InstaIndoor dataset.

Achieves 74% accuracy and 0.74 F1-Score on YouTube-8M subset.

Demonstrates the effectiveness of multi-modal fusion for indoor scene classification.

Abstract

Indoor scene recognition is a growing field with great potential for behaviour understanding, robot localization, and elderly monitoring, among others. In this study, we approach the task of scene recognition from a novel standpoint, using multi-modal learning and video data gathered from social media. The accessibility and variety of social media videos can provide realistic data for modern scene recognition techniques and applications. We propose a model based on fusion of transcribed speech to text and visual features, which is used for classification on a novel dataset of social media videos of indoor scenes named InstaIndoor. Our model achieves up to 70% accuracy and 0.7 F1-Score. Furthermore, we highlight the potential of our approach by benchmarking on a YouTube-8M subset of indoor scenes as well, where it achieves 74% accuracy and 0.74 F1-Score. We hope the contributions of this work pave the way to novel research in the challenging field of indoor scene recognition.