Enhanced Movie Content Similarity Based on Textual, Auditory and Visual Information

TL;DR

This paper demonstrates that combining low-level textual, auditory, and visual features significantly improves movie similarity estimation for content-based recommendations, outperforming metadata-only methods.

Contribution

It introduces a multimodal feature extraction approach for movies, integrating textual, visual, and audio cues with metadata to enhance similarity measures.

Findings

Multimodal features boost recommendation accuracy by over 50%.

All three modalities contribute significantly to similarity estimation.

First comprehensive approach using combined low-level features across modalities.

Abstract

In this paper we examine the ability of low-level multimodal features to extract movie similarity, in the context of a content-based movie recommendation approach. In particular, we demonstrate the extraction of multimodal representation models of movies, based on textual information from subtitles, as well as cues from the audio and visual channels. With regards to the textual domain, we emphasize our research in topic modeling of movies based on their subtitles, in order to extract topics that discriminate between movies. Regarding the visual domain, we focus on the extraction of semantically useful features that model camera movements, colors and faces, while for the audio domain we adopt simple classification aggregates based on pretrained models. The three domains are combined with static metadata (e.g. directors, actors) to prove that the content-based movie similarity procedure can be enhanced with low-level multimodal information. In order to demonstrate the proposed content representation approach, we have built a small dataset of 160 widely known movies. We assert movie similarities, as propagated by the individual modalities and fusion models, in the form of recommendation rankings. Extensive experimentation proves that all three low-level modalities (text, audio and visual) boost the performance of a content-based recommendation system, compared to the typical metadata-based content representation, by more than $50\%$ relative increase. To our knowledge, this is the first approach that utilizes a wide range of features from all involved modalities, in order to enhance the performance of the content similarity estimation, compared to the metadata-based approaches.