Movie Recommendation with Poster Attention via Multi-modal Transformer Feature Fusion

TL;DR

This paper introduces a multi-modal transformer-based movie recommendation system that fuses text and poster features using pre-trained models, achieving improved accuracy on standard benchmarks.

Contribution

It presents a novel multi-modal feature fusion approach combining BERT, ViT, and transformer architecture for enhanced movie recommendation accuracy.

Findings

Improved prediction accuracy on MovieLens datasets.

Effective multi-modal feature integration using pre-trained models.

Demonstrated potential for cross-modal recommendation applications.

Abstract

Pre-trained models learn general representations from large datsets which can be fine-turned for specific tasks to significantly reduce training time. Pre-trained models like generative pretrained transformers (GPT), bidirectional encoder representations from transformers (BERT), vision transfomers (ViT) have become a cornerstone of current research in machine learning. This study proposes a multi-modal movie recommendation system by extract features of the well designed posters for each movie and the narrative text description of the movie. This system uses the BERT model to extract the information of text modality, the ViT model applied to extract the information of poster/image modality, and the Transformer architecture for feature fusion of all modalities to predict users' preference. The integration of pre-trained foundational models with some smaller data sets in downstream applications capture multi-modal content features in a more comprehensive manner, thereby providing more accurate recommendations. The efficiency of the proof-of-concept model is verified by the standard benchmark problem the MovieLens 100K and 1M datasets. The prediction accuracy of user ratings is enhanced in comparison to the baseline algorithm, thereby demonstrating the potential of this cross-modal algorithm to be applied for movie or video recommendation.