MEANT: Multimodal Encoder for Antecedent Information

TL;DR

This paper introduces MEANT, a multimodal encoder designed for temporal stock market data, and a new dataset TempStock, demonstrating improved predictive performance by integrating price, text, and graphical information.

Contribution

The paper presents a novel multimodal encoder for temporal data and introduces TempStock, a large dataset for multimodal stock market analysis.

Findings

MEANT improves baseline performance by over 15%.

Textual information has a greater impact on performance than visual data.

The dataset TempStock contains over a million Tweets across S&P 500 companies.

Abstract

The stock market provides a rich well of information that can be split across modalities, making it an ideal candidate for multimodal evaluation. Multimodal data plays an increasingly important role in the development of machine learning and has shown to positively impact performance. But information can do more than exist across modes -- it can exist across time. How should we attend to temporal data that consists of multiple information types? This work introduces (i) the MEANT model, a Multimodal Encoder for Antecedent information and (ii) a new dataset called TempStock, which consists of price, Tweets, and graphical data with over a million Tweets from all of the companies in the S&P 500 Index. We find that MEANT improves performance on existing baselines by over 15%, and that the textual information affects performance far more than the visual information on our time-dependent task from our ablation study.