Code and Pixels: Multi-Modal Contrastive Pre-training for Enhanced Tabular Data Analysis

TL;DR

This paper introduces MT-CMTM, a multi-modal contrastive pre-training method that leverages image data to improve tabular data analysis, achieving better accuracy and robustness without relying on images during deployment.

Contribution

The paper presents a novel multi-modal contrastive pre-training approach that enhances tabular models by utilizing image data during training, without requiring images at inference.

Findings

MT-CMTM outperforms baseline models with 1.48% relative MSE improvement on HIPMP.

Achieves 2.38% higher accuracy on DVM dataset.

Demonstrates robustness and potential of multi-modal contrastive learning for tabular data.

Abstract

Learning from tabular data is of paramount importance, as it complements the conventional analysis of image and video data by providing a rich source of structured information that is often critical for comprehensive understanding and decision-making processes. We present Multi-task Contrastive Masked Tabular Modeling (MT-CMTM), a novel method aiming to enhance tabular models by leveraging the correlation between tabular data and corresponding images. MT-CMTM employs a dual strategy combining contrastive learning with masked tabular modeling, optimizing the synergy between these data modalities. Central to our approach is a 1D Convolutional Neural Network with residual connections and an attention mechanism (1D-ResNet-CBAM), designed to efficiently process tabular data without relying on images. This enables MT-CMTM to handle purely tabular data for downstream tasks, eliminating the need for potentially costly image acquisition and processing. We evaluated MT-CMTM on the DVM car dataset, which is uniquely suited for this particular scenario, and the newly developed HIPMP dataset, which connects membrane fabrication parameters with image data. Our MT-CMTM model outperforms the proposed tabular 1D-ResNet-CBAM, which is trained from scratch, achieving a relative 1.48% improvement in relative MSE on HIPMP and a 2.38% increase in absolute accuracy on DVM. These results demonstrate MT-CMTM's robustness and its potential to advance the field of multi-modal learning.