# JECL: Joint Embedding and Cluster Learning for Image-Text Pairs

## TL;DR

JECL is a novel clustering method for image-caption pairs that jointly learns representations and cluster assignments using alignment and regularization, outperforming existing methods on benchmark datasets.

## Contribution

JECL introduces a joint embedding and clustering framework for image-text data, combining alignment and regularization to improve clustering performance.

## Key findings

- Outperforms single-view and multi-view methods on benchmark datasets.
- Robust to missing captions and varying data sizes.
- Effectively learns joint representations and cluster assignments.

## Abstract

We propose JECL, a method for clustering image-caption pairs by training parallel encoders with regularized clustering and alignment objectives, simultaneously learning both representations and cluster assignments. These image-caption pairs arise frequently in high-value applications where structured training data is expensive to produce, but free-text descriptions are common. JECL trains by minimizing the Kullback-Leibler divergence between the distribution of the images and text to that of a combined joint target distribution and optimizing the Jensen-Shannon divergence between the soft cluster assignments of the images and text. Regularizers are also applied to JECL to prevent trivial solutions. Experiments show that JECL outperforms both single-view and multi-view methods on large benchmark image-caption datasets, and is remarkably robust to missing captions and varying data sizes.