Teach CLIP to Develop a Number Sense for Ordinal Regression

TL;DR

This paper introduces NumCLIP, a method to enhance CLIP's ability for ordinal regression by discretizing numerical concepts and applying a novel ranking loss, improving performance on various tasks.

Contribution

The paper proposes NumCLIP, a simple yet effective approach that improves CLIP's understanding of number concepts for ordinal regression through discretization and a new ranking loss.

Findings

10% accuracy improvement on historical image dating

3.83% accuracy improvement on image aesthetics assessment

Effective generalization to multiple ordinal regression tasks

Abstract

Ordinal regression is a fundamental problem within the field of computer vision, with customised well-trained models on specific tasks. While pre-trained vision-language models (VLMs) have exhibited impressive performance on various vision tasks, their potential for ordinal regression has received less exploration. In this study, we first investigate CLIP's potential for ordinal regression, from which we expect the model could generalise to different ordinal regression tasks and scenarios. Unfortunately, vanilla CLIP fails on this task, since current VLMs have a well-documented limitation of encapsulating compositional concepts such as number sense. We propose a simple yet effective method called NumCLIP to improve the quantitative understanding of VLMs. We disassemble the exact image to number-specific text matching problem into coarse classification and fine prediction stages. We discretize and phrase each numerical bin with common language concept to better leverage the available pre-trained alignment in CLIP. To consider the inherent continuous property of ordinal regression, we propose a novel fine-grained cross-modal ranking-based regularisation loss specifically designed to keep both semantic and ordinal alignment in CLIP's feature space. Experimental results on three general ordinal regression tasks demonstrate the effectiveness of NumCLIP, with 10% and 3.83% accuracy improvement on historical image dating and image aesthetics assessment task, respectively. Code is publicly available at https://github.com/xmed-lab/NumCLIP.