Pseudo Contrastive Learning for Diagram Comprehension in Multimodal Models

TL;DR

This paper introduces a novel training method using pseudo contrastive samples to improve diagram comprehension in vision-language models, significantly enhancing their ability to recognize fine-grained structural differences in diagrams.

Contribution

The paper presents a new training paradigm that leverages synthetic pseudo contrastive samples to boost diagram understanding in multimodal models, without altering original data.

Findings

Improved performance on flowchart dataset in image-text matching.

Enhanced accuracy in visual question answering tasks.

Demonstrated superiority over standard CLIP training methods.

Abstract

Recent multimodal models such as Contrastive Language-Image Pre-training (CLIP) have shown remarkable ability to align visual and linguistic representations. However, domains where small visual differences carry large semantic significance, such as diagram understanding, remain challenging due to the models' limited sensitivity to fine-grained structural variations. We propose a new training paradigm designed to enhance diagram comprehension in vision-language models. Our approach introduces pseudo contrastive samples generated by a diagram renderer that creates synthetic diagrams using randomly picked text elements. These samples highlight structural differences in diagrammatic imagery without requiring any modification or editing of the original data. By incorporating these pseudo contrastive samples into the training objective, the model learns to capture more precise and semantically consistent diagram structures. Empirical evaluations on a benchmark dataset of flowcharts demonstrate substantial improvements over standard CLIP and hard-negative CLIP training in both image-text matching and visual question answering tasks. The results underscore the value of domain-specific training strategies and contribute to advancing diagrammatic understanding within the broader context of vision-language learning.