Efficient Learning for Product Attributes with Compact Multimodal Models

TL;DR

This paper presents a label-efficient semi-supervised fine-tuning approach for compact vision-language models in e-commerce product attribute prediction, leveraging unlabeled data with minimal compute overhead.

Contribution

It introduces a novel semi-supervised fine-tuning method using PEFT and DPO that effectively utilizes unlabeled data for compact VLMs in e-commerce applications.

Findings

DPO-based fine-tuning outperforms supervised models on multiple verticals.

Accuracy improves as more unlabeled data is incorporated.

Method achieves efficient convergence with minimal compute.

Abstract

Image-based product attribute prediction in e-commerce is a crucial task with numerous applications. The supervised fine-tuning of Vision Language Models (VLMs) faces significant scale challenges due to the cost of manual or API based annotation. In this paper, we investigate label-efficient semi-supervised fine-tuning strategies for compact VLMs (2B-3B parameters) that leverage unlabeled product listings through Direct Preference Optimization (DPO). Beginning with a small, API-based, annotated, and labeled set, we first employ PEFT to train low-rank adapter modules. To update the adapter weights with unlabeled data, we generate multiple reasoning-and-answer chains per unlabeled sample and segregate these chains into preferred and dispreferred based on self-consistency. We then fine-tune the model with DPO loss and use the updated model for the next iteration. By using PEFT fine-tuning with DPO, our method achieves efficient convergence with minimal compute overhead. On a dataset spanning twelve e-commerce verticals, DPO-based fine-tuning, which utilizes only unlabeled data, demonstrates a significant improvement over the supervised model. Moreover, experiments demonstrate that accuracy with DPO training improves with more unlabeled data, indicating that a large pool of unlabeled samples can be effectively leveraged to improve performance.