Automatic Creative Selection with Cross-Modal Matching

TL;DR

This paper introduces a novel method for matching app images to search terms using a fine-tuned LXMERT model, significantly improving accuracy over existing models like CLIP and Transformer-ResNet baselines.

Contribution

The work presents a new approach that fine-tunes a pre-trained LXMERT model for cross-modal matching of images and text, outperforming previous models in app advertising relevance tasks.

Findings

Achieves 0.96 AUC on advertiser data, outperforming baselines by 8-14%.

Achieves 0.95 AUC on human ratings, outperforming baselines by 16-17%.

Significantly improves image-text matching accuracy in app advertising.

Abstract

Application developers advertise their Apps by creating product pages with App images, and bidding on search terms. It is then crucial for App images to be highly relevant with the search terms. Solutions to this problem require an image-text matching model to predict the quality of the match between the chosen image and the search terms. In this work, we present a novel approach to matching an App image to search terms based on fine-tuning a pre-trained LXMERT model. We show that compared to the CLIP model and a baseline using a Transformer model for search terms, and a ResNet model for images, we significantly improve the matching accuracy. We evaluate our approach using two sets of labels: advertiser associated (image, search term) pairs for a given application, and human ratings for the relevance between (image, search term) pairs. Our approach achieves 0.96 AUC score for advertiser associated ground truth, outperforming the transformer+ResNet baseline and the fine-tuned CLIP model by 8% and 14%. For human labeled ground truth, our approach achieves 0.95 AUC score, outperforming the transformer+ResNet baseline and the fine-tuned CLIP model by 16% and 17%.