AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising

TL;DR

AdTEC is a comprehensive benchmark for evaluating the quality of automatically generated ad texts across multiple dimensions, aiding the development of better natural language generation models for advertising.

Contribution

This paper introduces the first public benchmark for ad text evaluation, including a Japanese dataset, and analyzes the performance of PLMs and humans on these tasks.

Findings

PLMs perform well in some tasks but lag behind humans in others

The benchmark reveals specific challenges in automatic ad text evaluation

Humans still outperform models in certain quality assessment domains

Abstract

With the increase in the fluency of ad texts automatically created by natural language generation technology, there is high demand to verify the quality of these creatives in a real-world setting. We propose AdTEC (Ad Text Evaluation Benchmark by CyberAgent), the first public benchmark to evaluate ad texts from multiple perspectives within practical advertising operations. Our contributions are as follows: (i) Defining five tasks for evaluating the quality of ad texts, as well as building a Japanese dataset based on the practical operational experiences of building a Japanese dataset based on the practical operational experiences of advertising agencies, which are typically kept in-house. (ii) Validating the performance of existing pre-trained language models (PLMs) and human evaluators on the dataset. (iii) Analyzing the characteristics and providing challenges of the benchmark. The results show that while PLMs have already reached practical usage level in several tasks, humans still outperform in certain domains, implying that there is significant room for improvement in this area.