Multi-Facet Counterfactual Learning for Content Quality Evaluation

TL;DR

This paper introduces MOLE, a multi-facet counterfactual learning framework that enhances content quality evaluation by capturing multiple quality aspects using large language models and contrastive learning, aligning better with human judgments.

Contribution

The paper presents a novel multi-facet counterfactual learning approach that improves content quality evaluation by leveraging large language models and contrastive training strategies.

Findings

MOLE significantly improves correlation with human judgments.

It effectively captures multiple quality facets.

Demonstrated on two diverse datasets.

Abstract

Evaluating the quality of documents is essential for filtering valuable content from the current massive amount of information. Conventional approaches typically rely on a single score as a supervision signal for training content quality evaluators, which is inadequate to differentiate documents with quality variations across multiple facets. In this paper, we propose Multi-facet cOunterfactual LEarning (MOLE), a framework for efficiently constructing evaluators that perceive multiple facets of content quality evaluation. Given a specific scenario, we prompt large language models to generate counterfactual content that exhibits variations in critical quality facets compared to the original document. Furthermore, we leverage a joint training strategy based on contrastive learning and supervised learning to enable the evaluator to distinguish between different quality facets, resulting in more accurate predictions of content quality scores. Experimental results on 2 datasets across different scenarios demonstrate that our proposed MOLE framework effectively improves the correlation of document content quality evaluations with human judgments, which serve as a valuable toolkit for effective information acquisition.