FlashEvaluator: Expanding Search Space with Parallel Evaluation

TL;DR

FlashEvaluator introduces a parallel evaluation method that enables cross-sequence comparison and reduces computational complexity, significantly improving efficiency and accuracy in recommendation and NLP tasks, with proven practical benefits.

Contribution

The paper presents FlashEvaluator, a novel framework that allows cross-sequence token sharing and single-pass processing, enhancing both accuracy and efficiency over traditional independent evaluators.

Findings

Achieves sublinear computational complexity

Improves selection accuracy through cross-sequence comparison

Delivers revenue gains in a real-world recommender system

Abstract

The Generator-Evaluator (G-E) framework, i.e., evaluating K sequences from a generator and selecting the top-ranked one according to evaluator scores, is a foundational paradigm in tasks such as Recommender Systems (RecSys) and Natural Language Processing (NLP). Traditional evaluators process sequences independently, suffering from two major limitations: (1) lack of explicit cross-sequence comparison, leading to suboptimal accuracy; (2) poor parallelization with linear complexity of O(K), resulting in inefficient resource utilization and negative impact on both throughput and latency. To address these challenges, we propose FlashEvaluator, which enables cross-sequence token information sharing and processes all sequences in a single forward pass. This yields sublinear computational complexity that improves the system's efficiency and supports direct inter-sequence comparisons that improve selection accuracy. The paper also provides theoretical proofs and extensive experiments on recommendation and NLP tasks, demonstrating clear advantages over conventional methods. Notably, FlashEvaluator has been deployed in online recommender system of Kuaishou, delivering substantial and sustained revenue gains in practice.