Fusion Matters: Learning Fusion in Deep Click-through Rate Prediction Models

TL;DR

This paper introduces OptFusion, an automated method for learning optimal feature fusion strategies in CTR prediction models, significantly improving performance by jointly optimizing connection and operation choices.

Contribution

We propose a one-shot learning algorithm that automates fusion design in CTR models, addressing limitations of previous fixed or search-based approaches.

Findings

OptFusion outperforms baseline models on three large-scale datasets.

The method improves CTR prediction accuracy significantly.

It demonstrates high efficiency in learning effective fusion strategies.

Abstract

The evolution of previous Click-Through Rate (CTR) models has mainly been driven by proposing complex components, whether shallow or deep, that are adept at modeling feature interactions. However, there has been less focus on improving fusion design. Instead, two naive solutions, stacked and parallel fusion, are commonly used. Both solutions rely on pre-determined fusion connections and fixed fusion operations. It has been repetitively observed that changes in fusion design may result in different performances, highlighting the critical role that fusion plays in CTR models. While there have been attempts to refine these basic fusion strategies, these efforts have often been constrained to specific settings or dependent on specific components. Neural architecture search has also been introduced to partially deal with fusion design, but it comes with limitations. The complexity of the search space can lead to inefficient and ineffective results. To bridge this gap, we introduce OptFusion, a method that automates the learning of fusion, encompassing both the connection learning and the operation selection. We have proposed a one-shot learning algorithm tackling these tasks concurrently. Our experiments are conducted over three large-scale datasets. Extensive experiments prove both the effectiveness and efficiency of OptFusion in improving CTR model performance. Our code implementation is available here\url{https://github.com/kexin-kxzhang/OptFusion}.