Product-based Neural Networks for User Response Prediction over Multi-field Categorical Data

TL;DR

This paper introduces Product-based Neural Networks (PNN and PIN) for user response prediction on multi-field categorical data, addressing gradient issues and improving prediction accuracy through novel feature interaction models.

Contribution

The paper proposes kernel product and product-network in network architectures to enhance feature interaction modeling in deep neural networks for response prediction.

Findings

PNN and PIN outperform 8 baselines on multiple datasets.

PIN achieves 34.67% CTR improvement in online A/B testing.

Models demonstrate consistent improvements in AUC and log loss.

Abstract

User response prediction is a crucial component for personalized information retrieval and filtering scenarios, such as recommender system and web search. The data in user response prediction is mostly in a multi-field categorical format and transformed into sparse representations via one-hot encoding. Due to the sparsity problems in representation and optimization, most research focuses on feature engineering and shallow modeling. Recently, deep neural networks have attracted research attention on such a problem for their high capacity and end-to-end training scheme. In this paper, we study user response prediction in the scenario of click prediction. We first analyze a coupled gradient issue in latent vector-based models and propose kernel product to learn field-aware feature interactions. Then we discuss an insensitive gradient issue in DNN-based models and propose Product-based Neural Network (PNN) which adopts a feature extractor to explore feature interactions. Generalizing the kernel product to a net-in-net architecture, we further propose Product-network In Network (PIN) which can generalize previous models. Extensive experiments on 4 industrial datasets and 1 contest dataset demonstrate that our models consistently outperform 8 baselines on both AUC and log loss. Besides, PIN makes great CTR improvement (relatively 34.67%) in online A/B test.