Evaluating a Generative Adversarial Framework for Information Retrieval

TL;DR

This paper critically analyzes IRGAN, a GAN-based model for information retrieval, identifying its shortcomings and proposing improved models that outperform IRGAN on multiple tasks.

Contribution

The paper provides a detailed critique of IRGAN, highlights its limitations, and introduces two novel models inspired by self-contrastive estimation and co-training.

Findings

IRGAN's constant baseline term affects performance negatively

The generator component harms IRGAN's effectiveness

Proposed models outperform IRGAN on two of three tasks

Abstract

Recent advances in Generative Adversarial Networks (GANs) have resulted in its widespread applications to multiple domains. A recent model, IRGAN, applies this framework to Information Retrieval (IR) and has gained significant attention over the last few years. In this focused work, we critically analyze multiple components of IRGAN, while providing experimental and theoretical evidence of some of its shortcomings. Specifically, we identify issues with the constant baseline term in the policy gradients optimization and show that the generator harms IRGAN's performance. Motivated by our findings, we propose two models influenced by self-contrastive estimation and co-training which outperform IRGAN on two out of the three tasks considered.