Diversifying Multi-aspect Search Results Using Simpson's Diversity Index

TL;DR

This paper introduces a novel diversification method for multi-aspect search results using Simpson's Diversity Index, improving diversity and computational efficiency over previous approaches.

Contribution

The paper adapts Simpson's Diversity Index for search result diversification, extending it to multiple aspects with weighted balancing and fast approximation algorithms.

Findings

Outperforms previous diversification methods in accuracy

Reduces computational complexity significantly

Effective on Kaggle shoes dataset

Abstract

In search and recommendation, diversifying the multi-aspect search results could help with reducing redundancy, and promoting results that might not be shown otherwise. Many previous methods have been proposed for this task. However, previous methods do not explicitly consider the uniformity of the number of the items' classes, or evenness, which could degrade the search and recommendation quality. To address this problem, we introduce a novel method by adapting the Simpson's Diversity Index from biology, which enables a more effective and efficient quadratic search result diversification algorithm. We also extend the method to balance the diversity between multiple aspects through weighted factors and further improve computational complexity by developing a fast approximation algorithm. We demonstrate the feasibility of the proposed method using the openly available Kaggle shoes competition dataset. Our experimental results show that our approach outperforms previous state of the art diversification methods, while reducing computational complexity.