Algorithms and Architecture for Real-time Recommendations at News UK

TL;DR

This paper presents a new incremental collaborative filtering algorithm and scalable architecture enabling real-time news recommendations, demonstrated on clickstream data and deployed in production at News UK.

Contribution

It introduces a novel incremental update algorithm for collaborative filtering and a scalable system architecture for real-time news recommendations.

Findings

Effective incremental collaborative filtering algorithm

Scalable architecture for real-time recommendations

System deployed in production at News UK

Abstract

Recommendation systems are recognised as being hugely important in industry, and the area is now well understood. At News UK, there is a requirement to be able to quickly generate recommendations for users on news items as they are published. However, little has been published about systems that can generate recommendations in response to changes in recommendable items and user behaviour in a very short space of time. In this paper we describe a new algorithm for updating collaborative filtering models incrementally, and demonstrate its effectiveness on clickstream data from The Times. We also describe the architecture that allows recommendations to be generated on the fly, and how we have made each component scalable. The system is currently being used in production at News UK.