Data Science at Udemy: Agile Experimentation with Algorithms

TL;DR

This paper details Udemy's data science framework enabling full-stack data scientists to independently manage the entire lifecycle of algorithm development and experimentation, leading to improved recommendation and search systems.

Contribution

It introduces a comprehensive framework empowering data scientists to independently handle exploration, development, and analysis, facilitating rapid deployment of algorithms.

Findings

Algorithms tested and deployed in 2015

Insights from experiments led to new recommender system

Framework supports research in search, personalization, and topic generation

Abstract

In this paper, we describe the data science framework at Udemy, which currently supports the recommender and search system. We explain the motivations behind the framework and review the approach, which allows multiple individual data scientists to all become 'full stack', taking control of their own destinies from the exploration and research phase, through algorithm development, experiment setup, and deep experiment analytics. We describe algorithms tested and deployed in 2015, as well as some key insights obtained from experiments leading to the launch of the new recommender system at Udemy. Finally, we outline the current areas of research, which include search, personalization, and algorithmic topic generation.