Scalable Online Survey Framework: from Sampling to Analysis

TL;DR

This paper presents a scalable online survey framework that addresses sampling and analysis challenges in large-scale digital environments, with applications at LinkedIn for product insights and user experience monitoring.

Contribution

It introduces a comprehensive approach for scalable survey sampling and analysis, integrating in-product surveys with monitoring and A/B testing at LinkedIn.

Findings

Effective sampling strategies for multiple email surveys.

In-product surveys can serve as real-time monitoring tools.

Integration of surveys with A/B testing enhances decision-making.

Abstract

With the advancement in technology, raw event data generated by the digital world have grown tremendously. However, such data tend to be insufficient and noisy when it comes to measuring user intention or satisfaction. One effective way to measure user experience directly is through surveys. In particular, with the popularity of online surveys, extensive work has been put in to study this field. Surveys at LinkedIn play a major role in influencing product and marketing decisions and supporting our sales efforts. We run an increasing number of surveys that help us understand shifts in awareness and perceptions with regards to our own products and also to peer companies. As the need to survey grows, both sampling and analysis of surveys have become more challenging. Instead of simply multiplying the number of surveys each user takes, we need a scalable approach to collect enough and representative samples for each survey analysis while maintaining good user experience. In this paper, we start with discussions on how we handle multiple email surveys under such constraints. We then shift our discussions to challenges of in-product surveys and how we address them at LinkedIn through a survey study conducted across two mobile apps. Finally, we share how in-product surveys can be utilized as monitoring tools and connect surveys with A/B testing.