Enhancing the Interactivity of Dataframe Queries by Leveraging Think Time

TL;DR

This paper introduces opportunistic evaluation, a framework that reduces interactive latency in dataframe queries by prioritizing relevant computations and utilizing think time for background processing, enhancing user experience during data analysis.

Contribution

It presents a novel opportunistic evaluation framework that leverages think time and prioritizes computations to accelerate dataframe interactions, improving responsiveness in exploratory data analysis.

Findings

Opportunistic evaluation significantly reduces interactive latency.

User behavior analysis reveals ample opportunities for optimization.

Proposed methods effectively harness think time for background computation.

Abstract

We propose opportunistic evaluation, a framework for accelerating interactions with dataframes. Interactive latency is critical for iterative, human-in-the-loop dataframe workloads for supporting exploratory data analysis. Opportunistic evaluation significantly reduces interactive latency by 1) prioritizing computation directly relevant to the interactions and 2) leveraging think time for asynchronous background computation for non-critical operators that might be relevant to future interactions. We show, through empirical analysis, that current user behavior presents ample opportunities for optimization, and the solutions we propose effectively harness such opportunities.