Measuring and Managing Answer Quality for Online Data-Intensive Services

TL;DR

Ubora is a novel system that measures the impact of slow components on answer quality in online data services, enabling better online admission control and improving query throughput.

Contribution

It introduces Ubora, a system that efficiently measures answer quality by replaying network messages, applicable across various platforms, to improve online query management.

Findings

Ubora accurately measures answer quality faster than existing methods.

Using answer quality for admission control increases query throughput by 37%.

Ubora is compatible with platforms like Hadoop, Lucene, and question answering systems.

Abstract

Online data-intensive services parallelize query execution across distributed software components. Interactive response time is a priority, so online query executions return answers without waiting for slow running components to finish. However, data from these slow components could lead to better answers. We propose Ubora, an approach to measure the effect of slow running components on the quality of answers. Ubora randomly samples online queries and executes them twice. The first execution elides data from slow components and provides fast online answers; the second execution waits for all components to complete. Ubora uses memoization to speed up mature executions by replaying network messages exchanged between components. Our systems-level implementation works for a wide range of platforms, including Hadoop/Yarn, Apache Lucene, the EasyRec Recommendation Engine, and the OpenEphyra question answering system. Ubora computes answer quality much faster than competing approaches that do not use memoization. With Ubora, we show that answer quality can and should be used to guide online admission control. Our adaptive controller processed 37% more queries than a competing controller guided by the rate of timeouts.