Algorithmic and Statistical Perspectives on Large-Scale Data Analysis

TL;DR

This paper discusses how combining statistical and algorithmic approaches enhances large-scale data analysis, exemplified by feature selection in genetic data and community detection in networks.

Contribution

It presents two recent methods that integrate statistical and algorithmic ideas to improve large-scale data analysis tasks.

Findings

Improved algorithms for feature selection in genetic data

Enhanced community detection techniques in network analysis

Illustration of the synergy between statistical and algorithmic methods

Abstract

In recent years, ideas from statistics and scientific computing have begun to interact in increasingly sophisticated and fruitful ways with ideas from computer science and the theory of algorithms to aid in the development of improved worst-case algorithms that are useful for large-scale scientific and Internet data analysis problems. In this chapter, I will describe two recent examples---one having to do with selecting good columns or features from a (DNA Single Nucleotide Polymorphism) data matrix, and the other having to do with selecting good clusters or communities from a data graph (representing a social or information network)---that drew on ideas from both areas and that may serve as a model for exploiting complementary algorithmic and statistical perspectives in order to solve applied large-scale data analysis problems.