From Big Data To Important Information

TL;DR

This paper reviews how to extract meaningful insights from massive data about complex systems, emphasizing the importance of identifying impactful information to address scientific and practical challenges across various domains.

Contribution

It introduces a general framework for assessing the importance of information in complex systems and demonstrates its application in biology and social sciences.

Findings

Framework helps identify impactful information in large datasets

Application to biology informs ecology, biodiversity, pandemics, and lifespan

Application to social systems informs ethnic violence, food prices, and market panic

Abstract

Advances in science are being sought in newly available opportunities to collect massive quantities of data about complex systems. While key advances are being made in detailed mapping of systems, how to relate this data to solving many of the challenges facing humanity is unclear. The questions we often wish to address require identifying the impact of interventions on the system and that impact is not apparent in the detailed data that is available. Here we review key concepts and motivate a general framework for building larger scale views of complex systems and for characterizing the importance of information in physical, biological and social systems. We provide examples of its application to evolutionary biology with relevance to ecology, biodiversity, pandemics, and human lifespan, and in the context of social systems with relevance to ethnic violence, global food prices, and stock market panic. Framing scientific inquiry as an effort to determine what is important and unimportant is a means for advancing our understanding and addressing many practical concerns, such as economic development or treating disease.