The Promise and Prejudice of Big Data in Intelligence Community

TL;DR

This paper reviews the role of big data in intelligence, highlighting its potential and limitations, and emphasizes the need for domain expertise to effectively utilize big data for national security insights.

Contribution

It provides a comprehensive analysis of big data's limitations in intelligence applications and advocates for expert-guided analysis to improve decision-making.

Findings

Big data faces technical and ethical limitations in intelligence.

Expert domain knowledge is crucial for effective big data analysis.

Case studies show potential complications in nuclear intelligence applications.

Abstract

Big data holds critical importance in the current generation of information technology, with applications ranging from financial, industrial, academic to defense sectors. With the exponential rise of open source data from social media and increasing government monitoring, big data is now also linked with national security, and subsequently to the intelligence community. In this study I review the scope of big data sciences in the functioning of intelligence community. The major part of my study focuses on the inherent limitations of big data, which affects the intelligence agencies from gathering of information to anticipating surprises. The limiting factors range from technical to ethical issues connected with big data. My study concludes the need of experts with domain knowledge from intelligence community to efficiently guide big data analysis for timely filling the knowledge gaps. As a case study on limitations of using big data, I narrate some of the ongoing work in nuclear intelligence using simple analytics and argue on why big data analysis in that case would lead to unnecessary complications. For further investigation, I highlight cases of crowdsource forecasting tournaments and predicting unrest from social media.