Minor probability events detection in big data: An integrated approach with Bayesian testing and MIM

TL;DR

This paper proposes an integrated Bayesian testing approach combined with message importance measure to improve detection of rare events in big data, reducing miss rates for events with very small prior probabilities.

Contribution

It introduces a novel modified detection method that enhances Bayesian detection by incorporating message importance measure to better identify minor probability events.

Findings

Reduced miss testing rate for minor probability events

Effective detection of rare events in large datasets

Improved accuracy over traditional Bayesian detection

Abstract

The minor probability events detection is a crucial problem in Big data. Such events tend to include rarely occurring phenomenons which should be detected and monitored carefully. Given the prior probabilities of separate events and the conditional distributions of observations on the events, the Bayesian detection can be applied to estimate events behind the observations. It has been proved that Bayesian detection has the smallest overall testing error in average sense. However, when detecting an event with very small prior probability, the conditional Bayesian detection would result in high miss testing rate. To overcome such a problem, a modified detection approach is proposed based on Bayesian detection and message importance measure, which can reduce miss testing rate in conditions of detecting events with minor probability. The result can help to dig minor probability events in big data.