Large Scale Organization and Inference of an Imagery Dataset for Public Safety

TL;DR

This paper presents a hierarchical organization method for a large-scale public safety imagery dataset, enabling efficient processing and inference across terabytes of data using supercomputing resources.

Contribution

It introduces a novel hierarchical organization approach for large public safety imagery datasets and demonstrates its effectiveness with large-scale inference on supercomputing infrastructure.

Findings

Efficient dataset organization reduces compute and storage costs.

Successful large-scale inference across terabytes of imagery.

Hierarchical approach improves processing scalability.

Abstract

Video applications and analytics are routinely projected as a stressing and significant service of the Nationwide Public Safety Broadband Network. As part of a NIST PSCR funded effort, the New Jersey Office of Homeland Security and Preparedness and MIT Lincoln Laboratory have been developing a computer vision dataset of operational and representative public safety scenarios. The scale and scope of this dataset necessitates a hierarchical organization approach for efficient compute and storage. We overview architectural considerations using the Lincoln Laboratory Supercomputing Cluster as a test architecture. We then describe how we intelligently organized the dataset across LLSC and evaluated it with large scale imagery inference across terabytes of data.