Technical Report of the Video Event Reconstruction and Analysis (VERA) System -- Shooter Localization, Models, Interface, and Beyond

TL;DR

The VERA system integrates video synchronization, geolocation, and sound recognition to reconstruct events from videos, demonstrated by accurately localizing a shooter in the 2017 Las Vegas incident, aiding human rights and safety efforts.

Contribution

This paper introduces the VERA system, combining multiple AI capabilities into a web-based tool for event reconstruction from unstructured videos, with open-source code for community development.

Findings

Accurately localized the Las Vegas shooter using VERA with minimal initial data.

Demonstrated effective video synchronization and geolocation without metadata.

Enabled human-in-the-loop verification for improved accuracy.

Abstract

Every minute, hundreds of hours of video are uploaded to social media sites and the Internet from around the world. This material creates a visual record of the experiences of a significant percentage of humanity and can help illuminate how we live in the present moment. When properly analyzed, this video can also help analysts to reconstruct events of interest, including war crimes, human rights violations, and terrorist acts. Machine learning and computer vision can play a crucial role in this process. In this technical report, we describe the Video Event Reconstruction and Analysis (VERA) system. This new tool brings together a variety of capabilities we have developed over the past few years (including video synchronization and geolocation to order unstructured videos lacking metadata over time and space, and sound recognition algorithms) to enable the reconstruction and analysis of events captured on video. Among other uses, VERA enables the localization of a shooter from just a few videos that include the sound of gunshots. To demonstrate the efficacy of this suite of tools, we present the results of estimating the shooter's location of the Las Vegas Shooting in 2017 and show that VERA accurately predicts the shooter's location using only the first few gunshots. We then point out future directions that can help improve the system and further reduce unnecessary human labor in the process. All of the components of VERA run through a web interface that enables human-in-the-loop verification to ensure accurate estimations. All relevant source code, including the web interface and machine learning models, is freely available on Github. We hope that researchers and software developers will be inspired to improve and expand this system moving forward to better meet the needs of human rights and public safety.