Automated Discovery of Real-Time Network Camera Data From Heterogeneous Web Pages

TL;DR

This paper presents an automated system that efficiently discovers and aggregates real-time network camera data from thousands of heterogeneous web pages, significantly reducing manual effort and enabling large-scale data collection.

Contribution

It introduces a novel automated approach for discovering network cameras across diverse web page structures, outperforming manual parsing methods.

Findings

Successfully extracted 57,364 cameras from 237,257 web pages

Identified common web page characteristics for camera discovery

Automated system reduces manual effort significantly

Abstract

Reduction in the cost of Network Cameras along with a rise in connectivity enables entities all around the world to deploy vast arrays of camera networks. Network cameras offer real-time visual data that can be used for studying traffic patterns, emergency response, security, and other applications. Although many sources of Network Camera data are available, collecting the data remains difficult due to variations in programming interface and website structures. Previous solutions rely on manually parsing the target website, taking many hours to complete. We create a general and automated solution for aggregating Network Camera data spread across thousands of uniquely structured web pages. We analyze heterogeneous web page structures and identify common characteristics among 73 sample Network Camera websites (each website has multiple web pages). These characteristics are then used to build an automated camera discovery module that crawls and aggregates Network Camera data. Our system successfully extracts 57,364 Network Cameras from 237,257 unique web pages.