Ethosight: A Reasoning-Guided Iterative Learning System for Nuanced Perception based on Joint-Embedding & Contextual Label Affinity

TL;DR

Ethosight is a zero-shot video analytics system that uses joint embedding and reasoning with ontologies to achieve adaptable, low-cost, and continuous learning in real-world scenarios, reducing reliance on extensive data annotation.

Contribution

This work introduces Ethosight, a novel reasoning-guided iterative learning system that operates effectively on edge devices and supports continuous adaptation without catastrophic forgetting.

Findings

Effective in diverse complex use cases

Operates on low-cost edge devices

Supports continuous learning and adaptation

Abstract

Traditional computer vision models often necessitate extensive data acquisition, annotation, and validation. These models frequently struggle in real-world applications, resulting in high false positive and negative rates, and exhibit poor adaptability to new scenarios, often requiring costly retraining. To address these issues, we present Ethosight, a flexible and adaptable zero-shot video analytics system. Ethosight begins from a clean slate based on user-defined video analytics, specified through natural language or keywords, and leverages joint embedding models and reasoning mechanisms informed by ontologies such as WordNet and ConceptNet. Ethosight operates effectively on low-cost edge devices and supports enhanced runtime adaptation, thereby offering a new approach to continuous learning without catastrophic forgetting. We provide empirical validation of Ethosight's promising effectiveness across diverse and complex use cases, while highlighting areas for further improvement. A significant contribution of this work is the release of all source code and datasets to enable full reproducibility and to foster further innovation in both the research and commercial domains.