Exploiting Contextual Uncertainty of Visual Data for Efficient Training of Deep Models

TL;DR

This paper explores leveraging contextual uncertainty in visual data to improve deep model training through active learning, data curation, and human-in-the-loop strategies, enhancing robustness and reducing bias.

Contribution

It introduces the concept of contextual diversity for active learning, a data repair algorithm for fair data curation, and class-based annotation methods to improve model training under domain shifts.

Findings

Applicable across semantic segmentation, object detection, and image classification.

Reduces model bias and improves out-of-context object detection.

Enhances annotation efficiency with human-in-the-loop and zero-shot models.

Abstract

Objects, in the real world, rarely occur in isolation and exhibit typical arrangements governed by their independent utility, and their expected interaction with humans and other objects in the context. For example, a chair is expected near a table, and a computer is expected on top. Humans use this spatial context and relative placement as an important cue for visual recognition in case of ambiguities. Similar to human's, DNN's exploit contextual information from data to learn representations. Our research focuses on harnessing the contextual aspects of visual data to optimize data annotation and enhance the training of deep networks. Our contributions can be summarized as follows: (1) We introduce the notion of contextual diversity for active learning CDAL and show its applicability in three different visual tasks semantic segmentation, object detection and image classification, (2) We propose a data repair algorithm to curate contextually fair data to reduce model bias, enabling the model to detect objects out of their obvious context, (3) We propose Class-based annotation, where contextually relevant classes are selected that are complementary for model training under domain shift. Understanding the importance of well-curated data, we also emphasize the necessity of involving humans in the loop to achieve accurate annotations and to develop novel interaction strategies that allow humans to serve as fact-checkers. In line with this we are working on developing image retrieval system for wildlife camera trap images and reliable warning system for poor quality rural roads. For large-scale annotation, we are employing a strategic combination of human expertise and zero-shot models, while also integrating human input at various stages for continuous feedback.