Context Understanding in Computer Vision: A Survey

TL;DR

This survey reviews the role of various types of contextual information in enhancing the performance of computer vision tasks, categorizing contexts, models, datasets, and comparing context-based and context-free approaches.

Contribution

It provides a comprehensive categorization of context types, reviews existing models and datasets, and compares different integration methods in computer vision tasks.

Findings

Context improves object and event recognition accuracy.

Different context types offer complementary information.

Future directions include advanced context learning methods.

Abstract

Contextual information plays an important role in many computer vision tasks, such as object detection, video action detection, image classification, etc. Recognizing a single object or action out of context could be sometimes very challenging, and context information may help improve the understanding of a scene or an event greatly. Appearance context information, e.g., colors or shapes of the background of an object can improve the recognition accuracy of the object in the scene. Semantic context (e.g. a keyboard on an empty desk vs. a keyboard next to a desktop computer ) will improve accuracy and exclude unrelated events. Context information that are not in the image itself, such as the time or location of an images captured, can also help to decide whether certain event or action should occur. Other types of context (e.g. 3D structure of a building) will also provide additional information to improve the accuracy. In this survey, different context information that has been used in computer vision tasks is reviewed. We categorize context into different types and different levels. We also review available machine learning models and image/video datasets that can employ context information. Furthermore, we compare context based integration and context-free integration in mainly two classes of tasks: image-based and video-based. Finally, this survey is concluded by a set of promising future directions in context learning and utilization.