Reasoning about Actions over Visual and Linguistic Modalities: A Survey

TL;DR

This survey reviews recent progress in reasoning about actions across visual and linguistic modalities, highlighting tasks, datasets, models, challenges, and future directions in the interdisciplinary field.

Contribution

It provides a comprehensive overview of RAC in vision and language, summarizing existing work, benchmarks, and identifying key challenges and future research directions.

Findings

Summarizes current tasks and datasets in RAC

Analyzes performance of various models

Discusses challenges and future directions

Abstract

'Actions' play a vital role in how humans interact with the world and enable them to achieve desired goals. As a result, most common sense (CS) knowledge for humans revolves around actions. While 'Reasoning about Actions & Change' (RAC) has been widely studied in the Knowledge Representation community, it has recently piqued the interest of NLP and computer vision researchers. This paper surveys existing tasks, benchmark datasets, various techniques and models, and their respective performance concerning advancements in RAC in the vision and language domain. Towards the end, we summarize our key takeaways, discuss the present challenges facing this research area, and outline potential directions for future research.