Do Trajectories Encode Verb Meaning?
Dylan Ebert, Chen Sun, Ellie Pavlick

TL;DR
This paper explores whether object trajectories over time can inherently encode verb meanings, using a new dataset and comparing different representation learning methods, revealing promising correlations and improvements with pretraining.
Contribution
It introduces a novel dataset of agent-object interactions and demonstrates that trajectories can encode verb semantics, enhanced by self-supervised pretraining.
Findings
Trajectories correlate with some verbs like 'fall'
Self-supervised pretraining captures nuanced verb differences
Trajectory-based representations improve with abstraction techniques
Abstract
Distributional models learn representations of words from text, but are criticized for their lack of grounding, or the linking of text to the non-linguistic world. Grounded language models have had success in learning to connect concrete categories like nouns and adjectives to the world via images and videos, but can struggle to isolate the meaning of the verbs themselves from the context in which they typically occur. In this paper, we investigate the extent to which trajectories (i.e. the position and rotation of objects over time) naturally encode verb semantics. We build a procedurally generated agent-object-interaction dataset, obtain human annotations for the verbs that occur in this data, and compare several methods for representation learning given the trajectories. We find that trajectories correlate as-is with some verbs (e.g., fall), and that additional abstraction via…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Natural Language Processing Techniques · Topic Modeling
