To Find Waldo You Need Contextual Cues: Debiasing Who's Waldo
Yiran Luo, Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, Chitta Baral

TL;DR
This paper introduces a debiased dataset for the Person-centric Visual Grounding task, reducing biases present in the original dataset to enable more accurate benchmarking of models.
Contribution
The authors develop automated tools to filter and debias the dataset, creating a more balanced benchmark for PCVG that mitigates heuristic shortcuts.
Findings
Debiased dataset reduces model bias and heuristic performance.
Models trained on the debiased dataset outperform those trained on the original.
Wider gap between heuristic and supervised methods indicates improved evaluation.
Abstract
We present a debiased dataset for the Person-centric Visual Grounding (PCVG) task first proposed by Cui et al. (2021) in the Who's Waldo dataset. Given an image and a caption, PCVG requires pairing up a person's name mentioned in a caption with a bounding box that points to the person in the image. We find that the original Who's Waldo dataset compiled for this task contains a large number of biased samples that are solvable simply by heuristic methods; for instance, in many cases the first name in the sentence corresponds to the largest bounding box, or the sequence of names in the sentence corresponds to an exact left-to-right order in the image. Naturally, models trained on these biased data lead to over-estimation of performance on the benchmark. To enforce models being correct for the correct reasons, we design automated tools to filter and debias the original dataset by ruling out…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMultimodal Machine Learning Applications · Domain Adaptation and Few-Shot Learning · Human Pose and Action Recognition
