Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks

TL;DR

This paper introduces ReMem, a new benchmark for assessing foundational memorization in LVLMs, addressing issues of under-memorization and multi-hop reasoning to improve unlearning evaluation reliability.

Contribution

ReMem provides a comprehensive, principled benchmark with data scaling, reasoning-aware QA, and an exposure metric to better diagnose learning and unlearning in LVLMs.

Findings

ReMem ensures robust foundational learning in LVLMs.

The exposure metric quantifies information erasure effectiveness.

Experiments validate ReMem's reliability for diagnosing memorization issues.

Abstract

While Large Vision-Language Models (LVLMs) offer powerful capabilities, they pose privacy risks by unintentionally memorizing sensitive personal information. Current unlearning benchmarks attempt to mitigate this using fictitious identities but overlook a critical stage 1 failure: models fail to effectively memorize target information initially, rendering subsequent unlearning evaluations unreliable. Diagnosing under-memorization and the multi-hop curse as root causes, we introduce ReMem, a Reliable Multi-hop and Multi-image Memorization Benchmark. ReMem ensures robust foundational learning through principled data scaling, reasoning-aware QA pairs, and diverse visual contexts. Additionally, we propose a novel Exposure metric to quantify the depth of information erasure from the model's internal probability distribution. Extensive experiments demonstrate that ReMem provides a rigorous and trustworthy framework for diagnosing both learning and unlearning behaviors in LVLMs.