What Images are More Memorable to Machines?
Junlin Han, Huangying Zhan, Jie Hong, Pengfei Fang, Hongdong Li, Lars, Petersson, Ian Reid
TL;DR
This paper introduces a method to measure and analyze the memorability of images to machines, revealing that complex images tend to be more memorable for various machine models and pre-training methods.
Contribution
It proposes the MachineMem measurer, a self-supervised pipeline for quantifying machine memorability, and provides extensive analysis across different models and training methods.
Findings
Complex images are more memorable to machines.
Machine memorability differs from human memorability.
Analysis across 11 models and 9 training methods.
Abstract
This paper studies the problem of measuring and predicting how memorable an image is to pattern recognition machines, as a path to explore machine intelligence. Firstly, we propose a self-supervised machine memory quantification pipeline, dubbed ``MachineMem measurer'', to collect machine memorability scores of images. Similar to humans, machines also tend to memorize certain kinds of images, whereas the types of images that machines and humans memorize are different. Through in-depth analysis and comprehensive visualizations, we gradually unveil that``complex" images are usually more memorable to machines. We further conduct extensive experiments across 11 different machines (from linear classifiers to modern ViTs) and 9 pre-training methods to analyze and understand machine memory. This work proposes the concept of machine memorability and opens a new research direction at the…
Peer Reviews
Decision·ICLR 2024 Conference Withdrawn Submission
1. The problem is interesting and should be inspire some other researches. 2. Some findings in this paper shoule be inspiring.
1. The motivation of this paper should be more clear. After reading this paper, it is difficult to see the significance of the memorability, though it sounds interesting. For example, if we can exactly measure memorability, what can we use it to do in the fields of computer vision and machine learning? 2. I do not think the approach to measure memorability is reasonable. In section 3, the authors argue that they use ResNet-50 as the basic model, but there are two problems. (1) if ResNet-50 is p
The strengths of the paper are as follows: 1. The concept of machine memorability is novel and different from human memorability score. The concept can be potentially useful in understanding the network, though its not fully established in the paper how. The paper has described the pipeline for setting up the measurement well, using pre-training through rotation, repeat game (similar to human memorability), with ample references. 2. The paper has provided detailed analysis of correlations of the
The weaknesses are as follows: 1. The paper does not establish a downstream usecase for the scores. A simple experiment would be to analyze the correlation of the machine memorability scores with task output of the network. Does the network makes more mistakes on lower memorability score images? This could establish the usability of the scores better, 2. Since the pipeline is novel, the paper does not present any comparisons with a competing method. If the best usecase for the method is augmenti
This paper is thoughtfully constructed and easy to follow. Despite missing some important related work (see below), I would still describe the work as "scholarly." It seems well-grounded in a line of research on human memory. The authors do an excellent job comparing the results to those on humans. I enjoyed seeing the results from GANalyze and found those images quite instructive. Existing work on "what machines remember" contains many takeaways, some of which parallel those found here. Howev
This paper does not reference the existing literature on memorization in machine learning [important works include 0, 1, 2]. Recent work explores these questions in the context of SSL [3]. Particularly relevant is [4], which find that "which images are memorized" depends on the training data as a whole: if highly memorized examples are removed from the data, then other examples may be subject to higher memorization. In addition to work directly on memorization, there is relevant work on "members
The idea of machine memorability might be interesting.
(1) The paper does not provide a clear definition of machine memorability. Is it solely associated with the model’s structure (the number of parameters) or anything else, like how the model is trained? The current method for measuring the memorability seems to involve not only the structure but also how it is trained. If this is the case, the paper should provide some experiments that show how different pretraining strategies affect the memorability. Because of the missing definition, it’s hard
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsCell Image Analysis Techniques · Anomaly Detection Techniques and Applications · Domain Adaptation and Few-Shot Learning