Is Image Memorability Prediction Solved?

TL;DR

This paper presents an algorithm that predicts image memorability at human-level accuracy on a large dataset, discusses the limitations of current benchmarks, and suggests improvements for future evaluation methods.

Contribution

The paper introduces a CNN-based algorithm achieving human-level memorability prediction and critically examines the adequacy of existing benchmarks.

Findings

Achieved human-level performance on LaMem dataset

Identified limitations in current memorability benchmarks

Provided recommendations for future benchmark design

Abstract

This paper deals with the prediction of the memorability of a given image. We start by proposing an algorithm that reaches human-level performance on the LaMem dataset - the only large scale benchmark for memorability prediction. The suggested algorithm is based on three observations we make regarding convolutional neural networks (CNNs) that affect memorability prediction. Having reached human-level performance we were humbled, and asked ourselves whether indeed we have resolved memorability prediction - and answered this question in the negative. We studied a few factors and made some recommendations that should be taken into account when designing the next benchmark.

Tables

Table 1. Table 1 : Memorability prediction improves with classification. The table shows results of two tasks: image memorability prediction and image classification. For classification, we show results on two datasets: SUN  [ 23 ] and Places  [ 24 ] . The accuracy is computed for four architectures, all trained on ImageNet. Consistently, as classification accuracy improves, so does memorability prediction. We adopt the following naming and color-code convention for the setups: Network - Training dataset + Regression type .

Setup	Memorability Prediction			Classification
Vary Network	LaMem	SUN-Mem	Figrim	SUN	Places
ResNet152-ImageNet+XGBoost	0.64	0.64	0.56	-	54.74
VGG16-ImageNet+XGBoost	0.64	0.65	0.56	48.29	55.24
GoogLeNet-ImageNet+XGBoost	0.62	0.63	0.55	43.88	53.63
AlexNet-ImageNet+XGBoost	0.61	0.6	0.51	42.61	53.17

Table 2. Table 2 : Scenes are more important than objects. Memorability was predicted using AlexNet and ResNet152 , trained on objects (ImageNet) , on scenes (Places205 & Places365) , or on their combination (Hybrid1205 & Hybrid1365) . While scenes are more important than objects, their combination slightly improves the prediction. (The training datasets’ details are given in the appendix).

Setup	Memorability Prediction
Vary training data	LaMem	SUN-Mem	Figrim
AlexNet-ImageNet+XGBoost	0.61	0.6	0.51
AlexNet-Places205+XGBoost	0.61	0.64	0.55
AlexNet-Hybrid1205+XGBoost	0.64	0.65	0.57
ResNet152-ImageNet+XGBoost	0.64	0.64	0.56
ResNet152-Places365+XGBoost	0.65	0.66	0.56
ResNet152-Hybrid1365+XGBoost	0.67	0.66	0.57

Table 3. Table 3 : Fine-tuning may be unnecessary. Modifying only the regression layer provides comparable memorability prediction to re-training the entire network. The setup used by  [ 4 ] is utilized.

	Training approach	LaMem	SUN-Mem	Figrim
	Human consistency	0.68	0.75	0.74
[4]	AlexNet-Hybrid1205+SVR	0.61	0.63	-
[MemNet]	AlexNet-Hybrid1205+Fine-tune	0.64	0.53	-
Ours	AlexNet-Hybrid1205+XGBoost	0.64	0.65	0.57

Table 4. Table 4 : Memorability prediction results. The table compares our MemBoost algorithm results with those of  [ 4 ] . It shows that our insights lead to state-of-the-art memorability prediction on all three datasets.

		Approach	LaMem	SUN-Mem	Figrim
		Human consistency	0.68	0.75	0.74
[4]		AlexNet-Hybrid1205+SVR	0.61	0.63	-
	[MemNet]	AlexNet-Hybrid1205+Fine-tune	0.64	0.53	-
Our	[MemBoost]	ResNet152-Hybrid1365+XGBoost	0.67	0.66	0.57

Table 5. Table 5 : Memorability datasets. The memorability scores are the mean and standard deviation over the entire dataset. The consistency values are the average of the Spearman Rank Correlation between different groups of observers.

	Memorability Datasets
Properties	LaMem [4]	SUN-Mem [3]	Figrim [14]
Data type	Objects & scenes	397 Scenes	21 scenes
# Images	58,741	2,222	1,754
Mean memorability	75.6$\pm$12.4	67.5$\pm$13.6	66$\pm$13.9
Human consistency	0.68	0.75	0.74

Table 6. Table 6 : Training datasets of images of objects, of scenes and of both.

Training Datasets	Data type
ImageNet [22]	1000 objects
Places205 [24]	205 scenes
Places365 [25]	365 scenes
Hybrid1205 [24]=ImageNet+Places205	1000 objects + 205 scenes
Hybrid1365 [25]=ImageNet+Places365	1000 objects + 365 scenes

Full text

1 Introduction

Our lives wouldn’t be the same if we were unable to store visual memories. The vast majority of the population relies heavily on visuals to identify people, places and objects. Interestingly, despite different personal experiences, people tend to remember and forget the same pictures [1, 2]. This paper deals with the ability of algorithms to assess the memorability of a given image. Good memorability prediction could be useful for many applications, such as improving education material, storing for us things we tend to forget, producing unforgettable ads, or even presenting images in a way that is easier to consume.

Image memorability is commonly measured as the probability that an observer will detect a repetition of a photograph a few minutes after exposition, when presented amidst a stream of images [1, 2, 3, 4, 5, 6], as illustrated in Figure 1. According to cognitive psychological studies, this measurement determines which images left a trace in our long-term memory [3, 7, 8, 9].

Several methods for memorability prediction have been proposed over the years [3, 10, 11, 12]. A key observation is that both the type of scene and the type of objects in the image are highly related to its memorability [2, 4, 6]. Based on this observation, Khosla et al. [4] collected a large-scale dataset, called LaMem, and proposed MemNet for memorability prediction.

In this paper we describe a system that achieves what seem to be astonishing results—reaching the limit of human performance on LaMem. Does this mean that image memorability prediction is a solved problem? To answer this question, we look deeper into the factors that impact human performance. We discuss some factors that have been overlooked when building the existing datasets for image memorability. Our observations may lead to further studies not only in the design of meaningful datasets and effective algorithms for memorability prediction, but also in finding additional factors influencing memorability.

In the first part of the paper (Section 3), we propose a framework, MemBoost, for predicting image memorability, which achieves state-of-the-art results on all existing datasets. This is done by delving into the relation between networks for image classification and memorability prediction. Our study gives rise to three main insights on which we base the design of MemBoost: (i) As object classification CNNs improve, so does memorability prediction. (ii) Scene classification plays a bigger role in memorability prediction than object classification. This resolves conflicting opinions on the matter. (iii) It suffices to train a regression layer on top of a CNN, which is designed and trained for object & scene recognition, to achieve on par results with those attained by re-training the entire CNN for memorability prediction. This insight contradicts previous observations.

Since our prediction results are surprising, in the second part of the paper (Section 4) we re-visit some aspects that influence human performance in the memory game. Via empirical analysis we show that changing some of the design decisions in the data collection could lead to data that better represents human memorability. The main conclusion from this study is that reaching human performance on LaMem does not mean that memorability prediction has been solved. We further provide guidelines for building future datasets.

In summary, this paper makes three major contributions. First, it presents insights that should be the basis for memorability prediction algorithms. A couple of these insights have already been demonstrated for other tasks in computer vision. Second, the paper suggests a new framework that achieves state-of-the-art results, reaching the limit of human performance on LaMem. Third, we put in question the portrayal of human memorability by the current datasets and give some recommendations towards the creation of the next large dataset.

2 Previous Work

We start by describing how the ground-truth data has been collected. We then review image attributes that have been studied with regard to memorability.

2.0.1 The memory game.

Image memorability is commonly measured using a memory-game approach, which was originally proposed by Isola et al. [3]. Briefly, the participants view a sequence of images, each of which is displayed for a predefined period of time, with some gap in between image presentations, as illustrated in Figure 1. The task of the participants is to press a button whenever they see an identical repeat of an image at any time in the sequence [7, 8]. The participants receive feedback (correct or incorrect) whenever they press a key.

Unbeknown to the participants, the sequence of images is composed of “targets” and “fillers”, both are randomly sampled from the dataset. The role of the fillers is two-fold. First, they provide spacing between the first and the second repetition of a target. Second, responses on repeated fillers constitute a “vigilance task” that allowed us to continuously check that the participants are attentive to the task [11, 12]. Each target is repeated exactly once and each filler is presented at most once (vigilance-task fillers are sequenced to repeat exactly once).

The memorability score assigned to each target image is the percentage of correct detections by the participants. Throughout this paper, we refer to the memorability scores collected through the memory game as ground truth. It is believed that by randomizing the sequence each participant sees, the measurements depend only on factors that are intrinsic to the image, independent of extrinsic variables, such as display order, time delay, and local visual context.

2.0.2 Observations.

Since our study deals with the relation between scenes, objects, and memorability, we next provide a brief review of previous observations regarding which image attributes affect (or do not affect) image memorability.

1. Object category and scene category attributes [2, 3, 6, 10, 12, 13, 14]. A bunch of studies on this topic concluded that some object categories, such as people, vehicles, and animals, and some scene categories, such as indoor scenes, are more memorable than others.

2. Semantic attributes. Scene semantics go beyond just content and scene category [2, 3, 10]. Features such as spatial layout or actions are highly correlated with memorability and are an efficient way of characterizing memorability.

3. Saliency. While Khosla et al. [13] and Celikkale et al. [15] show a reasonable correlation between memorability and attention, Mancas et al. [11] show almost no correlation between the two. Furthermore, Khosla et al. [4] found a reasonable correlation between fixation duration to memorability. When considering object memorability, rather than image memorability, Dubey et al. [6] found high correlation with the number of unique fixation points within the object.

4. Object statistics [2, 3, 10]. The number of objects and the object area have low correlation with memorability and are ineffective at predicting memorability.

5. Aesthetics, interestingness, emotion and popularity. Both image aesthetics and interestingness show no correlation to memorability [2, 4], while they are correlated with each other. Popularity is correlated to memorability only for the most memorable images, but not for others [4]. Negative emotions, such as anger and fear, tend to be more memorable than those portraying positive ones [4].

6. Colors [2, 3, 10, 16, 17]. Colors have only weak correlation with memorability.

7. What people think is memorable [2]. Asking people to guess which images are the most memorable in a collection reveals low correlation to the actually memorable ones. Participants have wrong intuition, erroneously assuming that beautiful and interesting images will produce a lasting memory.

3 MemBoost: A System for Predicting Image Memorability

Motivated by the strong evidence for the correlation between scenes, objects and image memorability, we explored the utility of this correlation for training CNNs. We start our study with short descriptions of three insights we make on memorability prediction with CNNs. Then, we propose the MemBoost algorithm, which is based on these insights, and provides state-of-the-art results.

Our experimental setup follows that of [4] for a variety of datasets and for a variety of networks. Briefly, we randomly split the images in each memorability dataset into a train set and a test set. We performed experiments on three datasets: LaMem [4], SUN-Mem [3], and Figrim [14]. We note that LaMem is a relatively large dataset, consisting of $58,741$ images, whereas the other datasets consist of only $2,222$ and $1,754$ images, respectively; see the appendix for details on these dataset (Table 5). For SUN-Mem and Figrim, splitting was repeated for $25$ times, while for LaMem, we used $5$ splits due to its size.

In order to measure the prediction performance, we follow the common practice in the evaluation of memorability prediction. That is to say, rather than comparing memorability scores directly, we compare ranks as follows. The images in the test set are ranked both according to their ground-truth memorability scores and according to the algorithm predictions. The Spearman’s rank correlation ($\rho$) is computed between the two rankings.

3.1 Insights

In this section we suggest three insights and explore their validity, one by one, via thorough experiments.

3.1.1 1. Use a strong base CNN.

It has recently become common knowledge that the stronger your backbone CNN is, the better results you’d get, even if the base CNN was not trained for your specific task. In accordance with this, we compare in Table 1 four architectures: ResNet152 [18], VGG16 [19], GoogLeNet [20] and AlexNet [21]. All of them were trained on ImageNet [22] and XGBoost was used for the regression layer. The table shows that indeed, as classification networks improve, so do the corresponding memorability prediction networks. These results are consistent across all three datasets. This implies that it might be unnecessary to develop special architectures for memorability prediction. Instead, it suffices to update the relevant layers of the best-performing classification network.

3.1.2 2. Training on scene classification is more important than training on object classification.

Our second insight regards the essence of memorability: What makes an image memorable, the objects in it or the scene it describes? Answering this question is not only interesting theoretically, but it also has very practical implications, since it will enable a better selection of the training dataset for memorability prediction.

Previous works have found that scene category and object presence are, together, highly correlated with the memorability of an image (Spearman rank correlation $\rho=0.43$) [3, 10, 13]. In [2] it is claimed that this correlation is mostly due to the scene category itself, which appears to summarize much of what makes an image memorable. However, this observation has not been used for memorability prediction.

We empirically verify this claim and show, in Table 2, that CNNs that are trained on datasets of scenes (Places205 [24]) outperform CNNs that are trained on datasets of objects (ImageNet [22]). However, the accuracy is not as good as that obtained when training on both objects and scenes (Hybrid1205 [24] or Hybrid1365 [25]). This behavior is persistent across datasets and networks (results are shown for both AlexNet and ResNet152).

We note that these datasets do not provide statistics regarding the balance between scenes and objects. Section 4 discusses the validity of the training datasets. One such consideration is already clear—the right balance between objects and scenes should be sought after.

3.1.3 3. Re-training may be unnecessary.

This insight regards the training of classification networks with memorability data. Is it really necessary to fine-tune the entire network or is it sufficient to train just the last regression layer? Answering this question in the affirmative means that we can achieve good results even when we have neither a lot of memorability data nor much computational resources for training. This is important since such data is not widely available and is difficult to collect, whereas classification data is more widespread.

Khosla et al. [4] compare two approaches for re-training a CNN for memorability prediction. The first approach re-trains only the last regression layer using Support Vector Regression (SVR). In the second approach, called MemNet, the entire network is fine-tuned with memorability data. They achieve better results with MemNet and conclude that fine-tuning the entire CNN is essential.

We reach the opposite conclusion. We show that modifying only the regression layer can provide comparable memorability prediction to re-training the entire network. In particular, we took the same network setup as [4], using AlexNet trained on Hybrid1205. We eliminated the classification layer (top softmax layer) and considered the previous layer as features. We then replaced the classification layer by training a regressor model, which is based on boosted trees (using XGBoost library [26]) and maps the features to memorability scores.

As can be seen in Table 3, fine-tuning only the regression layer with XGBoost is a good idea. In particular, on the large-scale LaMem, training just the regression layer yields the same accuracy as MemNet ($0.64$ in both cases). On the smaller dataset SUN-Mem, training just the regression layer even gives better results than re-training the entire network ($0.65$ in comparison to $0.53$).

3.2 The MemBoost algorithm

Our next step is to utilize the observations from Section 3.1 to design a novel algorithm, called MemBoost. As suggested by insight (1), we select ResNet152 as our base network. We follow insight (2) and use a version named ResNet152-Hybrid1365 that was trained both on an object dataset (ImageNet [27]) and on a scene dataset (Places365 [25]). Last, in sync with insight (3), we modify only the regression layer, using XGBoost [26], to map deep features to memorability scores. Figure 2 illustrates the pipeline of our approach for acquiring state-of-the-art memorability prediction.

Table 4 summarizes our results. The top row of the table presents the memorability consistency across different groups of human observers. This serves as an upper bound. The next two rows show the previous best results, obtained by the two approaches of Khosla et al. [4]. The bottommost row of the table shows the results of our MemBoost, which closes the gap with human prediction results on the LaMem dataset. MemBoost provides a significant improvement over MemNet on both LaMem and SUN-Mem ([4] did not test on Figrim and we failed to reproduce their training).

3.2.1 Result analysis.

Figure 3 sheds light on where our predictions are more accurate than those of MemNet. Every point in the graph in Figure 3(a) corresponds to an image. The Y-axis shows the memorability score of the image. The X-axis shows the difference between MemBoost prediction error and MemNet’s. When MemBoost outperforms MemNet, this difference is positive and vice versa. A symmetric distribution of points around the equal-error line (X=0) would mean both methods have similar distributions of errors. As can be seen, on LaMem the distribution looks more like a Pac-Man, with its mouth at medium memorability scores (0.6-0.8). In Figure 3(b) a different view of the same behavior is given. This stands to show that when the images are not highly memorable or highly forgettable, our algorithm wins. We believe that this is so since such images are more challenging as the scenes/objects are less recognizable. A more powerful prediction algorithm is hence essential in these cases.

To complement our argument, we present in Figures 4 & 5 several example images and their corresponding scores. Figure 4 shows images that have medium memorability scores, where MemBoost outperforms MemNet. The common content of these images is having either memorable objects within forgettable scenes or forgettable objects within memorable scenes.

This observation is reinforced in Figure 5, which shows the images for which we got the largest gap in prediction accuracy between MemBoost and MemNet. Interestingly, these images are all highly memorable, as evident from their memorability scores. They all show common objects, but in unique scenes—text on a cake, hands coming out of a window, extreme repetition of objects, etc. Our algorithm, which combines scenes and objects, manages to predict that these are unique combinations, and hence memorable.

4 On the Validity of Current Memorability Scores

Recall that the upper bound on LaMem is the mean rank correlation between the memorability scores corresponding to different groups of people. As our solution reached this upper bound, we ask ourselves whether the memorability scores, obtained through the Memory Game described in Section 2.0.1, form a sufficient representation. We explore three key questions and answer them via experiments:

Does the number of observers per image suffice as a representative sample? 2. 2.

How consistent are the scores across observers? Should the mean score be utilized by itself, or should the variance be considered as well? 3. 3.

Should the order in which the images are displayed be taken into account?

4.0.1 1. What is a sufficient sample set size?

Recall that the memory game takes as scores the mean memorability over multiple participants. A key question then is ‘how many participants should attend the game for the mean scores to be meaningful?’

Isole et al. [3] show that as the number of participants increases, the mean scores become more stable. To prove stability they show that averaging memorability scores over groups of $40$ participants yields Spearman rank correlation of $\rho=0.75$ between different groups (on the SUN Memorability dataset). Similarly, Khosla et al. [4] measure rank correlation of $\rho=0.68$ on the Lamem dataset, and Bylinskii et al. [14] measure rank correlation of $\rho=0.74$ on the Figrim dataset.

One problem with these results is that they ignore the standard deviation of the correlation when computed over different splits into groups. That is, how stable are the rank correlations between groups? As it turns out, [14] report quite a large variability across splits, i.e., $\sigma\!=\!0.2$ on Figrim [14]. This raises questions regarding the use of memorability scores for prediction, since falling within the variance should be considered as success.

We therefore aim at studying the best group size needed for consistent image memorability scores. To do it, we repeated the Memory Game, as described in Section 2.0.1, using target and filler images randomly selected from Figrim. We evaluate human consistency across different group sizes as follows. We measured memorability for 45 target images, each scored by 275 participants on average. The participants were split into two equal-size groups and the mean score per image was computed for each group. We then computed the Spearman’s rank correlation between the scores of the two groups. This was done for 100 random splits into groups. We then computed both the mean and the variance of the correlation scores over all splits.

Figure 6 shows our results. For groups of $40$ participants, the consistency is $\rho=0.74$ with standard deviation of $\sigma=0.12$ (compared to $0.74$, $0.2$ respectively, reported in [14]). For groups of $100$ participants, the consistency significantly increases to $0.86$ ($\sigma=0.07$), while for groups of $135$ participants, it only slightly increases further to $0.88$ ($\sigma=0.05$).

We conclude that assigning memorability scores should be performed by averaging over groups bigger than $40$ participants. Since collecting memorability measurements for large numbers of participants requires a great deal of work, we recommend using $100$ observers, which seems like a good compromise between the consistency and the complexity of collecting the data.

4.0.2 2. How consistent are the scores? Should the variance be considered?

Having noted that the scores consistency varies, we further ask ourselves whether the mean scores, which are used by the existing datasets, have the same meaning for all images. That is, we question whether representing an image by its mean memorability score suffices, as maybe the variance per image should also be considered.

To answer this, we measured the variance of the memorability scores given by different groups of people, per image. This was done for 45 images and for group sizes between $40$ and $130$. Figure 7 shows our results. Every point in the graph represents a different image. The main conclusion from this graph is that the variance is not fixed. Some images are highly memorable by most people, while others are memorable by some and not so much by others. This suggests that one may want to represent image memorability using two numbers, the mean memorability score and the variance of the scores.

However, a second conclusion from Figure 7, is that the larger the number of participants, the smaller the variance across groups. Therefore, if the number of observers is sufficiently large, it may suffice to maintain a single number—the memorability score—which is the common practice. This supports our previous claim that a larger number of observers per image could lead to more stable scores, with higher consistency.

4.0.3 3. Should the order in which the images are displayed be taken into account?

In real life, images are always displayed in some context; for example, when looking at the newspaper or walking in a museum, the images intentionally appear in a certain order. In the street, the order of images is not intentional, but it definitely influences the scenes and objects we will remember or forget. Image memorability, however, is commonly measured across random sequences of images in order to isolate extrinsic effects such as the order of viewing. Is that the right practice? Should an image be assigned a single memorability score?

To assess the effect of image order on memorability, we designed a new version of the “Memory Game”, in which the only difference from the original memory game is that rather than randomly creating image sequences, we used fixed sets of sequences. For a fixed set of target and filler images (120 altogether, randomly chosen from Figrim), we created 5 orders.

We measured the correlation between different groups of observers. When shown sets of different order, the correlation was low $\rho=0.4$, $\sigma=0.2$, while when shown sets of the the same order, the correlation was high $\rho=0.7$, $\sigma=0.1$. An example of why this happens is displayed in Figure 8. Take for instance, the kitchen scene. When presented after different scenes (top), its memorability score is $0.66$, however, when presented after a sequence of bedrooms (bottom), its memorability score jumped to $0.98$.

This suggests that the order of images in the sequence should not be overlooked. The current practice of taking random orders and averaging might alleviates the influence of this extrinsic effect, but it does not consider it fully. In fact, it averages sequences where the image is memorable with sequences where it is not, making the mean scores an inaccurate representation. We conclude that a single score per image is not a sufficient representation.

5 Conclusion

This paper has studied the relation between convolutional neural networks for image classification and memorability prediction. It introduced MemBoost, a network that reaches the limit of human performance on the largest existing dataset for image memorability, LaMem.

MemBoost is based on three key observations: (i) As object classification CNNs improve, so does image memorability prediction. (ii) Scene classification plays a bigger role in memorability prediction than object classification, but their combination is preferable. (iii) It suffices to train a regression layer on top of a CNN for object & scene recognition to achieve on par results with those attained by re-training the entire CNN. These observations were examined one-by-one via extensive experiments and were thoroughly analyzed.

Since our network already achieves human performance on LaMem, the next stage in memorability prediction is to produce a larger, more challenging dataset. When doing so, various factors should be re-considered. We provide some guidelines for designing the next-generation memorability dataset. These guidelines regard the number of observers, the validity of maintaining one score per image, and the need to re-think the order of images. In the future, more factors may be studied.

Appendix 0.A Dataset details

Tables 5-6 provide the details on the datasets used throughout the paper.