Global Semantic Description of Objects based on Prototype Theory

TL;DR

This paper presents a new semantic description method for objects based on Prototype Theory, using a Computational Prototype Model to encode category meaning and improve object classification and clustering.

Contribution

It introduces a novel CPM and description model that leverage CNN-derived prototypes for interpretable, discriminative object descriptions, outperforming existing global encodings.

Findings

CPM effectively models internal semantic structure of categories.

Semantic distance correlates with object typicality.

Descriptor encoding outperforms other global encodings in clustering/classification.

Abstract

In this paper, we introduce a novel semantic description approach inspired on Prototype Theory foundations. We propose a Computational Prototype Model (CPM) that encodes and stores the central semantic meaning of objects category: the semantic prototype. Also, we introduce a Prototype-based Description Model that encodes the semantic meaning of an object while describing its features using our CPM model. Our description method uses semantic prototypes computed by CNN-classifications models to create discriminative signatures that describe an object highlighting its most distinctive features within the category. Our experiments show that: i) our CPM model (semantic prototype + distance metric) is able to describe the internal semantic structure of objects categories; ii) our semantic distance metric can be understood as the object visual typicality score within a category; iii) our descriptor encoding is semantically interpretable and significantly outperforms other image global encodings in clustering and classification tasks.

Tables

Table 1. Table 1: Two-dimensional conceptual map of prototypicality effects  (Geeraerts, 2010 ) .

	extensional	intensional
non-equality	Difference of typicality	Clustering into family
(salience effect,	and membership salience	resemblances
core/periphery)
non-discreteness	Fuzziness at the edges,	Absence of necessary and
(demarcation	membership uncertainty	sufficient definitions
problems, flexibly)

Table 2. Table 2: Available GSDP descriptor signature dimensions for each CNN classification model used as features extractor.

CNN Model	$\left|F\right|$	$F_{p\times q}$	${\chi }_{r\times r}$	$g^{jk}$	$\left|f\left(x\right)\right|$	$\left|\psi \right|$
CNN-MNIST	128	$16\times 8$	$8\times 8$	$2\times 1$	16	32
			$4\times 4$	$4\times 2$	64	128
CNN-CIFAR	512	$32\times 16$	$8\times 8$	$4\times 2$	64	128
			$4\times 4$	$8\times 4$	256	512
VGG16	4096	$64\times 64$	$16\times 16$	$4\times 4$	128	256
			$8\times 8$	$8\times 8$	512	1024
ResNet50	2048	$64\times 32$	$16\times 16$	$4\times 2$	64	128
			$8\times 8$	$8\times 4$	256	512

Table 3. Table 3: K-Means cluster metrics for each evaluated global image representation. Screenshot of K-Means measures for first 20 20 20 ImageNet categories ( 20 20 20 clusters). We show Homogeneity (H), Completeness (C), V-measure (V), Adjusted Rand Index (ARI) and Adjusted Mutual Information (AMI) clustering measures. We show in bold the best performance.

Descriptor	Size	FPS	Metrics Scores
			H	C	V	ARI	AMI
GIST	960	0.82	0.05	0.05	0.05	0.01	0.05
LBP	512	0.72	0.02	0.03	0.03	0.01	0.02
HOG	1960	33	0.04	0.04	0.04	0.01	0.03
Color64	64	8	0.12	0.12	0.12	0.04	0.11
Color_Hist	512	26	0.08	0.08	0.08	0.03	0.07
Hu_H_CH	532	6.9	0.04	0.04	0.04	0.01	0.02
VGG16	4096	15	0,87	0,88	0,88	0,78	0,87
VGG_PCA_256	256	12.5	0,89	0,90	0,89	0,82	0,89
GSDP_VGG_256	256	12.8	0,97	0,99	0,98	0,93	0,97
VGG_PCA_1024	1024	12.5	0,89	0,89	0,89	0,81	0,89
GSDP_VGG_1024	1024	11.6	0,94	0,98	0,96	0,84	0,94
ResNet50	2048	10.6	0,88	0,90	0,89	0,78	0,88
RNet_PCA_128	128	12.5	0,88	0,88	0,88	0,81	0,88
GSDP_RNet_128	128	9.6	0,97	0,98	0,98	0,93	0,97
RNet_PCA_512	512	12.5	0,89	0,90	0,90	0,82	0,89
GSDP_RNet_512	512	9	0,91	0,97	0,94	0,73	0,91

Equations

$$\delta (o_1,o_2)=\sum _{j=1}^m\left|{\omega }_{ij}\right|\left|f_j^1-f_j^2\right|,$$

$$\delta (\text{o},P_i)=\sum _{j=1}^m\left|{\omega }_{ij}\right|\left|f_j-{\mu }_{ij}\right|,$$

$$\mathrm{Pr}(|f_j-{\mu }_{ij}|\ge {\lambda }_{ij}{\sigma }_{ij})\le \min (1,\frac{1}{{\lambda }_{ij}^2}),$$

Full text

††journal:

1 Introduction

Memory is one of the most amazing faculties of the human being. It is generally considered as the brain ability to code, store, and retrieve information (Atkinson and Shiffrin, 1968). Semantic memory (Tulving, 2007), for instance, refers to general world knowledge that we accumulate throughout our lives (McRae and Jones, 2013). A relevant aspect of the functional neuroanatomy of the semantic memory resides in the representation of the meaning of objects and their properties (Martin, 2007). Several assumptions indicate that human beings are capable of: i) learning the most distinctive features of a specific object category (Martin, 2007; Thompson-Schill, 2003); * ii)* form categories and object semantic definitions (abstractions) at a very early age (Martin, 2007). Semantic memory involves the semantic definition of objects (Tulving, 2007) and, consequently, the success of object recognition, classification, and description tasks are causally related to the success of effectively recovering the learned knowledge (Tulving, 2007).

For several years, the fields of Computer Vision and Machine Learning have tried to build and learning pattern recognition methods with a similar performance of a human being for visual information processing. Although the state-of-the-art methods have achieved surprising results, there are still many challenges to achieve the discriminative power and abstraction of semantic memory to represent the semantic. How to describe and stand for objects, semantically? How to simulate the behavior of semantic memory in the representation of learned knowledge of objects’ features? How to extract and encode the object features to encapsulate the representation of the meaning (or semantic representation) of a specific object? How to learn the semantic definition of categories objects and use this definition in object recognition, classification, and description tasks? These are just some of the interesting questions that still occupy the investigation agenda of many research areas.

In this paper – motivated by the semantic memory behavior – we propose a mathematical model that attempts to represent the semantic definition of object categories. Also, we propose a procedure to introduce this semantic representation of object categories in the global description of objects features extracted from images.

The knowledge extraction models (high-level vision processes) from images are highly influenced by the methods used for detection, extraction, and representation of image relevant information. Consequently, the extraction of image relevant features has been the subject of Computer Vision research for decades. For several years, hand-crafted features (Bay et al., 2008; Lowe, 2004; Tola et al., 2008) and machine learning methods (Simonyan et al., 2014; Strecha et al., 2012) were the choice for image feature description tasks.

The advent of Convolutional Neural Networks (CNN) outperformed these traditional methods and enabled them to achieve a visual recognition model with similar behavior of semantic memory for classification tasks (He et al., 2016; Simonyan and Zisserman, 2014; Szegedy et al., 2017), sparking the tendency of images semantic processing with deep-learning techniques. The CNN-models success spawned numerous CNN-descriptors produced with different approaches that learn effective representations for describing image features (Han et al., 2015; Kim et al., 2018; Simo-Serra et al., 2015; Zagoruyko and Komodakis, 2015). Consequently, representations of image features extracted using deep classification models (He et al., 2016; Simonyan and Zisserman, 2014; Szegedy et al., 2017), or using CNN-descriptors are commonly referred as semantic feature or semantic signature.

Semantic feature term has been extensively studied in the field of linguistic semantic; it is defined as the representation of the basic conceptual components of the meaning of any lexical item (Fromkin et al., 2018). In the seminal work of Rosch (1975), the author analyzed the semantic structure of the meaning of words and introduced the concept of semantic prototype. According to Rosch (1975); Rosch and Mervis (1975), the representation of category semantic meaning is related to the category prototype, particularly to those categories naming natural objects.

Image semantic understanding is influenced by how are semantically represented the features of image basic components (e.g., objects), and the semantic relations between these basic components (Guo et al., 2016). CNN-description models (Han et al., 2015; Lin et al., 2016; Simo-Serra et al., 2015; Zagoruyko and Komodakis, 2015) and semantic description models (Han et al., 2017; Kim et al., 2018; Rocco et al., 2018) stand for the semantic information of image features using a range of different approaches. Nevertheless, none of these models codify the representation of the visual information based on the theoretical foundation of Cognitive Science to represent the semantic meaning.

In this paper, we rely on cognitive semantic studies related to the Prototype Theory for modeling the central semantic meaning of objects categories: the prototype. We propose a novel approach to take on the semantic features descriptions of objects based on prototypes. Our prototype-based description model uses the category’s prototype to find a global semantic representation of the basics conceptual components (objects) of the image meaning.

To achieve this goal, we bring to light the Prototype Theory as a theoretical foundation to represent the semantic meaning of the visual information accurately to represent — semantically — the basics components of the image: objects. The Prototype Theory proposes that human beings think a category in terms of abstract prototypes, defined by typical members of the category (Geeraerts, 2010; Rosch, 1975, 1978). This theory also exposes that successful execution of object recognition and description tasks in the human brain is inherently related to the learned prototype of the object category (Minda and Smith, 2002; Rosch, 1975, 1978; Zaki et al., 2003). The observations on the Prototype Theory raise the following two questions: i) Can a model of the perception system be developed in which objects are described using the same semantic features that are learned to identify and classify them? ii) How can the category prototype be included in the object global semantic description?

We address these two questions motivated by the human’s approach to describing objects globally. Human being uses the generalization and discrimination processes to build object descriptions that highlighting their most distinctive features within the category. For example, a typical human description: a dalmatian is a dog (generalization ability to recognize the central semantic meaning of dog category) that is distinguished by its unique black, or liver-colored spotted coat (discrimination ability to detect the semantic distinctiveness of object within the dog category). Figure 1 illustrates the intuition and principal concepts of our prototype-based description model. The main idea of our approach is to use the quality of features extracted with CNN-classification models both to represent the central semantic meaning of a specific category and learn the object distinctiveness within the category.

More specifically, our main contributions in this paper are as follows:

a Computational Prototype Model (CPM) based on Prototype Theory foundations, to stand for the central semantic meaning of object images categories. Our CPM model allows to interpret possible semantic associations between members within the category internal structure. 2. 2.

a semantic distance metric in object image CNN features domain, which can be understood as a measure of object typicality within the object category. 3. 3.

a prototype-based description model for global semantic description of objects images. Our semantic description model introduces, for the first time, the use of category prototypes in image global description tasks.

2 Related works

CNN descriptors

Descriptors extracted using CNN techniques have shown that it is possible, for a learning approach, to outperform the best techniques based on carefully hand-crafted features (Bay et al., 2008; Lowe, 2004; Tola et al., 2008). CNN descriptor models differ among themselves on how to compute the descriptors in their deep architectures, similarity functions learning, and its features extraction methods. Some approaches extract immediate activations of the model as a descriptor signature (Simonyan and Zisserman, 2014; Szegedy et al., 2017; Donahue et al., 2014; Long et al., 2014). Others methods use similarity convolutional networks (Han et al., 2015; Simo-Serra et al., 2015; Yi et al., 2016; Zagoruyko and Komodakis, 2015) and Siamese networks (Han et al., 2015; Zagoruyko and Komodakis, 2015; Yi et al., 2016) to learn discriminative representations. LIFT (Yi et al., 2016) learns each task involved in features management: detection, orientation estimation, and description. Lin et al. (2016) constructed a compact binary descriptor for efficient object matching based on features extracted with VGG16 model (Simonyan and Zisserman, 2014). Those CNN-descriptor models were more oriented to achieve discriminative features than representing the image semantic.

Semantic descriptors and semantic correspondence

Liu et al. (2011) proposed SIFT Flow method. SIFT Flow method generated the start of semantic flow family methods as a solution to the challenge of semantic correspondence (Bristow et al., 2015; Liu et al., 2011; Yang et al., 2014). Several of these methods combine their approaches with the extraction of hand-crafted features (Lowe, 2004; Tola et al., 2008). Some works (Han et al., 2017; Kim et al., 2018; Rocco et al., 2018) use the robustness of CNN-models for training deep learning architectures and address the problem of semantic correspondence. Kim et al. (2018) tackled the problem of semantic correspondence by constructing FCSS semantic descriptor. In general, CNN descriptors and semantic descriptors are trained to learn their semantic representations and use different deep learning architectures. Most of these features description models do not use the discriminative power of the features extracted using the well-known CNN-classification models (He et al., 2016; Simonyan and Zisserman, 2014; Szegedy et al., 2017). Moreover, none of these CNN-feature description approaches incorporates the foundation of the Cognitive Sciences to introduce meaning in the representations of image features.

Prototype Theory

The Prototype Theory (Rosch and Lloyd, 1978; Rosch and Mervis, 1975; Rosch, 1988; Geeraerts, 2010; Minda and Smith, 2002; Rosch, 1975, 1978; Zaki et al., 2003) analyzes the internal structure of categories and introduces the prototype-based concept of categorization. It proposes categories representation as heterogeneous and not discrete, where the features and category members do not have the same relevance within the category. Rosch (Rosch, 1975, 1978) obtained evidence that human beings store first the semantic meaning of category based on the degrees of representativeness (typicity) of category members, and then its specificities.

The category prototype was formally defined as the clear central members of a category (Geeraerts, 2010; Rosch, 1975; Rosch and Mervis, 1975). The attributes of these focal members are those that are structurally the most salient category properties, and conversely, a member occupies the focal position because it shows the most salient features of the category (Rosch and Mervis, 1975; Geeraerts, 2010). Rosch (Rosch, 1975, 1978, 1988) showed that human beings store the category knowledge as a semantic organization around the category prototype (prototypicality organization). Figure 2 shows an example of the prototypicality organization phenomenon (Rosch, 1978; Rosch and Lloyd, 1978; Geeraerts, 2010). Finally, object categorization is obtained based on the similarity of a new exemplar with the learned categories prototypes (Rosch, 1978, 1988).

Rosch (Rosch, 1975, 1978; Rosch and Lloyd, 1978; Rosch, 1988) showed the important of making distinctions between various phenomena that may be associated with prototypicality. For Geeraerts (2010) the concept of prototypicality is in itself a prototypically clustered one for four characteristics in which the concepts of non-discreteness and non-equality (either on the intensional or on the extensional level) play a major distinctive role. Four characteristics are frequently mentioned as typical of prototypicality in prototypical categories (Rosch, 1975, 1978; Geeraerts, 2010): i) categories exhibit degrees of typicality; not every member is equally representative in the category (extensional non-equality); ii) categories are blurred at the edges (extensional non-discreteness); iii) categories are clustering into family resemblance structure; that is, the category semantic structure takes the form of a radial set of clustered and overlapping members (intensional non-equality); and iv) categories cannot be defined by means of a single set of criteria (necessary and sufficient) attributes (intensional non-discreteness). The prototypicality effects (see Table 1) surmise the importance of the distinction between central and peripheral meaning of the object categories (Geeraerts, 2010).

3 Computational Prototype Model

Rosch (1975, 1978) showed that human beings learn the central semantic meaning of categories (the prototype) and include it in their cognitive processes. Based on these assumptions, our object semantic description approach follows the flow of conceptual processes presented in Figure 1 as a hypothesis for simulating the human behavior in object features description. Since our proposal requires as priori knowledge the prototypes representation of objects categories, we need a procedure to represent the prototype of a specific category.

3.1 Semantic Representation

Category semantic structure (i.e., central and peripheral meaning) is related with differences of typicality and membership salience of category members (extensional non-equality). The prototype can be understood as the “average” of the abstractions of all objects in the category (Sternberg and Sternberg, 2016); it summarizes the most representative members (or features) of the category. The combination between observed object features and features relevance for the category enables the grouping of objects into family resemblance (intensional non-equality). This approach justifies the object’s position within the semantic structure of the category and allows typical objects to be grouped into the semantic center of the category (prototypical organization).

Let $O$ be an universe of objects; $C=\left\{{c_{1},c_{2},...,c_{n}}\right\}$ be the finite set of objects categories labels that partition $O$; $O_{c_{i}}=\left\{{o\in O:{\small category(o)}=c_{i}}\right\}$ is the set of objects that share the same i-th category $c_{i}\in C$, $\forall i=1,...,n;$ and $F=\left\{{f_{1},f_{2},...,f_{m}}\right\}$ be a finite set of distinguishing features of an object.

Definition 1

Semantic prototype. We call the central meaning of the category $c_{i}\in C$, semantic prototype of $c_{i}$-category, or simply semantic prototype, to the “average” and standard deviation of each features of all typical objects within the $c_{i}$-category, along with a measure of the relevance of those features. Formally, our semantic prototype is a $3$-tuple $P_{i}=\left({M_{i},\Sigma_{i},\Omega_{i}}\right)$ where $\>\forall i=1,...,n;\forall j=1,...,m$:

1. i)

$M_{i}=\left[{\mu_{i1},\mu_{i2},...,\mu_{im}}\right]$ is a nonempty $m$-dimensional vector, where $\mu_{ij}$ is the mean of j-$th$ feature of features extracted for only typical objects of $c_{i}$-category; 2. ii)

$\Sigma_{i}=\left[{\sigma_{i1},\sigma_{i2},...,\sigma_{im}}\right]$ is a nonempty $m$-dimensional vector, where $\sigma_{ij}$ is the standard deviation of j-$th$ feature of features extracted for only typical objects of $c_{i}$-category; 3. iii)

$\Omega_{i}=\left[{\omega_{i1},\omega_{i2},...,\omega_{im}}\right]$ is a nonempty $m$-dimensional vector, where $\omega_{ij}$ is the relevance value of j-$th$ feature for the category $c_{i}\in C$.

Definition 2

Abstract prototype. The abstract semantic center of $i$-th category $c_{i}\in C$, most prototypical element of $i$-th category, ideal element of $i$-th category, or simply the abstract prototype of $i$-th category, is the $m$-dimensional vector $M_{i}\in P_{i}=\left({M_{i},\Sigma_{i},\Omega_{i}}\right)$ composed of the expected value of each features extracted for only typical objects of $c_{i}$-category.

3.2 Semantic Distance

Our description approach (see processes 4 - 5 in Figure 1) needs a distance measure to compute the discrepancy between object features and category-typical features (semantic prototype). The distance metrics $L1$ and $L2$ could be good options if it did not assume that all object features have the same relevance.

According to the Prototype Theory: i) each object feature has a relative relevance in the category and ii) the relevance (or salience) of each category member is in accordance with the number and type of features present in the object. This approach can establish a degree of prototypicality of a specific element within the category (extensional non-equality).

Some formal models of Experimental Psychology such as prototype model (Reed, 1972; Homa and Vosburgh, 1976), Multiplicative Prototype Model (MPM) (Minda and Smith, 2001, 2002) and Generalized Context Model (GCM) (Medin and Schaffer, 1978; Estes, 1986; Nosofsky, 1986; Zaki et al., 2003) proposed measures of semantic distances between stimulus that correspond to Prototype Theory foundations. Consequently, we generalized some of these semantics measures to propose a semantic distance metric (or dissimilarity function) that measures the discrepancy between two objects images (or between an object image and its semantic prototype) based on observed features.

Definition 3

Distance between objects. Let ${o_{1},o_{2}\in O_{c_{i}}}$ be a representative objects of $i$-th category $c_{i}\in C;$ $\emph{F}_{o_{1}},\emph{F}_{o_{2}}$ the features of objects $o_{1},o_{2}$ respectively. We defined the objects distance between $o_{1}$ and $o_{2}$ as the semantic distance given by:

[TABLE]

where $\omega_{ij}\in\Omega_{i},$ $f_{j}^{1}\in F_{o_{1}}\,$ and $f_{j}^{2}\in F_{o_{2}}\,,\>\forall i=1...n;\,\forall j=1...m$.

It is worth noting that our semantic distance between objects is a generalization of the psychological distance between two stimuli proposed in GCM formal model. Unlike the original formal Context Model (Medin and Schaffer, 1978), we assume that: i) object features (stimuli) are not binary values ($f_{j}\in\mathbb{R}$); ii) relevance ($\omega_{ij}$) (or cost of attention) of each $j$-th unitary object feature is forced to be strictly positive, but has no upper limit $(\sum_{j=1}^{m}\omega_{ij}\neq 1)$. We removed these constraints of GCM Model in order to model object features and object features relevance using the features and weights learned by classification models.

Definition 4

Prototypical distance. Let ${\emph{o}\in O_{c_{i}}}$ a representative object of $i$-th category $c_{i}\in C$, $\emph{F}_{o}$ the features of object o and $P_{i}=\left({M_{i},\Sigma_{i},\Omega_{i}}\right)$ the semantic prototype of $c_{i}$-category. We defined as prototypical distance between o and $P_{i}$ the semantic distance:

[TABLE]

where $\omega_{ij}\in\Omega_{i},\,\mu_{ij}\in M_{i},$ and $f_{j}\in F_{o}\,;$ $M_{i},\Omega_{i}\in P_{i}$ $\>\forall i=1...n;\,\forall j=1...m$.

Our prototypical distance is a generalization of semantic distance of MPM formal model (Minda and Smith, 2001, 2002). Different from MPM model assumptions, we assumed that prototype features are not features of a real member of $i$-th category, but features of expected ideal member (our abstract prototype) of $i$-th category ($M_{i}\in P_{i}$).

Definition 5

Features metric space. Let ${F}_{c_{i}}$ be a non empty set of all objects features of category $c_{i}\in C$. Since the distance function $\delta:{F}_{c_{i}}\times{F}_{c_{i}}\to\mathbb{R}^{+}$ satisfies the axioms of non-negativity, identity of indiscernible, symmetry and triangle inequality; $\delta$ is a metric in the features domain ${F}_{c_{i}}$. Consequently,  $({F}_{c_{i}},\delta)$ is a metric space or features metric space.

Proof

Let ${o_{1},\>o_{2},\>o_{3}\in O_{c_{i}}}$ objects members of i-th category ($c_{i}\in C$); $F_{1},\>F_{2},\>F_{3}$ the corresponding object features with $f_{j}^{1}\in{F}_{1},\>f_{j}^{2}\in{F}_{2},\>f_{j}^{3}\in{F}_{3}$; $\forall i=1,...,n;\forall j=1,...,m$.

• $\bullet$

$\delta(o_{1},o_{2})\geq 0$ (non-negativity).

Since all terms in Equation 1 are non negative ($\geq 0$), $\delta(o_{1},o_{2})\geq 0$ by definition;

• $\bullet$

$\delta(o_{1},o_{2})=0\Leftrightarrow o_{1}=o_{2}$  (identity of indiscernible).

• –

$\delta(o_{1},o_{2})=0\rightarrow o_{1}=o_{2}$.

If $\delta(o_{1},o_{2})=0$ then $\sum_{j=1}^{m}\left|\omega_{ij}\right|\left|f_{j}^{1}-f_{j}^{2}\right|=0$; consequently, since all terms in Equation 1 are non negative, the above expression is true if each element in the sum is zero. Then, $\forall\left|\omega_{ij}\right|$ $\neq 0,$ $\left|f_{j}^{1}-f_{j}^{2}\right|=0\rightarrow f_{j}^{1}=f_{j}^{2}\rightarrow o_{1}=o_{2}$;

• –

$o_{1}=o_{2}\rightarrow\delta(o_{1},o_{2})=0$.

If $o_{1}=o_{2}\rightarrow f_{j}^{1}=f_{j}^{2}\rightarrow\left|f_{j}^{1}-f_{j}^{2}\right|=0,\forall j=1...m;$ then $\delta(o_{1},o_{2})=0$;

• $\bullet$

$\delta(o_{1},o_{2})=\delta(o_{2},o_{1})$  (symmetry).

$\delta(o_{1},o_{2})$=$\sum_{j=1}^{m}\left|\omega_{ij}\right|\left|f_{j}^{1}-f_{j}^{2}\right|$ $=\sum_{j=1}^{m}\left|\omega_{ij}\right|$ $\left|f_{j}^{2}-f_{j}^{1}\right|$ $=\delta(o_{2},o_{1})$;

• $\bullet$

$\delta(o_{1},o_{3})\leq\delta(o_{1},o_{2})+\delta(o_{2},o_{3})$ (triangle inequality).

$\delta(o_{1},o_{2})+\delta(o_{2},o_{3})$ $=\sum_{j=1}^{m}\left|\omega_{ij}\right|\left|f_{j}^{1}-f_{j}^{2}\right|+$ $\sum_{j=1}^{m}$ $\left|\omega_{ij}\right|$ $\left|f_{j}^{2}-f_{j}^{3}\right|=\sum_{j=1}^{m}\left|\omega_{ij}\right|(\left|f_{j}^{1}-f_{j}^{2}\right|+\left|f_{j}^{2}-f_{j}^{3}\right|)$ and by absolute value property $\left|f_{j}^{1}-f_{j}^{2}\right|+\left|f_{j}^{2}-f_{j}^{3}\right|$ $\geq\left|f_{j}^{1}-f_{j}^{3}\right|$, then $\delta(o_{1},o_{2})+\delta(o_{2},o_{3})\geq\delta(o_{1},o_{3})$.

Also, note that if $E\subseteq{O}_{c_{i}}$ is a subset of $i$-th category, $\delta(E)=\sum\delta(\emph{o},P_{i}),\,\forall\emph{o}\in E$. Consequently, our prototypical distance satisfies the following properties: i) null empty set: ${\displaystyle\delta(\varnothing)=0}$;  ii) countable additivity: for all countable collections ${\displaystyle\{E_{k}\}_{k=1}^{\infty}}$ of pairwise disjoint sets in $E$, ${\displaystyle\delta\left(\bigcup_{k=1}^{\infty}E_{k}\right)=\sum_{k=1}^{\infty}\delta(E_{k})}$ (this property is easy to prove using mathematical induction).

Corollary 1

The prototypical distance function from ${F}_{c_{i}}$ to the extended real number line, $\delta:{F}_{c_{i}}\to\mathbb{R}^{+}$, is a measure. Consequently, $({F}_{c_{i}},\delta)$ is a measurable space.

Since our statement of $({F}_{c_{i}},\delta)$ is a measurable space, we can use the generalization of Chebyshev’s inequality (Chebyshev, 1867) to define the boundary of our semantic prototype representation. Chebyshev (1867) asserted that the probability that a scalar random variable $\xi$ with distribution $\Pr$ differs from its mean $\mu\in\mathbb{R}$ by more than $\lambda\in\mathbb{R}>0$ standard deviations $\sigma\in\mathbb{R}>0$ satisfies the relation:  $\Pr(|\xi-\mu|\geq\lambda\sigma)\leq\min(1,\frac{1}{\lambda^{2}})$.

Saw et al. (1984) and Stellato et al. (2017) approached the problem of formulating an empirical Chebyshev inequality given $N$ i.i.d samples from an unknown distribution $\Pr$, and their empirical mean $\mu_{N}$ and empirical standard deviation $\sigma_{N}$. Saw et al. (1984) and Stellato et al. (2017) derives a Chebyshev inequality bound with respect to the $(N+1)$-th sample. The Multivariate Chebyshev inequality (Stellato et al., 2017) can define the boundary for an ellipsoidal set centered at the mean.

Definition 6

Semantic prototype edges. Let $({F}_{c_{i}},\delta)$ be the metric space of object features of $i$-th category $c_{i}\in C$. Let $E\subseteq{F}_{c_{i}}$ be a set of features extracted for only typical objects of $c_{i}$-category, $N=\left|{E}\right|$, and $F_{o}$ the features of a object $o\in O_{c_{i}}$. We weakly defined as edges of our semantic prototype $P_{i}=\left({M_{i},\Sigma_{i},\Omega_{i}}\right)$, the threshold vector $\vv{\lambda_{i}}=\left[{\lambda_{i1},\lambda_{i2},...,\lambda_{im}}\right]$ that meets the expression:

[TABLE]

where $f_{j}\in F_{o}$, $\mu_{ij}\in M_{i}$ and $\sigma_{ij}\in\Sigma_{i}$,$\>\forall i=1...n;\,\forall j=1...m.$. Finally, given a probability bound, it is possible to compute a threshold vector $\vv{\lambda_{i}}$ and construct a confidence ellipsoidal set from the sample mean and covariance of only typical objects samples (a stronger semantic prototype edges definition can be performed using – completely– the  Stellato et al. (2017) statements).

Figure 3 shows the expected representation of category internal structure based on our Computational Prototype Model (CPM) [Semantic prototype (Definitions 1 and 2) + Semantic distance (Definitions 3 and 4)]. With our CPM model, we try to respect some important concepts of the Prototype Theory: i) category prototype edges are defined with our vector $\Sigma_{i}\in P_{i}=\left({M_{i},\Sigma_{i},\Omega_{i}}\right)$; ii) category edges are blurred (not sharp defined) because our semantic prototype is not computed with all category elements (only with typical elements); iii) objects representativeness (typicality) within the category is simulated with our prototypical distance.

3.3 Prototype Construction

Our semantic prototype representation can be easily computed by any model with the ability to extract object features of images ($F_{o}$) and learn the unitary relevance value ($\omega_{ij}$) of each j-$th$ object feature in i-$th$ category. We have also considered the elements typicality within the category to compute our semantic prototype. Consequently, we need objects datasets with annotations of objects typicality scores.

Moreover, our object description approach presented in Figure 1 attempts — following the human behavior — to use the same features extracted to classify and describe objects. First, we need to recognize the category to which the object belongs and then, find what the object features that distinguish it from others within the category are. However, how to model a global object description with similar behavior of the Figure 1 diagram?

To address these issues, we rely on the fact that CNNs provide outstanding performance in image semantic processing and classification tasks. We used CNN -classification models for features extraction, recognition, and classification of the visual information received as input (processes 1 to 4 in Figure 1). CNN-models, analogous to the human memory (Fuster, 1997), make associations that keep the knowledge in its connection structures. Our method downloads that knowledge of pre-trained CNN-models into a semantic structure (semantic prototype), which aims is to stand for the central semantic meaning of learned categories (see step 5 in Figure 1).

Definition 7

Convolutional semantic prototype. The convolutional semantic prototype of $i$-th category $c_{i}\in C$ is a $4$-tuple $P_{i}=\left({M_{i},\Sigma_{i},\Omega_{i},b_{i}}\right)$, where $M_{i},\Sigma_{i}$ are computed using features of $c_{i}$-category extracted from the fully convolutional layer of pre-trained CNN - classification models; and $\Omega_{i},b_{i}$ are the learned parameters (learned features relevance) of i-$th$ category in the softmax layer. Next, we refer to convolutional semantic prototype of the category as a semantic prototype.

Algorithm 1 details the computation of a semantic prototype for a specific category. Given a labeled object images dataset, for each object category in dataset, we use Algorithm 1 to compute the correspond semantic prototype (off-line processing). The resulting semantic prototypes dataset is used as prior knowledge in our prototype-based description model (see Figure 1). Figure 4 shows the main steps and concepts of our prototype construction algorithm.

Semantic prototype visualization

Visual representation of semantic prototypes allows presenting a visual summary of category typical features. Some approaches (Wohlhart et al., 2013; Li et al., 2018) learn prototypes representations in the image domain; and consequently, prototype visualization is simply the image learned. These approaches require a considerable computational expense to learn its prototypes visualization. Wohlhart et al. (2013) introduced the learning of image prototypes representations in the back-propagation process. On the flip side, Li et al. (2018) used an encoder-decoder deep architecture to learn prototypes visualization. Since our semantic prototype representation is constructed straightforwardly from pre-trained CNN classification models, the methods mentioned above are not appropriate to visualize our prototype representation.

Binder et al. (2016) proposed a circular visualization of semantic mean attribute vectors for concrete object noun categories. Consequently, a simple approach for visualizing our semantic prototype representation is to visualize the distribution values of each $m$-dimensional vector that compose the $P_{i}$-tuple of our semantic prototype definition.

Figure 5 shows an illustration of our semantic prototype representation corresponding to $i$-th category. We showed each tuple-member ($m$-dimensional vector) that composes the proposed semantic prototype of $i$-th category, $P_{i}=\left({M_{i},\Sigma_{i},\Omega_{i},b_{i}}\right)$. We represented the learned bias value $b_{i}$ as the bias $m$-dimensional vector $\vv{b_{i}}\in\mathbb{R}^{m},\,\vv{b_{i}}=\dfrac{b_{i}}{m}\cdot\vv{1}$ such that $b_{i}=\sum_{m}\vv{b_{i}}$.

It is noteworthy that our semantic prototype has a values distribution that is characteristic of $i$-th category it represents. I.e., our semantic prototype can be understood as a DNA chain that stands for the category members’ typical features. The semantic prototype representation uniqueness is guaranteed by the relevance vector ($\Omega_{i}$), which was learned specifically for that $i$-th category when the CNN-classification model was trained.

4 Global Semantic Descriptor

In the previous section, we presented a framework to encapsulate the central meaning (semantic prototype) of an object category. In this section, we present how to introduce that semantic prototype representation to simulate the object semantic description work-flow depicted in Figure 1.

4.1 Semantic Meaning

Some cognitive neuroscience researches have studied the effect of semantic meaning in object recognition task (Tulving, 2007; Martin, 2007; Collins and Curby, 2013). When an object has been previously associated with some type of semantic meaning in the brain, people are more prone to identify the object (Tulving, 2007; Martin, 2007) correctly. Studies (Tulving, 2007; Martin, 2007; Collins and Curby, 2013) have shown that semantic associations allow a much faster recognition of an object, even when the task of object recognition becomes increasingly difficult (varying points of view, occlusion) (Collins and Curby, 2013). Therefore, semantic associations based on object semantic meaning allow for faster object recognition.

Moreover, the fact that some CNN models (e.g., ResNet (He et al., 2016)) outperform the human-reported performance ($5.1\%$(Russakovsky et al., 2015)) on large-scale visual object classification tasks, generated some cognitive studies (Yamins et al., 2014; Cadieu et al., 2014; Khaligh-Razavi and Kriegeskorte, 2014; Cichy et al., 2017) to research the possible links between CNN models and visual system in the human brain. Cichy et al. (2017) suggested that deep neural networks perform spatial arrangement representations like those performed by a human being. Khaligh-Razavi and Kriegeskorte (2014) concluded that the weighted combination of features in the last fully connected layer of CNN models could thoroughly explain the inferior temporal cortex in the human brain. We lay hold of these theoretical foundations to model our representation of objects semantic meaning.

Definition 8

Semantic value. Let be $F_{o}$ observed features of an object ${\emph{o}\in O}$ ($F_{o}=\left\{f_{1},f_{2},...,f_{m}\right\}$). The semantic meaning of object features $F_{o}$ for category $c_{i}\in C$, summary value of features $F_{o}$, or simply semantic value of $F_{o}$ in $c_{i}$-category is an abstract value: $z=\sum_{m}\omega_{ij}f_{j}+b_{i},$ where $\omega_{ij}\in\Omega_{i},\,f_{j}\in F_{o}$. Consequently, the semantic value of ideal member of $c_{i}$-category, central semantic meaning of $c_{i}$-category or summary value of the semantic prototype $P_{i}=\left({M_{i},\Sigma_{i},\Omega_{i},b_{i}}\right)$ is the semantic value $\hat{z_{i}}=\sum_{m}\omega_{ij}\mu_{j}+b_{i},$ where $\omega_{ij}\in\Omega_{i},$ and $\mu_{ij}\in M_{i}$ are the abstract prototype features, $\forall i=1,...,n;\,\forall j=1,...,m.$

Note that our object semantic value is exactly the same value used to object categorization in softmax layer of CNN-classification models. Hence, our approach of object semantic description based on prototypes assumes as object semantic meaning vector, the semantic vector ($\vv{z}=\Omega_{i}\odot F_{o}+\vv{b_{i}}$) constructed with the element-wise operations to compute the object semantic value (Definition 8). Our semantic meaning representation uses a bias vector ($\vv{b_{i}}$) to uniformly dissolve the bias value in each semantic vector component ($b_{i}=\sum_{m}\vv{b_{i}})$. With this approach it is enough a sum of each semantic meaning vector component to recover the object semantic value ($z=\sum_{m}\vv{z})$. Accordingly, our semantic meaning vector contains the same semantic definitions used for CNN models to categorize an object within a specific category.

4.2 Semantic Difference

We stand for the semantic distinctiveness of an object for specific $c_{i}$-category as the semantic discrepancy between object features and features of the most prototypical (ideal) element of $c_{i}$-category (abstract prototype of $c_{i}$-category). Since object features ($F_{o}$) and abstract prototype of $c_{i}$-category ($M_{i}\in P_{i}$) belong to the same features domain (features metric space), we apply our prototypical distance as measure of the objects distinctiveness within a category.

Consequently, our approach assumes as object semantic distinctiveness vector, the semantic difference vector ($\vv{\delta}=\Omega_{i}\odot\left|F_{o}-M_{i}\right|$) constructed with the element-wise operations to compute the object prototypical distance (Definition 4). Our semantic difference vector is the weighted ($\Omega_{i}$) residual vector ($\vv{r}=\left|F_{o}-M_{i}\right|$) composed of absolute values of the difference between each object feature and each feature of $c_{i}$-category abstract prototype.

Note that our object semantic difference (or our prototypical distance) can be understood as the sum of absolute difference between the object semantic meaning vector ($\vv{z}$) and the central semantic meaning vector ($\vv{\hat{z_{i}}}$) of $c_{i}$-category. Thus, Equation 2 is equivalent to $\sum_{j=1}^{m}\left|\vv{z_{j}}-\vv{\hat{z_{ij}}}\right|$ = $\sum_{j=1}^{m}\left|\omega_{ij}f_{j}-\omega_{ij}\mu_{ij}\right|$ = $\delta(\emph{o},P_{i})$ when $\forall\omega_{ij}\in\Omega_{i}$, $\omega_{ij}\geq 0$ (we introduced this $\omega_{ij}$ constraint in the semantic distance of MPM model). Therefore, our object semantic difference representation has the advantage that elements vector sum is enough to retrieve the object prototypical distance ($\delta=\sum_{m}\vv{\delta})$.

Figure 6 depicts an overview of our prototype-based description model. Our Global Semantic Descriptor based on Prototypes (GSDP) uses as a requirement the prior knowledge of each category prototype (prototypes are precomputed off-line using Algorithm 1). After feature extraction and categorization processes (Figure 6b), we use the corresponding category prototype for semantic description of object features. We show in Figure 6e) the steps to introduce the category prototype into the global semantic description of object’s features. A drawback of our object semantic representation (Figure 6e) is having high dimensionality, since it is based on semantic meaning vector ($\vv{z}$) and semantic difference vector ($\vv{\delta}=\Omega_{i}\odot\vv{r}$). The large dimensionality of our feature vectors might make its use unfeasible in common computer vision tasks (Han et al., 2017; Kim et al., 2018). Figure 6 and Algorithm 2 detail the main steps of our approach; note that steps follow the same work-flow of human description hypotheses depicted in Figure 1.

4.3 Dimensionality Reduction

Several dimensionality reduction algorithms such as PCA (Abdi and Williams, 2010) and NMF (Lee and Seung, 2001) are based on discarding features that do not generate a meaningful variation. Although these approaches work on some tasks, after applying these algorithms, we lost the ability of data interpretation (Abdi and Williams, 2010). From the Prototypes Theory perspective, discarding features is no suitable when it is applied to the semantic space due to the absence of necessary and sufficient definitions to categorize an object (intensional non-discreteness). Occasionally when discarding features might lead in discarding elements of the category (Geeraerts, 2010). For instance, there may be some objects within the category that do not have some category typical features (flying is a typical feature of bird category; however, a penguin is a bird that does not fly).

We proposed a simple transformation function $f(x)$ to compress our global semantic representation of the object’s features (Figure 6e) in a low dimensional global semantic signature (Figure 6g). Our transformation function aims to reduce our semantic representation dimensionality while keeping the property of easy retrieve the object semantic meaning and object semantic difference from the final descriptor signature. Our final descriptor signature $(\psi)$ is computed by concatenating the corresponding signatures of semantic meaning vector ($\vv{z}$) and semantic difference vector ($\vv{\delta}$) compressed with our $f(x)$ transformation (see Algorithm 2).

Figure 7 shows the main steps of our $f(x)$ transformation. We use a square auxiliary matrix ($\chi_{r\times r}$) as a parameter to control the descriptor signature dimensionality. The auxiliary matrix dimensions is a parameter that allows us to control the final GSDP-signature size, i.e.; larger auxiliary matrix dimensionality leads to smaller GSDP signature; and vice versa.

The main steps showed in Figure 7 can be summarized as: 1) Resize the input vectors in the best 2D dimensional configuration of matrices ($p\times q$) whose dimensions are multiples of $r$ (auxiliary matrix dimension). 2-3) Compute the angles matrix ($\Theta_{r\times r}$) with angles formed by the position of each feature with respect to auxiliary matrix $\chi_{r\times r}$ center; to achieve uniqueness the diagonal angles were evenly distributed between the magnitudes of angles $\alpha$ and $\beta$. 4) - 5) Create unitary semantic gradient for each auxiliary matrix mapped within $p\times q$ matrices; each semantic gradient is constructed using the angle matrix ($\Theta_{r\times r}$), and magnitude and sign of semantic vectors computed using Definition 2 and 8. 6) Reduce the semantic gradient to 8-vectors similarly to SIFT approach (Lowe, 2004); 7) Concatenate, for each auxiliary matrix $\chi_{r\times r}$ mapped, the corresponding unitary 8D-signatures resulted of flow $4$-$6$. Algorithm 3 details all steps.

Hence, our final descriptor signature preserves the object semantic meaning (Property 1) and the object semantic difference (Property 2) presented in our first global semantic representation of object features (Figure 6e). Additionally, depending of the input vector, our descriptor can uses $f(x)$ transformation to construct global semantic representations (signatures) with different meanings within $i$-th category (Property 3). In other words, our descriptor can construct semantic representations (see Figure 6e) for: i) an object, ii) ideal category member (abstract prototype), and iii) category semantic meaning encapsulated with semantic prototype boundaries.

4.4 Descriptor Properties

Property 1

Semantic meaning preservation. The semantic descriptor signature preserves the object semantic value: $\sum_{l=0}^{\left|\psi\right|/2}\psi[l]=\hat{z}.$

Proof

To prove this, it suffices to follow backward through steps $6$ and $\left[9,17\right]$ of Algorithm 2 and 3, respectively. $\sum_{l=0}^{\left|\psi\right|/2}\psi$ = $\sum f\left({\alpha,\Omega_{i},b_{i},\textit{meaning}}\right)$ = $\sum\sum_{j}\sum_{k}g^{jk}$ = $\sum\Omega_{i}\odot\alpha+\bar{b_{i}}$= $\sum\vv{z}$ = $\hat{z};~{}\alpha\in\left\{{M_{i},F_{o}}\right\}.$

Property 2

Prototypical distance preservation.  If ${\emph{o}\in O_{c_{i}}}$ is a object of $i$-th category, the object signature $\psi_{o}$ preserves the object prototypical distance: $\sum_{l=\left|\psi_{o}\right|/2}^{\left|\psi_{o}\right|}\psi_{o}[l]=\delta(\emph{o},P_{i}).$

Proof

Similar to the previous proof, but using distinctiveness vector $(\left|\Omega_{i}\right|\odot\left|F_{o}-M_{i}\right|)$ through steps $7$ and $\left[9,17\right]$ of Algorithm 2 and 3, respectively.

$\sum_{l={\left|\psi\right|/2}}^{\left|\psi\right|}$ = $\sum f({\left|F_{o}-M_{i}\right|,\Omega_{i},b_{i},\textit{distinctiveness}})$

$\sum\sum_{j}\sum_{k}g^{jk}$ = $\sum\left|\Omega_{i}\right|\odot\left|F_{o}-M_{i}\right|$ = $\sum\vv{\delta}$ = $\delta(\emph{o},P_{i}).$

Property 3

Structural polymorphism. Our global semantic descriptor GSDP has the polymorphic property of describing, with the same structural representation, distinctly different semantic meanings within the $c_{i}$-category. Thus, our descriptor uses the category prototype $P_{i}=\left({M_{i},\Sigma_{i},\Omega_{i},b_{i}}\right)$ to construct different semantic signature taxonomies:

1. i)

an object $o\in O_{c_{i}}$, $\psi_{o}$ = $\psi(F_{o},\left|F_{o}-M_{i}\right|,\Omega_{i},b_{i})$ =

$f\left({F_{o},\Omega_{i},b_{i},\textit{meaning}}\right)$ $\oplus$

$f(\left|F_{o}-M_{i}\right|,\Omega_{i},b_{i},\textit{distinctiveness})$; 2. ii)

central semantic meaning of $i$-th category (abstract prototype), $\psi_{P_{i}}$ = $\psi(M_{i},\left|M_{i}-M_{i}\right|,\Omega_{i},b_{i})$

= $\psi(M_{i},\vv{0},\Omega_{i},b_{i})$; 3. iii)

semantic meaning of $i$-th category (semantic prototype), $\psi_{i}$ = $\psi(M_{i},\Sigma_{i},\Omega_{i},b_{i})$.

5 Experimental Evaluation

5.1 Experimental Setup

Aside from performing experiments using benchmark image datasets with fixed-size, size-normalized and centered images like MNIST (Lecun et al., 1998) and CIFAR (Krizhevsky and Hinton, 2010), we also evaluated our approach on ImageNet (Russakovsky et al., 2015) as real images dataset. For each image dataset, we used a CNN-classification model for feature extraction and classification (see Figure 6b). Thus, we used a CNN-MNIST and CNN-CIFAR models based on LeNet (Lecun et al., 1998) and Deep Belief Network (Krizhevsky and Hinton, 2010) architectures for image classification in MNIST and CIFAR datasets, respectively. Also, we conducted experiments in ImageNet using VGG16 (Simonyan and Zisserman, 2014) and ResNet50 (He et al., 2016) models as background of our global semantic description model. Note that our prototype-based description model depicted in Figure 6, is scalable and can easily be adapted to any other CNN-classification model.

Prototypes Dataset Construction

Our prototype-based description model requires prototypes dataset as category prior knowledge (see Figure 6c) to build object semantic representations (see Figure 6e) that stand for the object distinctiveness within the category. In the experiments, we computed prototypes datasets with CNN-MNIST, CNN-CIFAR, VGG16, and ResNet50 models in MNIST, CIFAR, and ImageNet datasets, respectively.

For feature extraction, we assumed as object features those extracted from the last dense layer (before the softmax layer) of the CNN-model. Our approach needs typical objects of categories or any information about typicality score (or typicality degree) of objects belonging a specific category to build the proposed semantic prototype properly. However, none of the images datasets used have this annotation. Lake et al. (2015) showed that the output of the last layer of CNN models could be used as a signal for how typical is an input image. Consequently, we used as typicality score of objects the strength of classification response to the category of interest. Specifically, we assumed as typical members of a category those elements that are — unequivocally — classified as category members (typicality score $>0.99$) by CNN models (see Figure 4). Finally, for each category in datasets, we extracted features of typical members and computed the correspond semantic prototype (see Definition 1) using Algorithm 1.

5.2 The Semantics behind our Computational Prototype Model

Achieving the member’s prototypical behavior within a category is one of the motivations and theoretical basis of our approach. Nevertheless, there is no defined metric to quantify whether our representation correctly captures the category semantic meaning. This lack of a metric is a consequence of the fact that there is no defined metric to evaluate the object typicality level within a category robustly; this skill is still reserved only for human beings.

In this section, we analyzed the semantics captured by our CPM Model (semantic prototype + prototypical distance). The CPM model pursues two main goals: i) capture, with the semantic prototype, the central semantic meaning of a specific object category; ii) simulate, in a comparable way to human beings, that visually typical elements of category are organized close (based on our prototypical distance metric) to category prototype. Since we do not have annotated images with the object typicality score to robustly evaluate the semantic captured by our representation, we used another approach to analyze the semantics behind our CPM model.

5.2.1 Central and Peripheral meaning

In this section, we analyzed the central and peripheral meaning captured by our CPM model. Since we defined abstract prototype as the abstract semantic center of category, we observed how relevant (or visually representative) – for the category – are those elements allocated by our CPM model in center and periphery of category. Expressly, this experiment aims to know what is the visual representativeness of category members closest and furthest from the category semantic center (our abstract prototype).

To achieve such a goal, we extracted object image features using a CNN model and computed our prototypical distance for all members of $i$-th category. Finally, the objects images are sorted in ascending order based on the prototypical distance value of each element. Figures 8 and 9 present some examples of central and peripheral meaning captured by our CPM model for images categories of ImageNet and MNIST datasets using VGG16 and CNN-MNIST models – respectively – as image feature extractors.

Notice that our proposal for object image semantic interpretation – using our CPM model in CNN image feature domain – attempts to assign a visual representativeness value (or typicality) to object image within the category to which it belongs.

Figure 9 shows the Top-5 closest and Top-5 furthest elements from category center (abstract prototype) detected by our CPM model in MNIST categories. For instance, our proposal finds as typical elements (Top-5 closest) of number three category the handwritten digits with features that are, undoubtedly, distinctive of $c_{3}$-category. Our CPM model also can find the peripheral meaning of the category. Members with fewer characteristic features of number three, or little readable, are placed in the periphery (Top-5 farthest) away from the central semantic meaning, but keeping the category features (it still belongs to the category). Similar to a human being, our CPM model can find the Top-5 farthest members of number three category that are a number $3$, but not a typical number $3$.

Figure 8 presents the semantic interpretation of visual image information performed by our CPM model in real object images of ImageNet dataset. Note that category members recognized by our CPM model as Top-5 closest members (left column) to category semantic center are easy recognized by human beings, as it exhibits the typical features of an object category. Also, we observed that Top-5 furthest elements (right column) from the semantic prototype (or less representative members of $i$-th category) detected by CPM model although retaining some category features are not easily recognized by human beings. That is, our CPM model can identify the most/least visually representative category members, and — correctly — recognizes as category periphery members those elements, where not all typical category features are identified: category typical colors, size, shape, etc. are not easily distinguishable; or object pose in the image does not exhibit these representative features of $i$-th category. The experiments performed allow to assume that our CPM model can capture the central/peripheral semantic meaning of images categories. But, we still need to answer to the question: Can our CPM model organize all category elements prototypically?

5.2.2 Prototypical Organization

The experiments in this section aim to visualize the internal semantic structure of the category using the semantic meaning encapsulated by our CPM model for each category member. Based on features extracted from objects images, we analyzed the object prototypical behavior observing where it is positioned within the category by our CPM model (using our prototypical distance). Visualizing the semantic position of each category member with respect to category abstract prototype constitutes a simple approach to see the internal semantic structure of the entire category structure. We need to corroborate that our CPM model can correctly interpret the object image features and position it semantically within the category, keeping a prototypical organization of the category.

Note that our CPM model uses $m$-dimensional object features from CNN image-features domain. Accordingly, visualizing the category’s internal structure is infeasible in $m$-dimensional features space since most techniques of data visualization are based on features discarding. From the perspective of the Prototype Theory foundations, features discarding approach can be problematic (intensional non-discreteness). For this reason, we used topology techniques to make object image interpretation based on all observed features. We constructed a map function to show that our CPM model can simulate the prototypical organization of members within a category.

Let $({F}_{c_{i}},\delta)$ and $(\mathbb{R}^{2},L_{1})$ be metric spaces (see Definition 5), and $\rho$ a function that maps image object features to $(\mathbb{R}^{2},l_{1})$ metric space using its semantic value and its prototypical distance. I.e., $\rho:{F}_{c_{i}}\to\mathbb{R}^{2}\mid\rho(o\in O_{c_{i}})$ = $\rho(F_{o})$ = $p(z_{o},\delta(o,P_{i}))$, where $F_{o}$ are the object features, $z_{o}$ is the object semantic value, $\delta(o,P_{i})$ is the object prototypical distance; the point  $p(x,y)\in\mathbb{R}^{2}$ and $L_{1}$ is L1-norm condition.

Let be the objects $o_{1},o_{2}\in O_{c_{i}}$, and $p_{1}$ = $\rho(o_{1})$, $p_{2}$ = $\rho(o_{2})$ be the corresponding mapped points in $(\mathbb{R}^{2},L_{1})$ metric space. Then, the Sum of Absolute Difference (SAD) between $p_{1}$ and $p_{2}$ is $L_{1}(p_{1},p_{2})$ = $L_{1}(\rho(o_{1}),\rho(o_{2}))$ = $L_{1}(p(z_{1},\delta(o_{1},P_{i})),p(z_{2},\delta(o_{2},P_{i})))$ = $|z_{1}-z_{2}|$ + $|\delta_{1}-\delta_{2}|$; using Definitions 3, 4, and 8 we have: $\delta(o_{1},o_{2})\leq L_{1}(p_{1},p_{2})\leq 2\delta(o_{1},o_{2})$. Consequently, for every $F_{o_{1}},F_{o_{2}}$ $\in F_{c_{i}}$ and $\varepsilon>0$, exists a $\varphi$ = $\frac{\varepsilon+1}{2}>0$ such that: $\delta(o_{1},o_{2})<\varphi$ $\Rightarrow$ $L_{1}(\rho(o_{1}),\rho(o_{2}))<\varepsilon$, i.e., $\rho$ is continuous. This means that every element of $\rho(o_{1})$ neighborhood in $(\mathbb{R}^{2},L_{1})$ metric space, also belongs into $o_{1}$ neighborhood in $({F}_{c_{i}},\delta)$ metric space (if $\rho(o_{1})$ = $p_{1}$, $\forall p\in\left\{p_{1}\>\textit{neighborhood}\right\}$, $\rho^{-}(p)\in\left\{o_{1}\>\textit{neighborhood}\right\}$). Consequently, the observed behavior of $i$-th category internal structure – in terms of distance metrics– in $(\mathbb{R}^{2},L_{1})$ metric space is equivalent to the behavior in feature metric space $({F}_{c_{i}},\delta)$.

Figure 10 shows an example of the internal semantic structure of MNIST and ImageNet images categories mapped using $\rho$. Note how Top-5 closest members (based on our prototypical distance) are mapped (in blue) and positioned near (based on L1 distance) to the mapped abstract prototype (in black). The Top-5 most visually representative members of each category in $({F}_{c_{i}},\delta)$ metric space are the same Top-5 most representative (closest to mapped abstract prototype) in $(\mathbb{R}^{2},L_{1})$ metric space. Likewise, the Top-5 fewer representative members (in red) continue to be positioned in the category peripheries, far away from the category abstract prototype  (our central semantic meaning representation). The experiments show a prototypical organization of mapped members within the category in $(\mathbb{R}^{2},L_{1})$ metric space. Consequently, based on $\rho$ properties, a similar grouping of objects based on family resemblance is preserved in CNN-features metric space.

Our approach to visualize the category internal structure also allows observing other semantic phenomena related to the object image visual representativeness. The experiments showed that object semantic value and object prototypical distance place the object image in a unique semantic position within the category internal structure. This result shows that our approach of constructing a semantic object representation (see Figure 6e) based on vector versions of semantic value and prototypical distance can be able to describe the object image semantically.

5.2.3 Image Typicality Score

We observed that the shape of category internal structure – in $(\mathbb{R}^{2},L_{1})$ metric space using our visualization approach – strongly depends on semantic values distribution and prototypical distance distribution. Consequently, in this section we analyzed the relationship between semantic value and prototypical distance variables. Also, we examined how the variations of these variables can influence on object image visual representativeness (typicality) within the category.

Our prototypical distance can be understood as the semantic difference between the object semantic meaning and the semantic meaning of category abstract-prototype (see Subsection 4.2). Specifically, if features relevance ($\Omega_{i}$) of $i$-th category is strictly positive ($\omega_{ij}\geq 0$, $\forall\omega_{ij}\in\Omega_{i}$) then, the variables prototypical distance and semantic value are – by construction – strongly correlated. However, experiments in MNIST, CIFAR and ImageNet datasets with each corresponding CNN-model showed that there is a small strength of a linear association between those two variables (Pearson coefficient values between $-0.3$ and $0.3$), but it does not conclude that we can generalize a behavioral pattern between object semantic value and prototypical distance. This Pearson correlation result is consequence of the fact that the weights learned in softmax layer (our feature relevance $\Omega_{i}$) of CNN models used for feature extraction (CNN-MNIST, CNN-CIFAR, VGG16 and ResNet50) are not strictly positive. Consequently, the semantic value is not a strong measure because the addends in the equation can cancel each other out (see Definition 8), and elements with same semantic value does not imply that elements are equal ($z_{o_{1}}=z_{o_{1}}\centernot\implies o_{1}=o_{2}$).

Lake et al. (2015) showed that semantic value can be used as a signal for how typical an input image looks like. In contrast to Lake et al. results, our experiments with VGG16 and ResNet50 models in ImageNet dataset showed that using the semantic value as object typicality score can be problematic because objects with same semantic value do not imply same image visual typicality. Figure 10 shows an example of this phenomenon. In Persian cat ImageNet category, the 5th element of Top-5 closest to category prototype (in blue) has a semantic value like 2nd position element of Top-5 furthest (in red) (both images semantic values are $\approx$ 2), but objects images are visually different. That is, the semantic value could be a necessary condition to image typicality representation, but it is not enough. On the other hand, note how our prototypical distance can capture the visual typicality difference between those two objects images.

We observed what is the image visual information behavior when one of those semantic variables (semantic value and prototypical distance) change. We kept constant the value of one variable, and then, we analyzed the visual representativeness of the corresponding object images when the value of another variable increase. Figure 11 shows an example of this experiment within the Persian-cat ImageNet category. Note how for a fixed prototypical distance (elements in red), the semantic value variation does not generate significant changes in image visual representativeness (typicality) within the category. In contrast, for a fixed semantic value (elements in blue), the prototypical distance variation generates typicality ordered changes in the image’s visual information. We observed that when prototypical distance increases, object image visual typicality decreases. In contrast, the experiments did not allow to generalize a behavior pattern between semantic value and image typicality.

Based on the results of our experiments, we assumed that our semantic prototype representation correctly captures the central semantic meaning of images categories. Even with different CNN models and images datasets, our CPM model organizes the internal category structure following a prototypical organization of category members. Besides, we showed that our prototypical distance influences elements arrangement around the category semantic prototype. Since our prototypical distance is a metric in CNN-feature domain, our semantic distance can be used as object image typicality score within the category (typicality score $(\emph{o})=1/\delta(\emph{o},P_{i})$).

5.3 Global Semantic Descriptor based on Prototypes

5.3.1 Descriptor Configuration

By construction, the dimensionality of our GSDP descriptor signature depends of the object image CNN-features dimensionality (image features extracted with CNN classification model used in background), and dimensions of auxiliary matrix ($\chi_{r\times r}$) used as parameter in our $f(x)$ transformation (see Figure 7 and Algorithms 2, 3). With higher auxiliary matrix dimensionality, smaller is our GSDP signature size; and vice versa.

Consequently, since we needed the size variation of image CNN-features to evaluate our prototype-based description model, we used different CNN models as images features extractors. CNN models selection criteria were based on trying to evaluate our semantic description approach in different contexts: image CNN-features with different sizes, CNN models with varied architecture and depth, and image datasets of diverse nature (image resolution, image type, etc.). Also, for each CNN model used, we configured (using the auxiliary-matrix parameter) our semantic descriptor to return GSDP-signatures with noticeably different dimensionality.

Table 2 presents details of GSDP descriptor settings used to construct each semantic GSDP signature evaluated in our experiments. For each CNN-model used, we exhibit the CNN-feature status at each step of the workflow of our dimensionality reduction function ($f(x)$) (see Figure 7). Table 2 shows the CNN classification models used as feature extractor; image CNN-feature length ($|F|$); new CNN-feature shape ($F_{p\times q}$) after apply Step 1 of our $f(x)$ transformation; auxiliary matrix dimension ($\chi_{r\times r}$) used as parameter (we used two different configurations for each CNN model); number of matrices that make up our semantic gradient ($g^{jk}$); dimensionality of intermediary feature constructed with our $f(x)$ transformation ($|f(x)|$); and final length of GSDP signature ($|\psi|$) for each descriptor setting.

Note that our global semantic descriptor executes twice the $f(x)$ transformation to reduce the dimensionality of semantic meaning and semantic difference representations (see Algorithm 2). Consequently, GSDP signatures dimensionality is two times the dimensionality of $f(x)$ transformation features.

5.3.2 Signature Semantic Information

The experiments in the image CNN-features domain showed that object semantic value and prototypical distance organize all category members prototypically in a specific (and unique) position within the category semantic structure. The key idea behind our GSDP semantic descriptor is to encapsulate, in a vector representation, the same semantic interpretation –of image object features– captured by our CPM model. In this section, we show that our GSDP descriptor encodes and preserves the semantic information contained in an object features (semantic value and prototypical distance) used by our CPM model for semantic interpretation of object image. Also, we show how retrieving from GSDP descriptor signatures that semantic information and reconstructing the prototypical organization of object category achieved in the image CNN-features domain.

Let be $({\psi}_{c_{i}},L_{1})$ the metric space of object descriptor signatures. Descriptor properties 1 and 2 allow to easily recover the object semantic value and prototypical distance from GSDP descriptor signatures. Property 3 enables us to build descriptor signatures for abstract prototypes of categories. Similarly to $\rho$ map, we can construct and show that map $\gamma:$ $({\psi}_{c_{i}},L_{1})$ $\to(\mathbb{R}^{2},L_{1})\mid\gamma(\psi_{o}\in\psi_{c_{i}})$ = $p(\sum_{0}^{\left|\psi\right|/2}\psi_{o},\sum_{\left|\psi\right|/2}^{\left|\psi\right|}\psi_{o})$ = $p(z_{o},\delta(o,P_{i}))$ is continuous. Hence, we can map all category descriptor signatures to $(\mathbb{R}^{2},L_{1})$ metric space using $\gamma$ function.

With $\gamma$ map approach, we can reproduce the same semantic analysis performed in CNN-feature space. Experiments showed that the category prototypical organization achieved in $(\mathbb{R}^{2},L_{1})$ metric space is identical regardless of which $\gamma$ map (to descriptor signature domain) or $\rho$ map (to CNN-feature domain) function is used (e.g.  Figure 10 and 11). Consequently, the behavior observed in $(\mathbb{R}^{2},L_{1})$ metric space is equivalent to the behavior in feature metric space $({F}_{c_{i}},\delta)$ and descriptor signatures metric space $({\psi}_{c_{i}},L_{1})$. This means that our GSDP descriptor signature preserves, in its taxonomy, the same semantic information used by our CPM model to interpret object image CNN-features (semantic value and prototypical distance).

5.3.3 Signature Taxonomies

By definition, our GSDP descriptor uses category semantic prototypes as semantic distinctiveness generator of category members signatures. Elements with similar semantic meanings and that sharing similar semantic differences with the abstract prototype, will have similar GSDP semantic signatures. That is, since abstract prototype can be understood as a DNA chain that stands for the typical CNN-features of category members, the abstract prototype signature can be understood as a number distribution (or smaller DNA chain signature) that stands for category members signatures.

Figure 12 shows an example of the signatures taxonomies constructed with our GSDP semantic descriptor. We showed GSDP signatures constructed using CNN - MNIST model as features extractor of MNIST dataset images (signatures size = $32$ since we used the GSDP minimal setting (see Table 2)). Also, we showed the structural polymorphism property (Property 3) of our GSDP descriptor to construct signatures for the central semantic meaning (abstract prototype) and category members. With our approach, category members will have semantic signatures with a similar representation of category abstract prototype signature. Notice that very typical category elements will have descriptor signatures similar to the abstract prototype signature, and elements that do not belong to the category will have a quite different GSDP signature.

Our GSDP descriptor attempts to build – using our semantic prototype representation – a specific signature distribution for each object image category. Figure 10 and 11 show that category elements can be grouped, based on the meaning captured by our CPM model, by their family resemblance within the object category. However, this does not mean that in $m$-dimensional image features space, there are no elements of other categories in the neighborhood of a specific element. Since t-SNE algorithm (Maaten and Hinton, 2008) can preserve the local structure, we used t-SNE to analyze the element neighborhood in $m$-dimensional space. Maaten and Hinton (2008) exposed that points which are close to one another in the high-dimensional dataset will tend to be close to one another in the t-SNE low-dimensional map.

We analyzed the discriminative power and t-SNE visualization performance of our GSDP semantic image representation versus features extracted using CNN classification models. For each CNN-model used as background by our GSDP descriptor, we compared the t-SNE performance of features family built with each CNN model. We performed the t-SNE visualization experiment for features-family constituted by CNN features, corresponding GSDP semantic signatures, and reduced PCA versions of CNN-features (we reduced CNN-features to same GSDP feature dimensions).

Figure 13 shows the performance of t-SNE algorithm with each features-family in several image datasets using Euclidean distance as similarity measure and 50 as perplexity value. Note how GSDP representations achieved the best performance on each features-family. We observed that our GSDP object image representations are compactly grouped and have greater separation between categories than those t-SNE clustering built with high dimensionality features of CNN models (and its correspond PCA-reduced versions). Therefore, we can assume that our global semantic descriptor can construct object category representations distribution with the ability to maximize inter-class elements differences and minimize the intra-class differences. That is, with our approach, elements in each category must be as similar as possible, and elements in different groups must be as different as possible.

5.3.4 Performance Evaluation

Clustering

Yang et al. (2016) showed that when the image features representations achieve good metrics in image clustering task, it can generalize well when transferred to other tasks. Based on these assumptions, we evaluated our semantic GSDP encoding to verify its usefulness and suitability in image clustering task. We evaluated our GSDP descriptor (version based in VGG16 and ResNet50 classification models) performance in clustering task with ImageNet dataset image. We compared our GSDP representation performance against the following image global descriptors: GIST Oliva and Torralba (2001), LBP Ojala et al. (2002), HOG Dalal and Triggs (2005), Color64 Li (2007), Color_Hist Song et al. (2004), Hu_H _CH Haralick et al. (1973); Hu (1962); Song et al. (2004), VGG16 features and ResNet50 features (and its correspond PCA-reduced versions).

We used K-Means algorithm for clustering $50,000$ images ($500\times$category) of first $100$ ImageNet dataset categories. The selection criteria of the K-Means algorithm is based on some similarities of the K-Means method with our image semantic representation approach. K-Means method minimizes the sum of squared errors between data points and their nearest cluster centers. This approach has similarities with our GSDP representation since GSDP signatures were constructed to organize features categories using as category organization center the abstract prototype signature.

We evaluated each image representations performance in image clustering task comparing its K-Means clustering metrics (Homogeneity, Completeness, V-measure, Adjusted Rand Index, Adjusted Mutual Information). For each global image representation, the experiment was conducted incrementally, starting with $3$ cluster (for $3$ categories) and incrementing a category for each K-Means algorithm iteration. At the end of each K-Means execution, the clustering metrics were saved. The idea behind our clustering experiment was to evaluate each image representation performance as the amount and diversity of objects images increased.

Table 3 shows a screenshot of K-Means clustering metrics achieved by each global image descriptor for the first $20$ ImageNet categories. Also, Table 3 shows features dimension and feature extraction velocity (frame per second - FPS) for each image representation approach. All experiments were performed on a standard computer, without the use of GPUs to be fair with handcraft features approaches.

Note that our object image semantic representation achieved the best performance among the image representations evaluated with the ImageNet images sample. Also, be mindful of our GSDP descriptor representation keeps the same semantic information used by our CPM model for VGG16 and ResNet50 features interpretation (See Section 5.3.2 and Figure 10), but with more discriminatory image representation and even lower feature dimension. Experiments showed that lowest dimensional GSDP representations obtained the best cost-benefit performance.

Figure 14 shows K-Means metrics history for VGG16 and ResNet50 features representation against the correspond GSDP signatures. We showed K-Means metrics behavior for each image representation when the number of clusters increases (until 100 categories) in each K-Means algorithm execution. Experiments showed that as object images variety increased, K-Means clustering metrics related to CNN-features deteriorated significantly, while K-Means clustering metrics achieved by our image semantic encoding remain above 0.9. Results showed that our semantic descriptor encoding significantly outperforms others image global encodings in terms of cluster metrics.

Classification

To evaluate our image semantic encoding performance with supervised and unsupervised learning techniques, we also evaluated the performance of our GSDP representation in an image classification task.

Our GSDP descriptor, by construction, builds objects image representations based on object category predictions made by CNN model used as background (See step depicted in Figure 6b and Algorithm 2 line 4). Consequently, a prediction error of CNN-classification models generates that our descriptor constructs an object’s image semantic representation using a wrong semantic prototype. This behavior is not problematic if we take into account that human beings will erroneously describe an object if it was previously wrong recognized.

In this experiment, we evaluated the performance of two GSDP semantic representations. Each GSDP semantic representation was constructed considering two different scenarios: i) images GSDP-signatures are made based on the prediction of CNN model used in the background (normal behavior of our GSDP descriptor); ii) images GSDP-signatures are made based on the prediction of an ideal classification model (100% accuracy) (a hypothetical behavior of our GSDP descriptor). We used as a prediction of an ideal classification model, the image category label annotated in ImageNet dataset. We conducted the experiment to analyze the possible performance of our GSDP representation if the prototype selection error is zero (prediction error of CNN-model used in background).

We performed our classification experiments using the KNN algorithm since, similar to t-SNE algorithm; elements are classified based on their local neighborhood. We analyzed the GSDP representation performance increasing the KNN algorithm parameter value (K neighbor) and using Euclidean and Manhattan distances as feature similarity measures.

Figure 15 shows KNN algorithm performance using VGG16 and ResNet50 representations against corresponding GSDP signatures constructed in those two scenarios. In the experiment, we used the same ImageNet images sample used for clustering task evaluation. Also, we variated the K-value to show that our GSDP encoding significantly outperforms VGG16 and ResNet50 encodings in the KNN classification task.

Experiments showed that our GSDP representation using the ResNet50 model reached a better performance than those constructed using the VGG16 model. Also, we observed that GSDP representations constructed using category labels (notated with _uL in Figure 15) are highly discriminative (mean error close to 0). Consequently, we can conclude that our semantic encoding of objects substantially improves its performance – in classification task – as the accuracy of the CNN-classification model used in background increases.

Our experiments showed that our image global semantic representation based on category prototypes could outperform other image global representations in some computer vision tasks. Also, note that our GSDP encoding can describe objects images while encapsulates in its image signature the object semantic information (object semantic value and object typicality score). Experiments showed that lowest dimensional GSDP representations (for each CNN model) were the ones that achieved the best size-performance trade-off.

6 Limitations and Future works

The proposed prototype-based description model was constructed strictly to describe objects images, not to describe scenes images. Note that even when our semantic descriptor was evaluated in images that representing scenes, our GSDP descriptor can outperform other image global representations. The results achieved encourage us to evaluate the generalization ability of our object semantic representation in other computer vision tasks as image retrieval and scene understanding.

As future work, the interpretative criteria of human beings is necessary to construct images dataset with typicality annotations and conclude if our model can interpret objects images similar to human beings.

7 Conclusion

Motivated by how human beings represent and relate the meanings attributed to objects, this research was based on the Prototype Theory to propose semantic representations of object categories and object images. Specifically, in this paper we introduced and evaluated two models based on Prototype Theory foundations: i) a Computational Prototype Model (CPM) and ii) a Prototype-based Description Model.

We proposed the CPM model to represent the internal semantic structure of object categories. Experiments showed that our CPM model was able to encapsulate relevant features of objects category in our semantic prototype representation. Also, we showed that our semantic distance metric could simulate semantic relationships in terms of visual typicality, between category members. Our experiments showed that a relationship could be established between our semantic distance metric and object image visual representativeness. Expressly, our prototypical distance can be understood as the object image typicality score. That is, our CPM model can capture the object’s visual typicality and the central and peripheral meaning of objects categories.

Based on the CPM model results, we proposed a prototype-based description model that uses the CPM model main components (semantic prototype + semantic distance metric) to construct a semantic representation of object’s image. Our prototype-based description model uses semantic prototypes of the CPM model to build a discriminatory signature that semantically describes object images highlighting its most distinctive features within the category.

Our novel Global Semantic Descriptor based on Prototypes (GSDP)111All source code, prototypes datasets, and GSDP tool tutorials are publicly available in our lab’s Github: https://github.com/verlab/gsdp. introduces a new approach to the semantic description of object images. GSDP descriptor does not need to be trained, and it is easily adaptable to be used with any CNN-classification model. As shown in the experiments in the ImageNet dataset with VGG16 and ResNet50 models, our global semantic descriptor is discriminative, small dimensioned, and encodes the semantic information of category members. We further showed that our GSDP object representation preserves in its taxonomy the object’s semantic meaning and the object typicality score.

Our Prototype-based Description Model proposes a starting point to introduce the theoretical foundation related to the representation of semantic meaning and the learning of visual concepts of the Prototype Theory in the CNN semantic descriptors family.

Acknowledgements.

This research was supported by funding from the Brazilian agencies: Coordenação de Aperfeiçoa- mento de Pessoal de Nível Superior (CAPES), Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), and Fundação de Amparo a Pesquisa do Estado de Minas Gerais (FAPEMIG).