Look Further to Recognize Better: Learning Shared Topics and Category-Specific Dictionaries for Open-Ended 3D Object Recognition

TL;DR

This paper introduces an open-ended 3D object recognition method that learns shared visual topics and category-specific dictionaries, improving scalability and accuracy in real-world, long-term robotic applications.

Contribution

It proposes a novel unsupervised approach that simultaneously learns general and specific object representations incrementally from new views.

Findings

Significant improvements over state-of-the-art in scalability and classification accuracy.

Effective learning from very few training examples.

Best computational performance achieved with Bag-of-Words and LDA variants.

Abstract

Service robots are expected to operate effectively in human-centric environments for long periods of time. In such realistic scenarios, fine-grained object categorization is as important as basic-level object categorization. We tackle this problem by proposing an open-ended object recognition approach which concurrently learns both the object categories and the local features for encoding objects. In this work, each object is represented using a set of general latent visual topics and category-specific dictionaries. The general topics encode the common patterns of all categories, while the category-specific dictionary describes the content of each category in details. The proposed approach discovers both sets of general and specific representations in an unsupervised fashion and updates them incrementally using new object views. Experimental results show that our approach yields significant improvements over the previous state-of-the-art approaches concerning scalability and object classification performance. Moreover, our approach demonstrates the capability of learning from very few training examples in a real-world setting. Regarding computation time, the best result was obtained with a Bag-of-Words method followed by a variant of the Latent Dirichlet Allocation approach.

Tables

Table 1. TABLE I: Object recognition performance for different parameters using restaurant object dataset.

Parameters	VS (m)		IW (bins)		SL (m)			V (generic words)					K (topics)			$V^c$ (specific words)
Values	0.02	0.04	4	8	0.07	0.08	0.09	50	60	70	80	90	70	80	90	50	60	70	80	90
Avg. Accuracy(%)	85	81	83	83	82	83	83	82	82	83	84	84	84	83	82	81	84	84	83	82

Table 2. TABLE II: Recognition performance

Approach	Accuracy	Time (s)
RACE[1]	0.87	1757
BoW[18]	0.89	196
LDA[19]	0.83	258
Local-LDA[5]	0.91	349
Generic Rep.	0.90	234
Our Work	0.94	287

Table 3. TABLE III: Summary of open-ended evaluation.

Approaches	#$\text{QCI}$	$\text{ALC}$	$\text{AIC}$	$\text{GCA}$	$\text{APA}$
RACE[1]	382.10	19.90	8.88	0.67	0.78
BoW[18]	411.80	21.80	8.20	0.71	0.82
LDA[19]	262.60	14.40	9.14	0.66	0.80
Local-LDA[5]	613.00	28.50	9.08	0.71	0.80
Our work	1344.40	39.80	13.12	0.70	0.79

Equations

$${\varphi }_{w_i,k}=\frac{n_{w_i,k}+\beta }{n_k+|\text{V}|\times \beta },$$

$${\text{z}}_k=\sum _{i=1}^{V}p({\text{w}}_i|{\text{z}}_k)\times {\text{w}}_{i}k\in \{1,\dots ,K\}.$$

$${\chi }^2(P,Q)=\frac{1}{2}\sum _{i=1}^{K+V^c}\frac{(P_i-Q_i{)}^2}{(P_i+Q_i)}.$$

Full text

I Introduction

Nowadays service robots are leaving the structured and completely known environments and entering human-centric settings. For these robots, object perception is a challenging task due to the high demand for accurate and real-time response under changing and unpredictable environmental conditions. Although many problems have already been understood and solved successfully, many challenges still remain. Open-ended object recognition is one of these challenges waiting for many improvements. Cognitive science revealed that humans learn to recognize object categories ceaselessly over time. This ability allows adapting to new environments, by enhancing their knowledge from the accumulation of experiences and the conceptualization of new object categories. Inspired by this, we approach object category learning and recognition from a long-term perspective and with emphasis on open-endedness. In this paper, open-ended implies that the set of object categories to be learned is not known in advance, and the training instances are extracted from online experiences of a robot, and become gradually available over time, rather than being completely available at the beginning of the learning process.

In such a complex, dynamic and realistic setting, no matter how extensive the training data used for batch learning, a robot might always face a new object. Therefore, apart from batch learning, the robot should be able to learn new object categories from very few training examples on-site supported by human-in-the-loop feedback.

Moreover, the robot may frequently face a new object that visually can be either not similar (basic-level) or very similar (fine-grained) to the previously learned object categories (see Fig. 1). This poses a significant challenge in situations when the robot needs to recognize visually similar categories for which only a few examples are available. In such situations, object representation plays a central role because the output of this module is used for learning as well as recognition. In this paper, we propose a new method to characterize an object based on a common set of latent visual topics, which is used to describe the content of all categories (basic-level), and a set of category-specific dictionaries for highlighting the small diversity within the different categories (fine-grained). The representation of the given object is finally obtained by concatenating the generic and specific representations. To the best of our knowledge, there is no other approach that can jointly learn a set of generic and category-specific features for encoding 3D object categories in an open-ended manner.

II Related work

In the last decade, various research groups have made substantial progress towards the development of learning approaches which support online and incremental object category learning [1][2]. In recent studies on object recognition, much attention has been given to deep Convolutional Neural Networks (CNNs). It is now clear that if in a scenario, we have a fixed set of object categories and a massive number of examples per category that are sufficiently similar to the test images, CNN-based approaches yield good results, notable recent works include [3][4]. In open-ended scenarios, these assumptions are not satisfied, and the robot needs to learn new concepts on-site using very few training examples. While deep learning is a very powerful and useful tool, there are several limitations to apply CNNs in open-ended domains. In general, CNN approaches are incremental by nature but not open-ended, since the inclusion of new categories enforces a restructuring in the topology of the network. Furthermore, training a CNN-based approach requires long training times and training with a few examples per category poses a challenge for these methods. In contrast, open-ended learning [5][6] allows for concurrent learning and recognition. Our approach falls into this category.

In the case of object representation, most of the recent approaches use either neural networks [7][4] or hand-crafted features [8]. These approaches may not be the best option for such domains since the object representation procedure is a built-in component of the system. Oliveira et al. [9] tackle this problem by proposing an approach for concurrent learning of visual codebooks and object categories in open-ended domains. Unlike our approach, they completely discard information related to the co-occurrence of local object features. Moreover, they did not consider fine-grained categorization. Existing approaches for fine-grained categorization heavily rely on accurate object parts/features annotations during the training phase. Such requirements prevent the wide usage of these methods. However, some works only use class labels and do not need exhaustive annotations. Geo et al. [10] proposed a Bag of Words (BoW) approach for fine-grained image categorization. This work is similar to ours since they represent objects using generic and specific representations. However there are some differences: their codebooks are constructed offline; therefore, the representativeness of the training set becomes critical to the performance of the system. Moreover, this approach is impractical for an open-ended domain (i.e., large number of categories) since the size of object representation is linearly dependent on the number of known categories. Zhang et al. [11] proposed a novel fine-grained image categorization system. Similar to our work, they only used class labels during the training phase. Unlike our approach, in [10][11] the representations of known categories do not change after the training stage. Moreover, they completely discarded co-occurrence (structural) information. This limitation might lead to non-discriminative object representations and, as a consequence, to poor object recognition performance.

Several types of research have been performed to assess the added-value of structural information. Canini et al. [12] extended an online version of Latent Dirichlet Allocation (LDA) and proposed an incremental Gibbs sampler for LDA (here referred to as I-LDA). In online-LDA and I-LDA, the number of categories is fixed, while in our approach the number of categories is growing. Kasaei et al. [5] proposed an open-ended object category learning approach just by learning specific topics per category, while our approach does not only learn a set of general topics for basic-level categorization, but also learn a category-specific dictionary for fine-grained categorization.

III Learning generic and category-specific representations

We organized our approach in three main phases: (i) feature extraction; (ii) learning generic representations; and (iii) learning category-specific representations. In the following we describe each of these phases in detail.

III-A Feature Extraction

In this work, we first represent a 3D object by a set of local shape features called spin-image [13]. The reason why we use spin-image rather than other 3D feature descriptors is that the spin-image is a pose-invariant feature, and therefore suitable for 3D perception in autonomous robots. Another advantage of the spin-image is that it only requires a surface normal - rather than a full reference frame - to compute the feature. As depicted in Fig. 2, the process of local feature extraction consists of two main phases: extraction of key points and computation of spin-images. For efficiency reasons, the number of key points in an object should be much smaller than the total number of points. Therefore, the object is first voxelized and then, the nearest neighbor point to each voxel center is selected as a key point (Fig. 2 a and b). Afterwards, the spin-image descriptor is used to encode the surrounding shape in each keypoint using the original point cloud. A spin-image is a local shape histogram, which is obtained by spinning a 2D histogram around the key point’s surface normal (see Fig. 2 c and d). Therefore, each object view is described by a set of spin-images, $\textbf{O}=\{\textbf{s}_{1},~{}\dots,~{}\textbf{s}_{N}\}$, where N is the number of key points. The obtained representation is then used as input to both generic and category-specific object representation processes.

III-B Generic Representation

Our generic object representation approach requires a dictionary with $V$ visual words. Usually, the dictionary is created via off-line clustering of training data. In this work, we create a pool of objects using 75% of the training data. Previously, we showed how a robot could create a pool of objects by exploring the environment [5]. To construct a pool of features, spin-images are computed for the selected points extracted from the pool of objects. Finally, the dictionary is constructed by clustering the features using the k-means algorithm [14]. The centers of $V$ computed clusters are defined as the visual words, $\textbf{w}_{i}:i\in\{1,~{}\dots,~{}V\}$.

We then modify the incremental LDA approach [12] to be suitable for this study by releasing the batch learning phase. Particularly, we have tried to get structural semantic features from low-level feature co-occurrences by discovering a set of topics using an incremental LDA model. The basic idea is to represent a given object as a histogram of topics (i.e., latent variables), where a distribution over visual words characterizes each topic. In this method, an object is first represented as a set of visual words, $\{\textbf{s}_{1},~{}\dots,~{}\textbf{s}_{N}\}\rightarrow\{{w}_{1},{w}_{2},...,{w}_{N}\}$, where each entry represents the index of one of the $V$ words of the dictionary. Next, the object should be described as a set of topics, $\{{w}_{1},{w}_{2},...,{w}_{N}\}\rightarrow\{{z}_{1},{z}_{2},...,{z}_{N}\}$, where each element of the topic set represents the index of one of the $K$ topics. Towards this goal, the probability of topic $\textbf{z}_{j}$ being assigned to a word $\textbf{w}_{i}$, given all other topics assigned to all other words is estimated by the fast collapsed Gibbs sampler. After the sampling procedure, the word-topic matrix can be estimated as follows:

[TABLE]

where $\beta$ is Dirichlet prior hyper-parameter that affect the sparsity of distributions, $n_{\textbf{w}_{i},k}$ shows the number of times visual word $\textbf{w}_{i}$ was assigned to topic $k$ and $n_{k}$ is the number of times a word was assigned to topic $k$. $\phi$ is a $K\times V$ matrix, which represents words-probability for each topic, $K$ is the number of topics, and $\phi_{{w}_{i},k}=p(\textbf{w}_{i}|\textbf{z}_{k})$. As it is shown in Algorithm 1, we also need $n_{o,k}$ counter through the sampling procedure, which shows the number of times topic $k$ is assigned to some visual words of object O. After inferring the word-topic matrix, we generate a set of $K$ spin-image like topics:

[TABLE]

Each topic is then normalized to provide further robustness to depth variations of the object in the scene. In this way, each topic is generated by combining all visual words. It is worth mentioning that the process of topic learning does not require an explicit distance metric. The proposed procedure is summarized in Algorithm 1. At some points, a user may instruct the robot to update topics. In this case, the robot retrieves the representation of all the instances of all categories and updates dictionaries as well as the parameters of the model including $n_{k}$ and $n_{w,k}$ incrementally (i.e., unsupervised learning) using Gibbs sampling. Figure 3 shows a sample set of learned topics.

For representing a given object using the learned topics, each local feature of the object is approximated by its nearest topic. Then the object is represented as a histogram of occurrences of topics $\textbf{h}^{t}=[h_{1},~{}h_{2},~{}\dots,~{}h_{K}]$ where the $i^{th}$ element of $\textbf{h}^{t}$ is the count of the number of spin-images assigned to a topic, $\textbf{z}_{i}$.

III-C Category-Specific Representation

While the generic representation works well for basic-level classification, it does not work for fine-grained categorization. The underlying reason is that most features from fine-grained categories are similar. Therefore, these categories would share lots of similar topics, and the proportion of discriminative topics would be minor. Therefore, it is desirable to develop an approach for learning a set of category-specific features from a few examples. We have tackled this problem by learning a category-specific dictionary for each category independently. This is a challenging task since the robot does not know in advance which object categories it will have to learn, which observations will be available, and when they will be available to support learning. In our current setup, a new instance of a specific category is stored in the robot’s memory in the following situations:

• •

When the teacher for the first time teaches a particular category, through a teach action, a new category is created and initialized with the set of object views.

• •

In the case of correct actions, the local features of the target object view are added to the category.

Assume at time $t_{0}$ (i.e., first teaching action) a dictionary is learned for category $c$, denoted as $V^{c}_{t_{0}}$, which represents the distribution of 3D shape features observed up to time $t_{0}$. Later at time $t_{1}$, a new training instance, which is represented as a set of spin-images, is taught by a teacher to category $c$ (i.e., supervised learning). The teaching instruction trigs the robot to retrieve the current dictionary of the category as well as the representation of the new object view and updates the relevant dictionary using an incremental K-means algorithm [14] (i.e. unsupervised learning). Such category-specific dictionary would highlight the differences of objects from different categories, and as a consequence improves the object recognition performance.

Similar to the generic representation, a given object is then represented as a histogram of occurrences of visual words, $\textbf{h}^{c}=[h_{1},~{}h_{2},~{}\dots,~{}h_{V^{c}}]$, where $V^{c}$ is the size of category-specific dictionary. The obtained histograms, $\textbf{h}^{t}$ and $\textbf{h}^{c}$, are then concatenated to form a single representation for the given object. Figure 4 shows ten sample visual words from a Mug category-specific dictionary. It is worth mentioning when dictionaries or topics are updated, the representations of known instances must be updated. At the moment, we do this by storing the features of each object view and using them to recompute the representations.

IV Object category learning and recognition

Concerning category formation, we use the instance-based learning and recognition (IBL) approach which considers category learning as a process of learning about the instances of the category, i.e., a category is represented simply by a set of known instances. IBL is a baseline approach to evaluate object representations. An advantage of the IBL approaches is that they can recognize objects using a very small number of experiments and the training phase is very fast. As discussed in the previous section, the teach and correct actions by the teacher lead the robot to create a new category or to modify an existing one (i.e., supervised learning). Whenever a new object is added to a category, the agent retrieves the current model of the category and updates the category model by storing the representation of new object views. In particular, our approach can be seen as a combination of a particular object representation, similarity measure and classification rule. Therefore, the choice of the similarity metric has an impact on the recognition performance.

With regards to the similarity measure, since the proposed object representation describes an object as a histogram, the dissimilarity between two histograms can be computed by different distance functions. We refer the reader to a comprehensive survey on distance/similarity measures provided by Cha [15]. After performing several cross-validation experiments, we concluded that two types of distance functions including Jensen-Shannon (JS) and chi-squared ($\chi^{2}$) distances are suitable to estimate the similarity between two instances. Both functions are in the form of a bin-to-bin distance function. Although the practical results of $\chi^{2}$ and JS are almost identical, $\chi^{2}$ is computationally more efficient. Therefore, we use the $\chi^{2}$ function to estimate the similarity of two instances. Mathematically, let $\operatorname{P}$ and $\operatorname{Q}$ $\in{\rm I\!R^{K+V^{c}}}$ be the representation of two objects:

[TABLE]

To assess the dissimilarity between a target object and stored instances of a certain category C, the target object should be first described by the general topics and the learned dictionary of the category C. Afterwards, the minimum distance between the target object and all stored instances of the category C is considered as the Object-Category-Distance (OCD). The target object is finally classified based on the minimum OCD.

V Result and discussion

Three types of experiments were carried out to evaluate the proposed approach. In this section, we first explain our experimental setup and then discuss the obtained results.

V-A Datasets and Baselines

The experimental evaluations were carried out on two 3D object datasets including the Restaurant Object dataset [6] and the Washington RGB-D Object dataset [16]. For the classical evaluation (i.e., 10-fold cross-validation), we mainly use the Restaurant Object Dataset since it has a small number of classes (10 categories) with significant intra-class variation that is suitable for performing extensive sets of experiments. The Washington RGB-D Object dataset [16] is used for open-ended evaluation. It consists of 250,000 views of 300 common household objects taken from multiple views and organized into 51 categories. We also report on a real-world demonstration using the Imperial College Domestic Environment Dataset [17]. This is a suitable dataset for this test since all scenes were captured under various clutter and contain objects with similar shapes (lipton vs. softkings) and objects with very different shapes (oreo vs. elite).

For comparison, we have selected four open-ended 3D object category learning and recognition approaches, including the RACE [1], BoW [18] based on the nearest neighbour classification rule, and Open-Ended LDA, which is a modified version of the standard smoothed LDA [19] and Local-LDA [5]. Moreover, we add another baseline, which is the proposed method without category-specific representation (here referred to as Generic Rep.).

V-B Classical Evaluation using Restaurant Object Dataset

The proposed approach has a set of parameters including $\langle\alpha$, $\beta$, G, VS, IW, SL, V, K and V${}^{c}\rangle$, that must be tuned to provide a good balance between recognition performance, memory usage, and processing speed. In this work, we assumed a symmetric Dirichlet prior for both $\alpha$ and $\beta$ parameters. Therefore, a high $\alpha$ value means that each object is likely to contain a mixture of most of the topics, and not a single specific topic. Likewise, a low $\beta$ value means that a topic may contain a mixture of just a few of the words. We carried out several experiments and concluded that $\alpha$ and $\beta$ should be set to 1.0 and 0.1 respectively. The number of Gibbs sampling iterations, G, is set to 50. A set of 900 experiments was performed for different values of the remaining six parameters. The voxel size (VS) parameter is related to the number of keypoints extracted from each object view. IW defines the size of the spin-image descriptor, which will be $(\operatorname{IW}+1)\times(2\times\operatorname{IW}+1)$ float (4 bytes). Support length (SL) determines the amount of space swept out by a spin-image. A summary of the experiments using Restaurant Object dataset [6] is reported in Table I. The parameter configuration that obtained the best average accuracy was selected as the default configuration (i.e., bold numbers). The accuracy of this configuration was 94$\%$. A complete experiment (including both learning and recognition phases) on average took 286.50 seconds.

V-B1 Comparative evaluation

Table II presents a summary of the obtained results. One important observation is that the overall performance of our approach is promising and the proposed representation is capable of providing a distinctive representation for the objects. Moreover, it was observed that the discriminative power of our approach was better than the other evaluated approaches which was 7 percentage points (p.p.) better than RACE, 5 p.p. better than BoW, 11 p.p. and 3 p.p. better than Open-Ended LDA (shared topics) and Local LDA (category-specific topics) respectively. Furthermore, to show the importance of considering the category-specific representation, we compared our approach with the Only Generic Rep. baseline. It was observed that the accuracy has been boosted by 4 p.p. when we use both generic and category-specific representations. The accuracy of object recognition based on variable size representation (i.e., RACE) was not as good as the other approaches. The Local LDA and BoW obtained an acceptable performance. In the case of experiment times, BoW achieves the best performance and RACE was the most computationally expensive approach. Overall, topic modelling approaches achieve a medium-level

experiment time. The underlying reason is that there is a Gibbs sampling procedure in the topic modelling based approaches which takes time to accurately represent the desired distribution.

V-C Open-Ended Evaluation

An evaluation protocol for open-ended learning systems was proposed in [20]. The idea is to emulate the interactions of a robot with the surrounding environment over large periods of time. This protocol is based on a Test-then-Train scheme.We developed a simulated teacher to follow the protocol and autonomously interact with the agen. The idea is that for each newly taught category, the simulated teacher repeatedly picks unseen objects of the currently known categories from a dataset and presents them to the agent. It progressively estimates the recognition accuracy of the agent and, in case this accuracy exceeds a given threshold ($\tau$ = 0.67, meaning accuracy is at least twice the error rate), introduces an additional object category. This way, the agent is trained online, and at the same time, the accuracy of the system is continuously estimated. In case the agent can not reach the classification threshold after a certain number of iterations (i.e., 100 iterations), the simulated teacher can infer that the agent is no longer able to learn more categories and terminates the experiment. We assess our experimental results using the metrics that were recently introduced in [9], including: (i) the average number of learned categories at the end of an experiment (ALC), an indicator of how much the system is capable of learning; (ii) the number of question/correction iterations (QCI) required to learn those categories and the average number of stored instances per category (AIC), indicators of how much memory does it take for learning?; (iii) Global Classification Accuracy (GCA), an accuracy computed using all predictions in an experiment, and the Average Protocol Accuracy (APA), indicators of how well the system learns.

V-C1 Results

A detailed summary of the obtained results is reported in Table III. In all approaches, as more categories are learned, the classification accuracy first decreases (performance degradation phase), and then starts going up again as more instances are introduced (recovery phase). Eventually, the agent reaches a break-point where it is no longer able to learn more categories. By comparing all approaches, it is visible that the agent learned (on average) more categories using our approach than with other state-of-the-art approaches. The agent with Local-LDA approach obtained acceptable scalability, while the scalability of the other approaches was very low and their performance drops aggressively when the number of categories increases. In particular, our approach on average learned around $11$ categories more than Local-LDA and $18$, $20$ and $25$ categories more than BoW, RACE and open-ended LDA approaches, respectively. It is also clear that the agent with our approach stored more instances per category (AIC) than the other approaches. This is expected since a higher number of categories known by the system tends to make the classification task more difficult.

Although on average, BoW and Local-LDA achieved better performance than our approach, the difference is minor, and the discriminative power of those approaches is lower. This is expected since the BoW and Local-LDA learned fewer categories, and it is easier to get better classification performance in fewer categories.

V-D System Demonstration

In this demonstration, the system initially had no prior knowledge, and all objects are recognized as Unknown. Later, a user interacts with the system and teaches all object categories including amita, colgate, lipton, elite, oreo and softkings to the system using the objects extracted from a scene captured from the blue camera pose as shown in Fig. 5 (left). The system conceptualizes those categories using the extracted object views. Afterwards, the system is tested by the remaining 12 scenes captured from different viewpoints (i.e., shown by red cameras). The system could recognize all objects properly by using the knowledge learned from the first scene. Some misclassifications also occurred throughout the demonstration. The underlying reason was that, at some points, the object tracking could not track the object properly and the distinctive parts of the object were not included in the object’s point cloud. This evaluation illustrates the process of learning object categories in an open-ended fashion. A video of this demonstration is available online at: https://youtu.be/zjucGaAwnTE

VI Conclusion

In this paper, we have tackled the problem of open-ended object category learning and recognition by proposing a new object representation, which is suitable for both fine-grained and basic-level object categorization. In particular, we described each object based on a set of general latent visual topics and a set of category-specific dictionaries. An extensive set of experiments was carried out to assess the performance of the proposed approach. Experimental results show that the overall classification performance obtained with the proposed approach is clearly better than the best accomplishments achieved with the state-of-the-art methods. Moreover, the best scalability was obtained with the proposed approach, followed by the Local-LDA [5]. Concerning computational time, the best result was obtained with BoW [18], immediately followed by the Open-Ended LDA approach [19]. A real demonstration proved that the agent could learn new categories from very few examples in an incremental and open-ended manner.