Learning Good Representation via Continuous Attention

TL;DR

This paper introduces a novel approach where continuous attention driven by intrinsic rewards from unsupervised learning enhances representation learning, improving object recognition in reinforcement learning agents.

Contribution

The paper proposes a new method combining unsupervised learning and reinforcement learning with intrinsic rewards to improve object representation learning.

Findings

Effective in few-shot object recognition tasks

Improves representation quality via continuous attention

Works with and without extrinsic rewards

Abstract

In this paper we present our scientific discovery that good representation can be learned via continuous attention during the interaction between Unsupervised Learning(UL) and Reinforcement Learning(RL) modules driven by intrinsic motivation. Specifically, we designed intrinsic rewards generated from UL modules for driving the RL agent to focus on objects for a period of time and to learn good representations of objects for later object recognition task. We evaluate our proposed algorithm in both with and without extrinsic reward settings. Experiments with end-to-end training in simulated environments with applications to few-shot object recognition demonstrated the effectiveness of the proposed algorithm.

Tables

Table 1. Table 1 : Results on object attention task

RL Policy	Random	Autoencoder	Prediction
Frequency (%)	48$\pm$6	51$\pm$10	51$\pm$8
Duration	7.1 $\pm$ 4.4	9.6 $\pm$ 5.4	12.1 $\pm$ 11.8

Table 2. Table 2 : Object recognition errors by AE and prediction modules

5-way-n-shot	1	3	5	7
AE	27.0%	11.3%	11.0%	7.0%
Prediction	36.0%	28.9%	18.2%	14.0%

Equations

$$r_{ae}=L_{ae}=||s_t-{\hat{s}}_t|{|}_2$$

$$r_{pred}=L_{pred}=||\varphi (s_{t+1})-\hat{\varphi }(s_{t+1})|{|}_2$$

Full text

1 Introduction

In Artificial Intelligence(AI), our ultimate goal is to develop intelligent machines and robots that could automatically acquire skills and knowledge under the guidance of intrinsic motivation [21] before given any specific tasks. The learned skills and knowledge will help improve the performance of downstream tasks such as recognition, detection, navigation, and planning. In this paper, we focus on studying how to learn good representation or features of objects in a simulated 3D environment driven by intrinsic motivation.

Intrinsic motivation plays an important role in human development and learning, and researchers in many areas of cognitive science have emphasized that intrinsically motivated behavior is vital for intellectual growth. Inspired by human infants that exhibit strong intrinsic motivation and self-supervised capability to interact with the objects in the environment, in this paper we propose a novel setting with deep neural networks for an agent to learn representations of objects in a 3D environment driven by the interaction between unsupervised learning and reinforcement learning modules. The RL module is responsible for collecting data from the environment and the UL module is responsible for extracting good features from the collected data and meanwhile for providing intrinsic rewards for the agent to explore the environment. The first skill for the agent to acquire is to learn a policy to attend to objects. Our observation is that random exploration cannot achieve frequent continuous object attention or continuous focusing on an object (as shown in Figure 1) which is import for an agent to do online learning of the representation of the environment and objects.

This work contrasts with traditional passive unsupervised learning. Instead of learning representation of the given input, the agent actively choose attention direction and training data for learning meaningful/informative features as demonstrated in the experiment section. We propose an alternative to unsupervised learning that takes advantage of active control where to collect data in the environment via an exploration policy.

The main contribution is our scientific discovery that good representation can be learned via continuous attention during the interaction between Unsupervised Learning(UL) and Reinforcement Learning(RL) modules driven by intrinsic motivation. Specifically, we designed intrinsic rewards generated from UL modules for driving the RL agent to focus on objects for a period of time and to learn good representations of objects for later object recognition task.

We evaluate our proposed algorithm in both with and without extrinsic reward settings. Experiments with end-to-end training in simulated environments with applications to few-shot object recognition demonstrated the effectiveness of the proposed algorithm.

The rest of the paper is organized as follows. We briefly review related work in Section 2. The detailed description of the proposed deep neural network architecture and algorithm are given in Sections 3 and 4, respectively, followed by experimental results in Section 5 and conclusion in Section 6.

2 Related Works

Our work is closely related to intrinsic motivation, representation learning and their application to few-shot object recognition. Here, we briefly review some related work in these areas.

**Representation Learning: ** Learning good data representations (or features) is one of the main goals for any machine learning algorithms –no matter whether it is a supervised or unsupervised one. For unsupervised learning, we want the learned representations of the data makes it easier to build classifiers or other predictors upon them. The question is how to evaluate the quality of learned representations. Goodfellow et al. [6] proposed a number of empirical tests for directly measuring the degree to which learned representations are invariant to different transformations. Higgins et al. [7] devised a protocol to quantify the degree of disentanglement learned by different models. Bengio et al. [4] list some properties/attributes a good representation should possess, such as sparsity, distributed among multiple explanatory factors, hierarchical organization with more abstract and invariant concepts [9, 28] higher in the hierarchy, temporal and spatial coherence. The listed properties provide us guide to design appropriate deep neural network architecture and algorithm for learning such representations.

**Intrinsic Motivation: ** The goal of RL agents is to maximize discounted cumulative reward [24, 16, 14]. Reward functions should be defined generically and lead to good long-term outcomes for an agent. However, most RL algorithms suffer from sparse rewards. Reward shaping [18] provides a way to deal with sparse extrinsic rewards; anther way is intrinsic motivation [21]. Intrinsic rewards such as prediction gain [3], learning progress [19], compression progress [22], variational information maximization [8, 26], have been employed to augment the environment’s reward signal for encouraging to discover novel behavior patterns.

**UL+RL: ** Combining UL and RL modules have been proposed in previous work. Most work [23, 10, 20] focus on improving reinforcement learning performance using unsupervised learning tasks to achieve sample efficiency with higher return. In contrast, our work focus on learning good representation of objects via continuous attention driven by interaction between UL and RL modules. The learned RL exploration policy helps the UL module learn better representation (features) of objects by providing informative data to the UL module.

**Attention: ** Attention mechanism has been proposed to help visualize, interpret the inner working of deep neural networks and to improve the performance of many machine learning tasks such as machine translation [2, 27], image classification [15, 5, 11, 12], visual question answering [29, 31, 1], and image captioning [30, 32, 17]. Unlike these previous work whose goal is to estimate a 2D spatial or 1D temporal attention map, our work focus on training an agent to attend to objects in a 3D environment for a period of time in order to learn a good representation of the object.

3 Deep Neural Network Architecture

See Figure 2 for the general framework which includes a RL and a UL modules: UL module provides intrinsic rewards to the RL module while RL module provides training data to the UL module.

**RL module architecture: ** The RL module in Figure 2 follows the A3C architecture [14]. First, the input state $s_{t}$ is passed through a sequence of four convolution layers with 32 filters each, kernel size of 3x3, stride of 2 and padding of 1. A RELU activation unit is used after each convolution layer. Then, two separate fully connected layers are used to predict the value function and the action from the feature representation of the forth convolution layer.

Figure 3 shows two designs of the UL module: autoencoder and prediction modules.

**UL autoencoder architecture: ** In Figure 3(a), the encoder is a sequence of three convolution layers with 16, 32, 64 filters, respectively, kernel size of 4x4, stride of 4 and padding of 0. The decoder is a sequence of three deconvolution layers corresponding to three convolution layers in the encoder. The reconstruction error is used as intrinsic rewards to the RL module.

**UL prediction architecture: ** As shown in Figure 3(b), the prediction module follows the design in [20]. It contains a dynamic forward model and a dynamic inverse model. The dynamic forward model predicts the feature of next state given the feature of the current state and agent’s action and the prediction error is used as intrinsic reward to the RL module. The dynamic inverse model predicts the agent action that causes the transition of current state to the next state. The feature is extracted by a sequence of three convolution layers with 16, 32, 64 filters, respectively; kernel size of 6x6, 3x3, 3x3, respectively; stride of 6, 3, 3, respectively; and padding of 0.

The autoencoder focuses on learning the static features of the scene and provides the signal to make the learned representation compressed and meanwhile containing enough information to reconstruct the scene. This is complement to the signal provided by the forward dynamics model which focuses on encoding the dynamic aspect of the scene.

4 Algorithm

The algorithm includes two phases. The goal of phase one is to learn representation of objects in unsupervised setting via interaction between UL and RL modules; while the goal of phase two is to use the learned representation to improve the performance of object recognition in supervised setting. Self-designed task in phase one is to learn object representation from different points of view. Recognition training stage: provide label for image of an object from a few points of views; Recognition test stage: test recognition of an object from the rest points of views.

4.1 Intrinsic Rewards Generated by UL Modules

The core of learning (both for human and machine) is to learn from mistake and to learn from failure. We use the following two loss functions as intrinsic rewards for autoencoder and prediction modules, respectively. These intrinsic rewards drive the agent to explore areas where there are large reconstruction and prediction errors and therefore provide the UL module more diverse and informative data.

[TABLE]

[TABLE]

5 Experiments

5.1 Experimental Setting

Environments The first environment we evaluate on is XWorld2D 111https://github.com/PaddlePaddle/XWorld/games/xworld with navigation task which is used to demonstrate that auxiliary unsupervised learning modules improve policy learning when there are only sparse extrinsic rewards. Figure 4(a) illustrates a snapshot of XWorld2D, which is a 2D world for an agent to learn vision and language skills. In this paper, we train the agent on navigation tasks given by a virtual teacher. The goal is given by a word such as “fish”. The agent should learn simultaneously the visual representation of the object and the action control that moves the agent to the goal. The agent gets a reward only when it reaches the goal. Therefore, this is a typical sparse reward reinforcement learning problem.

The second one is a simulated 3D environment – XWorld3D 222https://github.com/PaddlePaddle/XWorld/games/xworld3d which is used to demonstrate that the learned representation via interaction between UL and RL modules helps improve the performance of downstream few-shot learning task. Figure 4(b) illustrates a snapshot of XWorld3D. In this paper, we train the agent to explore the 3D world without given it any tasks in phase one. The agent sees the world in a first-person view, learns simultaneously the visual representations of the objects, and how to explore the world driven by intrinsic motivation. Therefore, this is a typical reinforcement learning problem without extrinsic reward which mimics how a baby learns a model of the environment without any guide.

Training details All agents are trained using visual inputs. The input RGB images are resized to 64x64. Following the asynchronous training protocol in A3C, all the agents were trained asynchronously with sixteen workers using stochastic descent. We used RMSprop [25] optimizer with a learning rate of $10^{-5}$ , a damping factor of 0.01, and a gradient moving average decay of 0.95. The gradient has a momentum of 0.9. The batch size is set to 32. The training details described in this section apply to all the methods in both environments.

Baseline method “a3c + ae” and “a3c + pred” denote two “UL+RL” algorithms. We compare our algorithms with a baseline method where A3C module is replaced with a random policy which generates actions by uniformly sampling from a set of actions (move_forward, move_backward, move_left, move_right, turn_left, and turn_right).

5.2 Results with Extrinsic Rewards

Figure 4(a) shows the XWorld2D simulation environment which we used to conduct experiments with extrinsic reward.

Figure 5 illustrates the average rewards obtained by the agent in XWorld2d navigation tasks with and without intrinsic rewards. The results demonstrate that auxiliary UL module improve RL policy learning for achieving higher average reward when there are only sparse extrinsic rewards.

5.3 Interaction between UL and RL Modules

Figure 6 shows the XWorld3D simulation environment which we used to conduct experiments without extrinsic reward.

Figure 7 illustrates how the intrinsic rewards or unsupervised learning task performance changes as the reinforcement learning policy improves. For autoencoder (Figure 7(a)), the reconstruction error is large at first and then drops to a low level and stays low, which means that the RL exploration policy samples data that cover the whole environment for training the autoencoder so that it learns how to encode and decode the state of the environment in an efficient way.

For dynamic forward module (Figure 7(b)), the agent first approach one object (a cat) as shown in Figure 6(a) and learns how to predict the feature of the next state and the prediction error drops once the forward module learns how to predict the state of the object (the cat) in Figure 6(a). The intrinsic reward drives the agent to approach to another new object (a dog) in the environment (Figure 6(b)(c)) and makes the prediction error increase again as shown in Figure 7(b).

Dynamic inverse prediction is a relatively easier task comparing with dynamic forward prediction – it only needs to learn how to predict self-motion given the features of the previous and current states. Therefore, once the module learns how to do it the inverse prediction error drops and stays at a low level as shown in Figure 7(c).

5.4 Object Attention Using Different RL Policies

The first skill for the agent to acquire is to learn a policy to attend to objects. Our observation is that random exploration cannot achieve continuous and frequent object attention. See Table 1 for a list of the mean and standard deviation of object attention frequencies and durations for three policies – random exploration, autoencoder, and prediction modules. The object attention frequency is calculated as the ratio between the total number of frames with an object present and the total number of frames within a time window; the object attention duration is calculated as the number of consecutive frames with the same object present. Table 1 demonstrates that the policies driven by both UL modules achieves higher object attention frequency (51%) and longer continuous focus on the same object (9.6 and 12.1 frames, respectively) than random exploration policy does (frequency: 48% and duration: 7.1 frames).

As shown in Fig 7(b), the error increase corresponds to emergence of object attention. Both autoencoder and prediction modules exhibit increased frequency of consecutive frames with an object present coinciding with the increase in intrinsic rewards or reconstruction and forward prediction errors, respective ly.

The RL agents driven by UL modules tend to attend to where they are going and more informative directions, for example they look straight ahead to an object when moving forward while agents trained with random exploration sometimes look in less informative directions (e.g. looking at the empty scene without any object present).

5.5 Qualitative Evaluation of Learned Features

We qualitatively evaluate the arrangement of learned features using t-SNE representations [13]. Figure 8 visualize the features learned in phase one by autoencoder and prediction modules, respectively. It demonstrates that both UL modules can learn discriminative features for grouping objects of the same category into the same cluster.

5.6 Quantitative Evaluation of Learned Features

Figure 9 shows some snapshots of the XWorld3D environment for object recognition task. In the training stage, we provide labels for images of an object from a few randomly sampled points of views, while in the test stage, we evaluate recognition performance on the object from the rest points of views.

As a proof of concept of the idea, Table 2 presents the object recognition results by two UL modules using k-nearest neighbors (KNN). The recognition errors decrease as the training data increase and achieves 7.0% and 14.0% for AE and prediction module, respectively, when the number of training data gets 7. The reason that AE module achieves lower recognition error than prediction module is probably that AE module converge to a lower reconstruction error than prediction error by prediction module within the same number of iterations (450k) as demonstrated in Fig 7(a)(b).

6 Conclusion and Future Work

In this paper we present our scientific discovery that good representation can be learned via continuous attention during the interaction between Unsupervised Learning and Reinforcement Learning modules driven by intrinsic motivation. Specifically, we designed intrinsic rewards generated from UL modules for driving the RL agent to focus on objects for a period of time and to learn good representations of objects for later object recognition task. We evaluate our proposed algorithm in both with and without extrinsic reward settings. Experiments with end-to-end training in simulated environments with applications to few-shot object recognition demonstrated the effectiveness of the proposed algorithm. From the experiments we observe that the policies driven by both UL modules achieves higher object attention frequency and longer continuous focus on the same object than random exploration policy does. Both UL modules can learn discriminative features for grouping objects of the same category into the same cluster.

Several parts of the presented approach could be tuned and studied in more detail. The proposed deep neural network architecture could be used in a semi-supervised setting. The DNN could be extended to incorporate other UL tasks. We leave these questions for future research. We consider the present work as initial steps on the way to making an agent perform unsupervised learning in a real environment.