# PR Product: A Substitute for Inner Product in Neural Networks

## TL;DR

This paper introduces the PR Product, a novel substitute for the inner product in neural networks, which improves gradient behavior and enhances performance across various models and tasks.

## Contribution

The paper proposes the PR Product, a new vector operation that maintains forward pass consistency while improving gradient properties, applicable to multiple neural network modules.

## Key findings

- Enhanced robustness in image classification on CIFAR datasets.
- Competitive or superior performance in image captioning on MS COCO.
- Applicable to various neural network layers without altering forward propagation.

## Abstract

In this paper, we analyze the inner product of weight vector w and data vector x in neural networks from the perspective of vector orthogonal decomposition and prove that the direction gradient of w decreases with the angle between them close to 0 or {\pi}. We propose the Projection and Rejection Product (PR Product) to make the direction gradient of w independent of the angle and consistently larger than the one in standard inner product while keeping the forward propagation identical. As a reliable substitute for standard inner product, the PR Product can be applied into many existing deep learning modules, so we develop the PR Product version of fully connected layer, convolutional layer and LSTM layer. In static image classification, the experiments on CIFAR10 and CIFAR100 datasets demonstrate that the PR Product can robustly enhance the ability of various state-of-the-art classification networks. On the task of image captioning, even without any bells and whistles, our PR Product version of captioning model can compete or outperform the state-of-the-art models on MS COCO dataset. Code has been made available at:https://github.com/wzn0828/PR_Product.