# NoPPA: Non-Parametric Pairwise Attention Random Walk Model for Sentence Representation

## TL;DR

NoPPA is a non-parametric, trainless sentence embedding model that leverages pre-trained word embeddings and word frequencies, outperforming traditional bag-of-words methods and rivaling state-of-the-art non-parametric approaches.

## Contribution

This study introduces the first non-parametric attention mechanism that breaks the bag-of-words assumption for sentence representation.

## Key findings

- Outperforms all bag-of-words-based methods on eight classification tasks.
- Provides comparable or better performance than existing non-parametric methods.
- Visualizations show understanding of topics, phrases, and causalities.

## Abstract

We propose a novel non-parametric/un-trainable language model, named Non-Parametric Pairwise Attention Random Walk Model (NoPPA), to generate sentence embedding only with pre-trained word embedding and pre-counted word frequency. To the best we know, this study is the first successful attempt to break the constraint on bag-of-words assumption with a non-parametric attention mechanism. We evaluate our method on eight different downstream classification tasks. The experiment results show that NoPPA outperforms all kinds of bag-of-words-based methods in each dataset and provides a comparable or better performance than the state-of-the-art non-parametric methods on average. Furthermore, visualization supports that NoPPA can understand contextual topics, common phrases, and word causalities. Our model is available at https://github.com/JacksonWuxs/NoPPA.