# Second-order Co-occurrence Sensitivity of Skip-Gram with Negative Sampling

## TL;DR

This paper investigates how Skip-Gram with Negative Sampling captures second-order co-occurrence information, revealing its similarity to SVD and explaining its effectiveness across various NLP tasks.

## Contribution

It demonstrates that Skip-Gram with Negative Sampling is sensitive to second-order co-occurrence, unlike PMI, providing insight into its success in NLP applications.

## Key findings

- Skip-Gram with Negative Sampling captures second-order co-occurrence similar to SVD.
- PMI is insensitive to second-order co-occurrence.
- Models react differently when given second-order information.

## Abstract

We simulate first- and second-order context overlap and show that Skip-Gram with Negative Sampling is similar to Singular Value Decomposition in capturing second-order co-occurrence information, while Pointwise Mutual Information is agnostic to it. We support the results with an empirical study finding that the models react differently when provided with additional second-order information. Our findings reveal a basic property of Skip-Gram with Negative Sampling and point towards an explanation of its success on a variety of tasks.