Learning Features from Co-occurrences: A Theoretical Analysis

TL;DR

This paper provides a theoretical analysis of word representation methods based on co-occurrences, exploring how different context features and functions influence the effectiveness of word classification tasks.

Contribution

It offers a novel theoretical framework for understanding co-occurrence-based word representations and explains why multiple context features can improve performance.

Findings

Analyzes the impact of context features on word classification accuracy.

Explains the benefits of using multiple context features over single ones.

Provides insights into the theory of feature learning and machine learning.

Abstract

Representing a word by its co-occurrences with other words in context is an effective way to capture the meaning of the word. However, the theory behind remains a challenge. In this work, taking the example of a word classification task, we give a theoretical analysis of the approaches that represent a word X by a function f(P(C|X)), where C is a context feature, P(C|X) is the conditional probability estimated from a text corpus, and the function f maps the co-occurrence measure to a prediction score. We investigate the impact of context feature C and the function f. We also explain the reasons why using the co-occurrences with multiple context features may be better than just using a single one. In addition, some of the results shed light on the theory of feature learning and machine learning in general.