Normalized Web Distance and Word Similarity

TL;DR

This paper introduces the normalized web distance (NWD), a method leveraging web page counts from search engines to measure semantic similarity between words and phrases, replacing the earlier NGD approach.

Contribution

The paper presents the NWD as a novel, practical approach for quantifying semantic similarity using publicly available web data, adapting from the previous NGD method.

Findings

NWD effectively measures word similarity based on web data.

NWD replaces NGD due to search engine restrictions.

The method is applicable to various linguistic and cognitive tasks.

Abstract

There is a great deal of work in cognitive psychology, linguistics, and computer science, about using word (or phrase) frequencies in context in text corpora to develop measures for word similarity or word association, going back to at least the 1960s. The goal of this chapter is to introduce the normalizedis a general way to tap the amorphous low-grade knowledge available for free on the Internet, typed in by local users aiming at personal gratification of diverse objectives, and yet globally achieving what is effectively the largest semantic electronic database in the world. Moreover, this database is available for all by using any search engine that can return aggregate page-count estimates for a large range of search-queries. In the paper introducing the NWD it was called `normalized Google distance (NGD),' but since Google doesn't allow computer searches anymore, we opt for the more neutral and descriptive NWD. web distance (NWD) method to determine similarity between words and phrases. It