Accurate Estimators for Improving Minwise Hashing and b-Bit Minwise Hashing
Ping Li, Christian Konig

TL;DR
This paper introduces improved estimators for minwise hashing and b-bit minwise hashing, enhancing accuracy especially for low resemblance and high containment set pairs in search and database applications.
Contribution
The paper presents systematic improvements to existing minwise hashing estimators, particularly benefiting applications involving low similarity and high containment scenarios.
Findings
Improved estimators significantly reduce bias in low resemblance cases.
Enhanced accuracy for high containment set comparisons.
Method outperforms existing estimators in practical scenarios.
Abstract
Minwise hashing is the standard technique in the context of search and databases for efficiently estimating set (e.g., high-dimensional 0/1 vector) similarities. Recently, b-bit minwise hashing was proposed which significantly improves upon the original minwise hashing in practice by storing only the lowest b bits of each hashed value, as opposed to using 64 bits. b-bit hashing is particularly effective in applications which mainly concern sets of high similarities (e.g., the resemblance >0.5). However, there are other important applications in which not just pairs of high similarities matter. For example, many learning algorithms require all pairwise similarities and it is expected that only a small fraction of the pairs are similar. Furthermore, many applications care more about containment (e.g., how much one object is contained by another object) than the resemblance. In this paper,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSpam and Phishing Detection · Advanced Image and Video Retrieval Techniques · Text and Document Classification Technologies
