This Is a Local Domain: On Amassing Country-Code Top-Level Domains from Public Data
Raffaele Sommese, Roland van Rijswijk-Deij, Mattijs Jonker

TL;DR
This paper investigates how public data sources like Certificate Transparency logs and Common Crawl can be used to estimate the presence of country-code top-level domains (ccTLDs) on the Web, providing a proxy for more comprehensive Web censuses.
Contribution
It demonstrates that public data can effectively approximate ccTLD coverage, revealing significant Web presence without direct access to domain registries.
Findings
Public data covers 43%-80% of ccTLDs
Coverage of ccTLDs increases over time
Public sources reveal substantial Web presence under ccTLDs
Abstract
Domain lists are a key ingredient for representative censuses of the Web. Unfortunately, such censuses typically lack a view on domains under country-code top-level domains (ccTLDs). This introduces unwanted bias: many countries have a rich local Web that remains hidden if their ccTLDs are not considered. The reason ccTLDs are rarely considered is that gaining access -- if possible at all -- is often laborious. To tackle this, we ask: what can we learn about ccTLDs from public sources? We extract domain names under ccTLDs from 6 years of public data from Certificate Transparency logs and Common Crawl. We compare this against ground truth for 19 ccTLDs for which we have the full DNS zone. We find that public data covers 43%-80% of these ccTLDs, and that coverage grows over time. By also comparing port scan data we then show that these public sources reveal a significant part of the Web…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsData Quality and Management · Internet Traffic Analysis and Secure E-voting
