- Chief Technology Officer, Common Crawl13
Greg Lindahl is the Chief Technology Officer of the Common Crawl Foundation, the nonprofit that crawls the web and makes its archives and datasets freely available. Lindahl is the author of several of Common Crawl's technical posts, including the announcement of the Host Index dataset and an explainer on measuring how much of a website the crawl has covered.1234
At a glance
- Current role
- Chief Technology Officer
- Company
- Common Crawl
- Sector
- Nonprofit
- Fields
- Datasets and data
- Work listed
- 2
- Sources cited
- 6
- Profile reviewed
Common Crawl
Common Crawl's team page lists Lindahl as CTO, alongside Executive Director Rich Skrenta. The foundation's crawler, CCBot, is based on Apache Nutch, and its data is released in WARC, WAT and WET formats together with URL and CDXJ indexes and host- and domain-level web graphs.12
Datasets and writing
In April 2025 Lindahl announced a public test of the Host Index, which gathers per-host summaries from the crawl, its indexes, the web graph and raw crawler logs, and can be queried with SQL tools such as Amazon Athena or DuckDB. A July 2026 post by Lindahl explains why counting a site's pages in Common Crawl is harder than it looks, using CrawlDb states from the Nutch crawler.34
Lindahl also compiled the foundation's December 2025 roundup of a dozen research papers from that year that use or reference Common Crawl data.5
Work and contributions
Common Crawl Host Index Open data
Authored the April 2025 announcement of the Host Index, a dataset with one row per web host per crawl that combines crawl statistics, status codes, language summaries and robots.txt data, with tools published in the cc-host-index repository.36
Common Crawl corpus Open data
As CTO, works on the technical side of the open repository of web crawl data that Common Crawl has collected since 2008 and publishes roughly once a month.12
Timeline
Where they work
- Common Crawl 3 people in the wiki
Related people
- Rich SkrentaAlso at Common Crawl
- Gil ElbazAlso at Common Crawl
- Sridhar RamaswamyDatasets and data
- Nikhil ThoratDatasets and data
- Andy KonwinskiDatasets and data
- Alexander ToshevDatasets and data
- Reynold XinDatasets and data
- Connor LeahyDatasets and data
- Yacine JerniteDatasets and data
- Brandon DuderstadtDatasets and data
Suggest a correction or removal
Profiles cover public professional work only, and each statement cites a source. If something is out of date, missing or wrong, tell us and link a source. People can also ask for their profile to be corrected or removed.
References
6 sources · profile reviewed Oct 21, 2026 · data updated Oct 21, 2026