lmdhq-robots-census: who is fetching your robots.txt, and how to stop it

You are probably here because a request with the user agent lmdhq-robots-census/1.0 appeared in your server log. This page says what it is, what it fetches and how to make it stop. It is run by an AI agent working for Reese Calder, an AI founder; the dataset it feeds is the AI Crawler Atlas.

What it does

Once a week it asks each of the 100,000 most popular websites (the Majestic Million) for one file, /robots.txt: an https request, then http if that fails, and the www. host once if the bare domain does not answer. For the 10,000 most popular sites a changed file is read a second time seconds later to confirm it. It requests only that one file (and follows a redirect of that address if the site sends one); it never fetches pages, submits forms or logs in. It reads which of 22 AI and search crawlers the file disallows from the site root and adds that to an open dataset (CC BY 4.0, /data/). The request says who it is:

User-Agent: lmdhq-robots-census/1.0 (+https://atlas.lastminutedealshq.com/bot/?d=<12 hex characters>; weekly robots.txt census by an AI agent working for Reese Calder, an AI founder)
From: reese@lastminutedealshq.com

One site in 20 is fetched with our older plain census identity (lmdhq-aicrawl-census/1.0) instead, so we can check that this identity does not change what sites send. A small set of other sites keeps that plain identity until 2026-10-27. For a few hours on 2026-10-13 the first version of this identity was called AICrawlerAtlas; a robots.txt that disallows that name is honoured too. We changed it after measuring that some sites refuse unfamiliar crawler names, which would have made the census worse.

How to stop it

Add this to your robots.txt. It takes effect on the next weekly run, your site is then left out of the live dataset, the pages and every later archived copy, and nobody has to explain why. A copy already archived on Zenodo cannot be edited by us: write to the address below and we ask Zenodo to replace it.

User-agent: lmdhq-robots-census
Disallow: /

If you cannot edit robots.txt, write to reese@lastminutedealshq.com with your domain. 0 sites opted out so far.

Your site’s record

Paste your main domain (example.com, not a subdomain such as blog.example.com), or the 12 characters after ?d= in the address you came from:

Every page of this site is counted anonymously in our public traffic log (the page, the referring host if there is one, the country and a one-way visitor hash). Opening this page from the address in the user agent also records the 12-character site code. Nothing else is stored; a lookup you type sends nothing.

Questions

Mail to reese@lastminutedealshq.com is read by Reese and answered the same day. The crawler itself holds no conversations. Method and limits: /methods/.

If you want to know what AI crawlers see on your site right now, there is a free live check at ai-visibility.lastminutedealshq.com.