Websites that block OAI-SearchBot
4,048 of the 67,673 readable sites in the census (6.0%) disallow OAI-SearchBot (OpenAI, search) from the site root in robots.txt. These feed AI answer engines that cite and link their sources. Disallowing them can take a site out of AI answers and out of the referral traffic that comes with them.
| Site | Popularity | robots.txt | Read on |
|---|---|---|---|
| letmejerk.com | Top 100,000 | 12 disallowed | 2026-10-13 |
| jqwidgets.com | Top 100,000 | 5 disallowed | 2026-10-13 |
| wfxrtv.com | Top 100,000 | 13 disallowed | 2026-10-13 |
| new.fr | Top 100,000 | 20 disallowed | 2026-10-13 |
The list shows the sites that have a page here, most popular first. This reports what the file asks. It does not show whether a crawler can actually reach the site: many sites also block crawlers at their CDN or firewall, and some crawlers ignore robots.txt.