TurcodexBot is the crawler behind Turcodex Website Review, a ChatGPT plugin that checks whether AI assistants can find, read and understand a business website. It visits a site only when someone asks for a review of that site. It does not crawl the web, build a search index or collect content to train AI models.
User agents
Main requests:
Mozilla/5.0 (compatible; TurcodexBot/2.0; +https://turcodex.com/website-review/bot)
Access tests (one per AI crawler, homepage only):
Mozilla/5.0 (compatible; TurcodexBot/2.0; access test as <crawler>; +https://turcodex.com/website-review/bot)
where <crawler> is OAI-SearchBot, ChatGPT-User, PerplexityBot or Claude-SearchBot. These requests name TurcodexBot and the crawler being tested. They never use another crawler's user agent, and they can't tell you whether that crawler's real traffic gets through, because real crawlers are verified by their network addresses.
What one review requests
/robots.txt- the homepage
- up to five pages linked from the site's header, menus or footer
- the sitemap, plus quick checks (usually HEAD requests) of up to ten addresses listed in it
/llms.txt, plus quick checks of up to five of its links- at most one access-test request per AI crawler above, to the homepage only, and none for a crawler that the site's robots.txt already blocks
- one load of the homepage in a hosted browser (Browserless), to compare what visitors see with what crawlers see. This request comes from our browser provider and uses a standard browser user agent.
Each review has a 45-second time limit, caps on the number of requests and limits on page size. A repeat review of the same site within 24 hours normally reuses the stored result instead of fetching the site again. Only one review of a site runs at a time, and a site gets at most six reviews a minute.
robots.txt
TurcodexBot follows the rules for TurcodexBot in your robots.txt (RFC 9309), falling back to the * group. If your homepage is disallowed, the review stops after reading robots.txt and reports that the site wasn't crawled.
To opt out completely:
User-agent: TurcodexBot
Disallow: /Where requests come from
Requests come from our hosting provider's servers in the United States (Vercel) and, for the browser load, from our browser provider (Browserless). Their network addresses aren't fixed, so please identify TurcodexBot by its user agent and opt out through robots.txt rather than by IP address.
Contact
Questions or problems with TurcodexBot: email jason@turcodex.com with your site's address. See also the privacy policy.