Technical SEO for AI: Robots.txt, GPTBot & llms.txt Explained | 3.4. AEO Course by Ahrefs

Hey, it's Sam Oh, and welcome to the fourth lesson in this module, which is on the technical side of AEO. Now, I know technical can sound intimidating, but this lesson isn't about rewriting your site's code. It's about making sure that AI can actually access and understand your content. And while access and understand might sound

rudimentary for some of you, the reality is a lot of sites are accidentally blocking AI without even knowing it. According to our data, around 5.9% of 140 million websites are blocking GPTBot, which is Open AI's crawler. That's millions of sites that are invisible to ChatGPT. So, in this lesson, I've got six technical checks

and tips for you to make sure AI can find you so that it can promote you. Let's get started. So, the first thing you need to check is your robots.txt file. Robots.txt is a file on your site that tells crawlers what they can and can't access. And the thing is, it's not just Google's crawler you need to think

about anymore. There are now dozens of AI-specific bots that crawl the web. The main ones you should know about are GPTBot and OAI search bot from Open AI, ClaudeBot from Anthropic, and Google extended from Google. If any of these are blocked in your robots.txt, you're asking those AI platforms not to crawl your content. And assuming they

obey your rules, they sure won't be recommending your pages then if they don't know what's on them. Now, you might not have blocked these bots intentionally, but a lot of sites inherit robots.txt rules from templates or old configurations. And some platforms add blocks by default. For example, Cloudflare has a feature called instruct AI bot traffic with robots.txt.