We mentioned in last newsletter that ChatGPT could now surf the web. Let’s make sure your website can be included in these searches!
Here’s what you need to know in a nutshell:
ChatGPT Search combines Bing’s and Open AI
The platform employs three distinct crawlers:
- The OAI-SearchBot serves as the primary crawler for search functionality
- ChatGPT-User handles real-time user requests and enables direct interaction with external applications
- GPTBot, manages AI model training and can be blocked without affecting search visibility
You Can Help SearchBots by Updating Robots.txt
Your website’s robots.txt (you can find this at https://yoursite.com/robots.txt) should specifically allow OAI-SearchBot. You can do this by editing your robots.txt file to add these two lines:
User-agent: OAI-SearchBot
Allow: *
You can then test your robots.txt file using Google Search Console
How to Know if ChatGPT Visited Your Website
If you are running Google Analytics 4 (GA4), you can look at referral traffic (this link will show you all referrers, not just AI bots).
I prefer to build an exploration in GA4 that is filtered for the 3 chatbots that have visited my site. Note that as more chatbots visit, they will not be captured in this particular exploration and I would need to tweak it.
If you’d like to tackle this on your own, this link should get you to explorations where you’d have to create a new one. Below are the dimensions and metrics I used:
The regex in the screenshot above is below for easy copying:
(?i)\b(?:\S+\.)?(chatgpt|gemini|perplexity)(?:\.\S+)?\b
All of this is just to ensure your content is indexable – next issue we will discuss how to make sure your content quality is worthy of being shown!
