We mentioned in last newsletter that ChatGPT could now surf the web. Let’s make sure your website can be included in these searches!
Here’s what you need to know in a nutshell:
ChatGPT Search combines Bing’s and Open AI
The platform employs three distinct crawlers:
- The OAI-SearchBot serves as the primary crawler for search functionality
- ChatGPT-User handles real-time user requests and enables direct interaction with external applications
- GPTBot, manages AI model training and can be blocked without affecting search visibility
You Can Help SearchBots by Updating Robots.txt
Your website’s robots.txt (you can find this at https://yoursite.com/robots.txt) should specifically allow OAI-SearchBot. You can do this by editing your robots.txt file to add these two lines:
User-agent: OAI-SearchBot
Allow: *
You can then test your robots.txt file using Google Search Console
How to Know if ChatGPT Visited Your Website
If you are running Google Analytics 4 (GA4), you can look at referral traffic (this link will show you all referrers, not just AI bots).
I prefer to build an exploration in GA4 that is filtered for the 3 chatbots that have visited my site. Note that as more chatbots visit, they will not be captured in this particular exploration and I would need to tweak it.
If you’d like to tackle this on your own, this link should get you to explorations where you’d have to create a new one. Below are the dimensions and metrics I used:
The regex in the screenshot above is below for easy copying:
(?i)\b(?:\S+\.)?(chatgpt|gemini|perplexity)(?:\.\S+)?\b
All of this is just to ensure your content is indexable – next issue we will discuss how to make sure your content quality is worthy of being shown!
- About the Author
- Latest Posts
Darlene is a search marketer, analyst, speaker, trainer, and owner of DriveTraffic Digital Marketing. She has been working in the digital marketing industry since the mid 90s and has narrowed her specialty to SEO, Google Ads and Google Analytics. She is especially interested in Google’s new Search Generative Experience and how businesses will need to adapt to remain visible. Want to see how DriveTraffic can help? Let’s chat!
