What if AI crawlers could bypass downloading entire websites and only receive structured JSON data instead? Serving AI bots this way could drastically reduce inefficiencies, improve response times, and optimize resource use. Imagine delivering just the essential data—product names, prices, descriptions—without unnecessary UI, animations, or assets.
This idea was inspired by Vercel’s blog post on AI crawlers and their growing influence on SEO. Here are some highlights that led to this thought:
AI crawlers are gaining ground:
GPTBot, Claude, AppleBot, and PerplexityBot made 1.3 billion fetches last month, accounting for 28% of Googlebot’s activity.JavaScript rendering limitations:
Only Google’s Gemini and AppleBot fully render JavaScript. Others, like GPTBot and Claude, fetch JavaScript files but don’t execute them, leaving dynamic content partially ignored.Different content priorities:
GPTBot emphasizes HTML, while Claude focuses on images. This indicates varying strategies or early-stage optimization by AI crawlers.Significant inefficiencies:
Over 30% of requests by AI crawlers hit invalid URLs, highlighting the need for better URL strategies and reduced waste.Why developers should care:
Server-side rendering is more vital than ever to ensure AI crawlers access meaningful content, alongside regular users.
These insights inspired the concept of serving JSON-only responses to AI crawlers, streamlining interactions between websites and bots.
For the full analysis and practical tips, read the original blog post here:
The rise of the AI crawler - Vercel
New research reveals how ChatGPT, Claude, and other AI crawlers process web content, including JavaScript rendering, assets, and other behavior and patterns—with recommendations for site owners, devs, and AI users.
vercel.com
Top comments (1)
Yes, it's possible to serve only data to AI crawlers. This can be achieved by structuring your website in a way that allows AI crawlers to access only the data while keeping other parts of the site hidden or restricted. This is often done using specific tags, metadata, or robots.txt configurations to direct the crawlers to the appropriate data while excluding non-essential content or pages.
However, it’s important to ensure that your site's user experience remains intact, and you comply with SEO best practices to ensure that search engines index your content effectively. It's also essential to keep in mind that many AI crawlers may still analyze the entire structure of your page to make sense of the data, so it’s crucial to strike a balance between providing valuable content and protecting sensitive information.
For seamless management of your data and website structure, tools like SubtitleEdit can assist in organizing your content.