GPTBot vs OAI-SearchBot: Which AI Crawlers to Allow
ChatGPT’s crawler spent 34.82% of its fetches on pages that returned a 404. Claude’s spent 34.16%. Googlebot, over the same period, spent 8.22%. Those figures come from Vercel and MERJ’s December 2024 analysis of their network and nextjs.org. A third of one crawler’s budget hitting dead pages is not a content problem. It is a …
Read more “GPTBot vs OAI-SearchBot: Which AI Crawlers to Allow”