AI Crawler

A web crawler operated by an AI company to index content for use in AI answer generation. Examples include GPTBot (OpenAI), Google-Extended (Google), and PerplexityBot (Perplexity). These crawlers function similarly to traditional search crawlers but feed data into AI retrieval systems.