About IndexablyBot
Our web crawler and how it works
What is IndexablyBot?
IndexablyBot is the web crawler used by Indexably.io to analyze web pages for AI visibility optimization. When users scan their websites through our platform, IndexablyBot fetches the page content to evaluate how well it's optimized for AI systems like ChatGPT, Perplexity, Claude, and Gemini.
User-Agent String
Mozilla/5.0 (compatible; IndexablyBot/1.0; +https://indexably.io/bot) — This is the User-Agent string our crawler uses when making requests to your website.
Our Crawling Ethics
robots.txt Compliance: We fully respect robots.txt directives. If your robots.txt blocks IndexablyBot or all crawlers from a path, we will not crawl it. Rate Limiting: We limit our request frequency to avoid overloading your servers. We also respect Crawl-delay directives. User-Initiated Only: IndexablyBot only crawls pages when a user explicitly requests a scan through our platform. No Data Storage: We analyze content for AI visibility metrics but do not permanently store or redistribute your page content. Transparent Identification: We clearly identify ourselves with a descriptive User-Agent and link back to this page.
Controlling IndexablyBot Access
You can control IndexablyBot's access to your website using your robots.txt file. Block IndexablyBot from your entire site: User-agent: IndexablyBot / Disallow: /. Block IndexablyBot from specific paths: User-agent: IndexablyBot / Disallow: /private/ / Disallow: /admin/ / Allow: /. Set a crawl delay (seconds between requests): User-agent: IndexablyBot / Crawl-delay: 10.
Contact Us
If you have questions about IndexablyBot, notice any issues with our crawler's behavior, or need assistance configuring access, please contact us. Email: indexablysupport@frontbrain.io. Website: indexably.io/contact
Frequently Asked Questions
Why is IndexablyBot visiting my site? Someone used Indexably.io to analyze your website's AI visibility. This could be the site owner, a team member, or an SEO consultant working on the site. How often does IndexablyBot crawl? IndexablyBot only crawls when a user initiates a scan. We don't continuously crawl or index websites. Most sites are scanned once and may be re-scanned when the user wants updated metrics. Does IndexablyBot execute JavaScript? By default, IndexablyBot fetches raw HTML without executing JavaScript, similar to how most AI crawlers (ChatGPT, Perplexity, Claude) access content. Premium users can optionally enable JavaScript rendering to see how their site appears to JS-capable crawlers like Google's. Is IndexablyBot a search engine? No. IndexablyBot is an analysis tool, not a search engine. We don't index your content for search or make it discoverable to others. We only analyze pages to provide AI visibility scores and recommendations to the person who requested the scan. How do I re-allow IndexablyBot after blocking it? 1. Update your robots.txt file to remove the Disallow rule for IndexablyBot. 2. Alternatively, add an explicit Allow rule for the paths you want scanned. 3. Wait up to 24 hours for our cache to refresh (or contact us to clear it immediately). 4. Try scanning your site again through Indexably.io. Example robots.txt to allow IndexablyBot: User-agent: IndexablyBot / Allow: /