411bz Authority Intelligence Crawler

The 411bz Authority Intelligence Crawler analyzes publicly accessible web pages to measure AI cite-ability, authority structure, and competitive positioning within industry verticals.

We crawl only publicly accessible content — pages that any web browser can access without authentication, payment, or special access.

Our crawler identifies itself with the following User-Agent string:

You will always see this identifier in your server logs.

• robots.txt — We honor all directives including Disallow rules
• Crawl-Delay — We respect specified delays between requests
• Rate Limiting — Maximum 1 request per second per domain
• Max Pages — We never crawl more than 50 pages per domain per session
• No Authentication Bypass — We never attempt to access protected content
• No Paywall Circumvention — We do not bypass any access restrictions

From publicly accessible HTML, we extract structural and semantic signals:

• Heading structure (H1-H6 hierarchy)
• JSON-LD structured data types
• FAQ and Q&A pattern presence
• Definition clarity patterns
• List and step instruction density
• Entity reinforcement consistency
• Topic cluster strength
• External citation density
• AI discovery signals (llms.txt, robots.txt AI directives)

We do not store raw HTML content. We extract numerical scores and structural metadata only.

To exclude your site from our crawler:

1. Add to your robots.txt:User-agent: 411bz-AuthorityCrawler Disallow: /
2. Or email us at crawler@411bz.com

We process opt-out requests within 24 hours and permanently exclude opted-out domains.

Extracted signals feed into the 411bz Authority Economics Engine to compute:

• AI Cite-Ability Index (ACI) — LLM extraction readiness
• Authority AI Index (AAI) — Composite authority positioning
• Vertical competitive benchmarks
• Reinforcement recommendations

We never sell raw crawl data. We publish only aggregate indices and anonymized vertical benchmarks.

For questions about our crawling practices:

• Email: crawler@411bz.com
• Web: 411bz.ai