Methodology
How we audit your site.
No black box. Every score on every report comes from one of three pipelines, all documented below.
Discovery
When you submit a URL we fetch robots.txt to discover any listed sitemaps, then walk the sitemap (and any nested sitemap indexes) up to a depth of 3. If we find fewer than 5 URLs that way we fall back to crawling links from the homepage. We canonicalise URLs (lowercase host, https, strip tracking parameters, sort query params), drop assets and auth-required paths, then sample down to the page cap for your tier.
Discovery happens in under 2 seconds for almost every site. The page cap is 3 (free preview) or 100 (Full audit).
Performance
Every page is scanned through Google's PageSpeed Insights v5 API, which runs the same Lighthouse engine Google uses to grade your site for ranking. We collect six lab performance metrics — LCP, CLS, TBT, FCP, TTFB and Speed Index — for both mobile and desktop viewports, plus Google's Core Web Vitals (LCP, CLS, INP) from real-user field data where the Chrome UX Report has coverage.
Free preview runs 1 pass per page. The paid Full audit runs 3 passes and reports the median — this is the recommended methodology because Lighthouse results can vary 10–20% run-to-run.
SEO Readiness
Each page goes through 12 deterministic checks against its rendered HTML and the site's robots.txt / sitemap.xml:
- HTTPS
sitemap.xmlpresent and parseablerobots.txtpresent- Title tag (length 10–70 chars)
- Meta description (length 50–170 chars)
- Canonical URL via
<link rel="canonical"> - Open Graph tags (title, description, image — all three)
- Twitter Card meta
- JSON-LD structured data presence
- Exactly one H1 per page
<html lang="…">attribute- Viewport meta tag
Each check is pass / warn / fail. The page score is (pass + warn × 0.5) / total × 100.
AI Readiness
Seventeen checks specific to AI-era discoverability, organized in three pillars. Site-level signals (robots.txt policies, the CDN gate, /llms.txt, sitemap freshness) are evaluated once at the domain root and applied across the report; page-level signals (structured data, headings, server-rendered content, snippet directives, consent walls, author schema) are assessed per page.
Access — can AI crawlers fetch the site at all?
- Answer-engine crawler access — whether
robots.txtrestricts the live retrieval / citation crawlers that gate whether you show up in AI answers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User). Blocking these fails the check. - Training-crawler stance — the opt-out tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot). Reported but never penalized: blocking training is a legitimate IP decision with no effect on AI-search citations.
- Search-index crawlers — Googlebot and Bingbot, because ChatGPT search and Copilot ride Bing's index and AI Overviews ride Google's.
- CDN / firewall gate — we fetch the homepage with an AI bot user-agent and diff status + content against a normal browser fetch. The probe runs from our IPs, so a CDN that allow-lists official bot IP ranges may still admit the real crawler — the check says "likely", and the evidence is shown.
- Snippet directives —
nosnippet,max-snippet,noaiin robots meta tags orX-Robots-Tagheaders.
Comprehension — once fetched, can they parse it?
/llms.txtand/llms-full.txt— the emerging standard from llmstxt.org; content quality is validated separately in the report.- Structured data, validated — every JSON-LD block is parsed and recognized types are checked for required properties (a Product without
offers, an Article withoutdatePublishedare flagged with the missing fields). - Server-rendered content — word count in the initial HTML, with explicit detection of empty JS app shells. AI crawlers don't execute JavaScript.
- Semantic HTML — presence of
<main>,<article>,<nav>,<header>. - Heading outline — H1 presence and level continuity (no H2→H4 skips); LLM retrieval is chunk-based and follows the outline.
- Open Graph metadata completeness.
- RSS/Atom feed advertised in
<head>. - Sitemap freshness —
<lastmod>presence. - Consent wall — consent-manager markup fronting a thin server-rendered page means crawlers read the banner, not the content.
Trust — can they attribute and date it?
- Author / Organization schema in JSON-LD or
<meta name="author">— provenance signals trusted by AI assistants. - Content dates —
datePublished/dateModifiedin structured data; without them answer engines can't judge freshness.
Recommendations
We group raw Lighthouse audit failures into root-cause themes (e.g. "render-blocking resources" instead of 14 separate audit IDs), sequence them by effort (small / medium / large), and attach a plain-English business-impact line to each. The effort estimate in dev-weeks is calibrated from past audits.
Limits and honest caveats
- Lab measurements come from Google's US-based test infrastructure. Real-user experience in other regions may differ.
- Lab data reflects a cold first visit. Returning visitors with a warm cache load faster.
- Only public pages are scanned. Authenticated pages, paywalled content, and bot-blocked areas are excluded.
- Mobile is emulated on a throttled slow-4G connection. Desktop is unthrottled.
- PageSpeed Insights has a public rate limit. Free preview without an API key works but is slow. We use a key on paid scans.