How to Fix Crawlability Problems

What the crawlability check tests

SiteCurl runs five crawlability checks: whether a robots.txt file exists, whether it accidentally blocks all crawlers, whether an XML sitemap exists at /sitemap.xml, whether the sitemap is referenced in robots.txt, and whether AI crawlers are blocked.

Why it matters

If your robots.txt blocks all crawlers, your site will not appear in any search results. A missing sitemap means search engines can only find pages by following links, which may miss deep or new content. These are foundational SEO settings that must be correct before any on-page optimization matters.

How to fix it

Create or fix robots.txt

Every site needs a robots.txt at the domain root. The most permissive version:

User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml

If your robots.txt contains Disallow: / under User-agent: *, remove that line or change it to Allow: /. The Disallow: / directive blocks every search engine from your entire site.

Add an XML sitemap

Create a sitemap.xml file that lists every page you want indexed. Most CMS platforms generate this automatically:

• WordPress: Install Yoast SEO or Rank Math (both generate sitemaps)
• Shopify: Auto-generated at /sitemap.xml
• Rails: Add a sitemap route that renders XML

Reference the sitemap in robots.txt

Add a Sitemap: directive to your robots.txt so crawlers can find it immediately:

Sitemap: https://yoursite.com/sitemap.xml

How to verify the fix

Visit https://yoursite.com/robots.txt in a browser and confirm it allows crawling. Visit https://yoursite.com/sitemap.xml and confirm it lists your pages. Then run a SiteCurl scan to verify all crawlability checks pass.

Related checks

Crawlability works alongside your robots.txt setup and AI crawler access settings. Also check your canonical URLs to avoid confusing search engines about which page version to index.

Start a free trial to audit your site’s crawlability.