Robots, Sitemap, and Indexing Health Checklist
Make sure the right pages can be found.
The Open buttons send the prompt text as currently shown below to that site only when you click. Fill in the placeholders first.
Check my site's crawl and index controls. Domain and platform: [DOMAIN_AND_PLATFORM]. Contents of robots.txt: [PASTE_ROBOTS]. Sitemap URLs and a sample: [PASTE_SITEMAP_SAMPLE]. Indexing report summary from the search console (indexed, excluded, errors by reason): [INDEX_REPORT_SUMMARY]. Check each item and mark Pass, Fix, or Unknown, with an explanation: [ ] robots.txt exists, is valid, and does not block important sections, CSS, or JavaScript files needed for rendering [ ] It references the sitemap location [ ] Pages I want out of search use noindex (and are not blocked from crawling, or search engines cannot see the tag) [ ] Sitemaps include only indexable canonical URLs that return a 200 status [ ] Lastmod dates reflect real changes [ ] Sitemap size and count are within limits and organized (for example by content type) [ ] The search console has the sitemap submitted and shows no unexpected errors [ ] Reasons for "Crawled, not indexed" and "Discovered, not indexed" are reviewed by page type [ ] Staging and test environments are not indexable (authentication preferred) [ ] HTTPS, single canonical domain version Return a table of the exclusion reasons with counts, my interpretation and the recommended action for each, a corrected robots.txt proposal, and a monthly routine. Warn that robots.txt is public and is not a security tool.
Fill in before use
Optional: type your details and the prompt above updates. Nothing is saved or sent.
Replace each bracketed placeholder with your own details, then paste into Any AI or another assistant.
See an example result (fictional data)
robots.txt: FIX. "Disallow: /assets/" blocks CSS and JavaScript needed to render pages. Sitemap: FIX. 312 URLs listed; 41 return 301 and 9 are noindex. Indexing report: "Crawled, not indexed" 120 URLs, mostly tag pages (thin). Exclusion reasons | count | action Duplicate without canonical | 64 | add canonical Crawled, not indexed | 120 | noindex tags, improve or merge Page with redirect | 41 | remove from sitemap Staging site: noindex only; add password protection. Reminder: robots.txt is public and not a security tool.
Illustrative only, made-up data. Your results will differ and must be reviewed.