Technical SEO
Google shows a page only after crawling it, processing the content and deciding to store it in the index. When a page is left out, the causes on the site's side are few and can be checked: robots.txt, noindex, a canonical pointing to another URL, an HTTP status other than 200, a page that looks like an error, or content that only appears after JavaScript runs. The guides below explain each case, plus redirects, structured data, AI crawlers and llms.txt.
Crawling and indexing: robots.txt, sitemaps and HTTP status
Why Google doesn't index a page: robots.txt, noindex, canonicals, sitemaps, HTTP status, JavaScript and the Search Console reasons, with a checklist.
Go deeper
- Technical SEO
AI crawlers in robots.txt: search, user access and training
GPTBot, ClaudeBot, OAI-SearchBot, Google-Extended: what each AI crawler does, whether it respects robots.txt, and how to block training but stay in search.
- Technical SEO
Redirects: chains, loops and how to fix them
How Google handles 301, 302, 307 and 308, what chains and loops do to crawling and speed, and how to map, fix and test your redirects.
- Technical SEO
llms.txt: what it is, what it doesn't do and how to create one
llms.txt summarizes a site for AI agents. See the format, what Google and other companies say about it, when it's worth creating and how to write a good one.
- Technical SEO
Structured data: what Google uses and how to validate it
Which structured data still earns rich results on Google in 2026, the rules for reviews, and how to validate without confusing schema.org with Google.
Frequently asked questions
- Does blocking a page in robots.txt remove it from Google?
- Not necessarily. robots.txt controls crawling, not indexing: a blocked URL can still appear if other sites link to it. To remove a page from the index, use noindex and keep the page crawlable.
- Does structured data improve ranking on Google?
- It makes the page eligible for rich results, such as stars and breadcrumbs, when it follows Google's rules. It doesn't guarantee that the rich result will appear.
- Do I need llms.txt to appear in Google's AI answers?
- No. Google says Search, including its generative AI features, doesn't use llms.txt or any special markup for AI. The file may be useful to other tools and agents.
- Does blocking GPTBot keep my site out of ChatGPT?
- GPTBot is OpenAI's training crawler. ChatGPT search uses a different crawler, OAI-SearchBot, and each one is controlled by its own line in robots.txt. You can block training and keep search.
Other topics
Accessibility
If your company has its headquarters or a commercial representative in Brazil, its website must be accessible to people with disabilities: that is what article 63 of Brazil's Inclusion Law (Lei Brasileira de Inclusão, Law 13,146/2015) requires.
5 guidesPrivacy
Brazil's LGPD (Lei Geral de Proteção de Dados, Law No. 13,709/2018) never mentions cookies, but it applies to any cookie, pixel or script that collects personal data from site visitors.
3 guidesPerformance
Core Web Vitals are three Google metrics that measure the real experience of visitors: LCP measures loading, INP measures responsiveness to clicks and taps, and CLS measures visual stability.
6 guidesSecurity
The first security step on a public website is to check what any visitor can see from the outside: whether every request goes to HTTPS with a valid certificate, whether the security headers are present, whether any files are exposed, whether the CMS and plugins are up to date and which third-party scripts run on the page.
3 guides
Partners
Partner benefits with one coupon.
WPLIGHT20% OFFGet the offer from WP Rocket (opens in a new tab)
25% OFFGet the offer from Perfmatters (opens in a new tab)
Partner links. We may earn a commission at no extra cost to you.