What Is Robots.txt?
Robots.txt is a plain-text file placed in a website’s root directory that instructs search engine crawlers which pages or sections of the site they are allowed or disallowed to crawl.
Why it matters
Robots.txt helps manage crawl budget and keep crawlers away from low-value or sensitive areas (like admin pages or internal search results). But it’s easy to misconfigure — a single stray “Disallow: /” can accidentally block your entire site from search engines. Note that blocking a page in robots.txt prevents crawling, not necessarily indexing; use a noindex tag to keep pages out of the index.
Example
A line reading “Disallow: /wp-admin/” tells crawlers to skip the WordPress admin folder, while allowing the rest of the site to be crawled normally.
Ready to put this into practice?
Auditing crawl directives like this is standard in our technical SEO services — let’s talk about your goals.