What is robots.txt?

The robots.txt file is a plain text file placed at the root of your website (https://example.com/robots.txt) that tells search engine crawlers which pages or sections of your site they should or should not crawl.

How It Works

When a search engine crawler visits your site, it first checks the robots.txt file for instructions. The file uses a simple syntax:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Key Directives

• User-agent — Which crawler the rules apply to (* = all crawlers).
• Disallow — Paths the crawler should not access.
• Allow — Exceptions to disallow rules.
• Sitemap — The URL of your XML sitemap.

Important Notes

• robots.txt is a suggestion, not a security measure. Malicious bots can ignore it.
• Blocking a page in robots.txt does not prevent it from appearing in search results if other pages link to it.
• To prevent indexing, use the noindex meta tag or X-Robots-Tag header instead.

WordPress Default robots.txt

WordPress generates a virtual robots.txt file by default if no physical file exists. It typically blocks /wp-admin/ while allowing admin-ajax.php. For more control, create a physical robots.txt file in your WordPress root directory.

What InspectWP Checks

InspectWP checks whether your WordPress site has a robots.txt file, what rules it contains, and whether a sitemap reference is included.