If your InspectWP crawl fails, returns an empty report or visibly only analyzes a challenge page, security mechanisms on the target site are almost always the cause. This guide walks you through every common roadblock, from Cloudflare to Wordfence to .htaccess, and shows how to relax or briefly disable them for the duration of a crawl.
1. Why crawls fail
Typical symptoms that tell you a security barrier is in the way:
- Timeout / aborted crawl: The site doesn't respond, or only after 30+ seconds.
- Empty or near-empty report: No title, no plugins, no theme detected, most likely a challenge or block page was crawled.
- HTTP 403 / 429 / 503: A firewall rejected the request or rate-limit was triggered.
- Wrong content: The screenshot shows a Cloudflare checking page, a Wordfence block page, a coming-soon screen or a login form instead of your real website.
2. Before you start
InspectWP uses a real headless Chrome browser and appears like a regular visitor with a current Chrome. The crawl comes from a pool of changing server IPs. There is neither a fixed crawler IP nor an InspectWP marker in the user agent that you could whitelist. That means: if your bot protection blocks or challenges unknown visitors, it also hits InspectWP, and that is the actual root cause. The fix is therefore always to relax or briefly disable the respective protection for the duration of the crawl.
3. Cloudflare
Cloudflare is the most common reason for failed crawls. Log into the Cloudflare dashboard and check:
- Security → Bots: Set Bot Fight Mode and Super Bot Fight Mode to Off.
- Security → Settings: Temporarily set Security Level to Essentially Off or Low.
- Security → WAF → Tools: Make sure Under Attack Mode is off (use High or lower instead).
- Custom Rules: If you have your own WAF rules, check whether one of them blocks user agents or IPs.
InspectWP can often pass a Managed Challenge on its own. If that fails, the only fix is to lower the Security Level or turn off Bot Fight Mode for the duration of the crawl.
4. Wordfence
Wordfence is the most popular WordPress security plugin and often blocks crawlers very aggressively. Here's how to handle it:
- Wordfence → Tools → Live Traffic: Check whether requests were blocked at the time of the crawl. The entry tells you which rule fired.
- Wordfence → Firewall → All Firewall Options: Briefly switch the firewall to Learning Mode or Disabled.
- Rate Limiting: Significantly raise the thresholds for “How many page views can a crawler visit per minute”.
- Block fake Google crawlers: This option may block InspectWP, disable it temporarily.
5. Sucuri, Solid Security, iThemes Security, All-In-One Security (AIOS)
Other well-known security plugins use very similar mechanisms. Look specifically for:
- Brute-force protection / 404 detection
- Rate-limiting for unknown user agents
- Featured / recommended block lists
- Country blocking
Disable the relevant feature or raise the thresholds temporarily.
6. Limit Login Attempts / Loginizer
These plugins block IPs after failed login attempts. InspectWP never tries to log in, but: if your server has just registered other failed logins from the crawl IP range, the IP may already be banned. Check the plugin's block list and remove the entry if needed.
7. Anti-bot and anti-spam plugins
CleanTalk, Blackhole for Bad Bots, StopBadBots and friends operate on heuristics and block any unusual user agent. The only fix is to briefly disable them for the duration of the crawl.
8. Coming-soon and maintenance plugins
Plugins like SeedProd, WP Maintenance Mode, Elementor Coming Soon or WP Maintenance show external visitors a placeholder page. InspectWP then analyzes that placeholder, not your real site. Bypass links offered by some plugins usually do not work for external crawls. Solution: briefly deactivate the plugin, run the crawl, reactivate the plugin.
9. Caching and optimization plugins
WP Rocket, LiteSpeed Cache, W3 Total Cache and similar tools can produce odd crawl results when aggressive optimization is enabled, for example when JavaScript is delayed or combined. Recommendations:
- Clear the cache before crawling
- Keep „Bot Cache“ / „Cache for logged-out users“ enabled, otherwise InspectWP may see a stale version
- Check JavaScript-Delay / lazy-render options, InspectWP does wait for interaction, but extreme delays cause timeouts
10. Password protection, members-only, restricted content
A page that's only accessible to logged-in users or behind HTTP basic auth cannot be crawled by InspectWP. Make sure the URL you want to analyze is publicly reachable without login. Briefly deactivate plugins like Restrict Content Pro, MemberPress or Password Protected, or set the target page to public.
11. .htaccess and nginx, IP, country and user-agent blocks
At server level, crawlers are often blocked via Deny or RewriteRule. Examples from typical .htaccess files you can comment out temporarily:
# Bot user-agent block (common)
RewriteCond %{HTTP_USER_AGENT} (bot|crawler|spider) [NC]
RewriteRule .* - [F,L]
# IP block
Deny from 1.2.3.4
# Country block via mod_geoip
SetEnvIf GEOIP_COUNTRY_CODE RU BlockCountry
Deny from env=BlockCountryFor nginx, the equivalent looks like:
if ($http_user_agent ~* (bot|crawler|spider)) {
return 403;
}Comment out those lines for the duration of the crawl.
12. Hosting-side protection mechanisms
Some hosts run their own Web Application Firewall (WAF) or ModSecurity rules that you won't see in the plugin or .htaccess audit. Ask their support to relax the bot protection for the duration of the crawl. Known examples:
- All-Inkl, IONOS, Strato: Bot protection in the hosting panel, contact support or disable from the customer menu.
- SiteGround: AI anti-bot, Smart-WAF, under Site Tools → Security.
- Kinsta, WP Engine: Proprietary bot detection, ask support to relax it.
- Hetzner / cloud providers: Rarely WAF issues, but GeoIP restrictions are possible.
13. CSP, X-Frame-Options and robots.txt
For clarity: a robots.txt with Disallow: / does not stop InspectWP from crawling, we don't strictly honor robots.txt. Content-Security-Policy and X-Frame-Options on the other hand may prevent individual sub-requests (iframes, third-party scripts); that's normal and not an error. The blockers that actually matter are 403/429/503 responses on the main document.
14. Rate limiting and Fail2Ban
At server level, fail2ban, mod_evasive or nginx limit_req can ban the crawl IP within seconds, especially with many parallel sub-requests. If you have SSH access, check /var/log/fail2ban.log or iptables -L. Raise the thresholds for the duration of the crawl or remove the banned IP from the ban list.
15. Checklist before re-crawling
- ☐ Cloudflare Bot Fight Mode off
- ☐ Wordfence firewall in Learning Mode
- ☐ Security plugin (Sucuri / Solid / AIOS) on a moderate setting
- ☐ Coming-soon / maintenance plugin deactivated
- ☐ Caching plugin cache cleared
- ☐ .htaccess / nginx checked for user-agent/IP blocks
- ☐ Hosting panel: bot protection / WAF reviewed
- ☐ Page is publicly reachable without login
- ☐ Browser cache cleared and a fresh test crawl run
16. When nothing helps
Email us at hello@inspectwp.com with the domain and the approximate time of the failed crawl.
After a successful crawl, don't forget to re-enable all security mechanisms!