What to do when the crawl fails, temporarily disabling security mechanisms

If your InspectWP crawl fails, returns an empty report or visibly only analyzes a challenge page, security mechanisms on the target site are almost always the cause. This guide walks you through every common roadblock, from Cloudflare to Wordfence to .htaccess, and shows how to disable them for the duration of a crawl or whitelist InspectWP.

1. Why crawls fail

Typical symptoms that tell you a security barrier is in the way:

• Timeout / aborted crawl: The site doesn't respond, or only after 30+ seconds.
• Empty or near-empty report: No title, no plugins, no theme detected, most likely a challenge or block page was crawled.
• HTTP 403 / 429 / 503: A firewall rejected the request or rate-limit was triggered.
• Wrong content: The screenshot shows a Cloudflare checking page, a Wordfence block page, a coming-soon screen or a login form instead of your real website.

2. Before you start

InspectWP uses a real headless Chrome browser and does not disguise itself as a search-engine bot. The user agent contains the marker InspectWP. That means: if you generally block bots, you'll also block InspectWP, and that is the actual root cause. Whitelisting is usually the cleaner solution rather than fully turning protection off.

3. Cloudflare

Cloudflare is the most common reason for failed crawls. Log into the Cloudflare dashboard and check:

• Security → Bots: Set Bot Fight Mode and Super Bot Fight Mode to Off.
• Security → Settings: Temporarily set Security Level to Essentially Off or Low.
• Security → WAF → Tools: Make sure Under Attack Mode is off (use High or lower instead).
• Custom Rules: If you have your own WAF rules, check whether one of them blocks user agents or IPs.

If you don't want to disable bot protection entirely, create a Cloudflare WAF custom rule with action Skip for user agents that contain InspectWP.

4. Wordfence

Wordfence is the most popular WordPress security plugin and often blocks crawlers very aggressively. Here's how to handle it:

• Wordfence → Tools → Live Traffic: Look for blocked requests from the InspectWP IPs 195.201.17.43 and 46.224.183.125 and add them under Whitelisted IPs.
• Wordfence → Firewall → All Firewall Options: Briefly switch the firewall to Learning Mode or Disabled.
• Rate Limiting: Significantly raise the thresholds for “How many page views can a crawler visit per minute”.
• Block fake Google crawlers: This option may block InspectWP, disable it temporarily.

5. Sucuri, Solid Security, iThemes Security, All-In-One Security (AIOS)

Other well-known security plugins use very similar mechanisms. Look specifically for:

• Brute-force protection / 404 detection
• Rate-limiting for unknown user agents
• Featured / recommended block lists
• Country blocking

Disable the relevant feature or raise the thresholds temporarily.

6. Limit Login Attempts / Loginizer

These plugins block IPs after failed login attempts. InspectWP never tries to log in, but: if your server has just registered other failed logins from the crawl IP range, the IP may already be banned. Check the plugin's block list and remove the entry if needed.

7. Anti-bot and anti-spam plugins

CleanTalk, Blackhole for Bad Bots, StopBadBots and friends operate on heuristics and block any unusual user agent. The only fix is: briefly disable them, or add the InspectWP user agent to the plugin's whitelist.

8. Coming-soon and maintenance plugins

Plugins like SeedProd, WP Maintenance Mode, Elementor Coming Soon or WP Maintenance show external visitors a placeholder page. InspectWP then analyzes that placeholder, not your real site. Bypass links offered by some plugins usually do not work for external crawls. Solution: briefly deactivate the plugin, run the crawl, reactivate the plugin.

9. Caching and optimization plugins

WP Rocket, LiteSpeed Cache, W3 Total Cache and similar tools can produce odd crawl results when aggressive optimization is enabled, for example when JavaScript is delayed or combined. Recommendations:

• Clear the cache before crawling
• Keep „Bot Cache“ / „Cache for logged-out users“ enabled, otherwise InspectWP may see a stale version
• Check JavaScript-Delay / lazy-render options, InspectWP does wait for interaction, but extreme delays cause timeouts

10. Password protection, members-only, restricted content

A page that's only accessible to logged-in users or behind HTTP basic auth cannot be crawled by InspectWP. Make sure the URL you want to analyze is publicly reachable without login. Briefly deactivate plugins like Restrict Content Pro, MemberPress or Password Protected, or set the target page to public.

11. .htaccess and nginx, IP, country and user-agent blocks

At server level, crawlers are often blocked via Deny or RewriteRule. Examples from typical .htaccess files you can comment out temporarily:

# Bot user-agent block (common)
RewriteCond %{HTTP_USER_AGENT} (bot|crawler|spider) [NC]
RewriteRule .* - [F,L]

# IP block
Deny from 1.2.3.4

# Country block via mod_geoip
SetEnvIf GEOIP_COUNTRY_CODE RU BlockCountry
Deny from env=BlockCountry

For nginx, the equivalent looks like:

if ($http_user_agent ~* (bot|crawler|spider)) {
    return 403;
}

Comment out those lines for the duration of the crawl.

12. Hosting-side protection mechanisms

Some hosts run their own Web Application Firewall (WAF) or ModSecurity rules that you won't see in the plugin or .htaccess audit. Ask their support to whitelist the InspectWP IPs 195.201.17.43 and 46.224.183.125. Known examples:

• All-Inkl, IONOS, Strato: Bot protection in the hosting panel, contact support or disable from the customer menu.
• SiteGround: AI anti-bot, Smart-WAF, under Site Tools → Security.
• Kinsta, WP Engine: Proprietary bot detection, request whitelisting via support.
• Hetzner / cloud providers: Rarely WAF issues, but GeoIP restrictions are possible.

13. CSP, X-Frame-Options and robots.txt

For clarity: a robots.txt with Disallow: / does not stop InspectWP from crawling, we don't strictly honor robots.txt. Content-Security-Policy and X-Frame-Options on the other hand may prevent individual sub-requests (iframes, third-party scripts); that's normal and not an error. The blockers that actually matter are 403/429/503 responses on the main document.

14. Rate limiting and Fail2Ban

At server level, fail2ban, mod_evasive or nginx limit_req can ban the crawl IP within seconds, especially with many parallel sub-requests. If you have SSH access, check /var/log/fail2ban.log or iptables -L. A short-term whitelist of the InspectWP server IP solves the issue.

15. Checklist before re-crawling

• ☐ Cloudflare Bot Fight Mode off
• ☐ Wordfence firewall in Learning Mode or InspectWP IP whitelisted
• ☐ Security plugin (Sucuri / Solid / AIOS) on a moderate setting
• ☐ InspectWP IPs 195.201.17.43 and 46.224.183.125 added to plugin/hosting whitelist
• ☐ Coming-soon / maintenance plugin deactivated
• ☐ Caching plugin cache cleared
• ☐ .htaccess / nginx checked for user-agent/IP blocks
• ☐ Hosting panel: bot protection / WAF reviewed
• ☐ Page is publicly reachable without login
• ☐ Browser cache cleared and a fresh test crawl run

16. When nothing helps

Email us at hello@inspectwp.com with the domain and the approximate time of the failed crawl.

After a successful crawl, don't forget to re-enable all security mechanisms!