Crawler

WebPulseNowBot

Last updated August 27, 2026

WebPulseNowBot is the crawler behind Web Pulse Now, a website-monitoring service. It checks pages for SEO, accessibility, performance, security and uptime problems, and it reads pages — it never changes anything. Most of its traffic is monitoring: sites placed under continuous audit by the customer who administers them. It also runs one-off audits of a single site at a customer's request, including sites they are evaluating rather than operating, which are capped per account and leave nothing running afterwards.

1. How to identify it

Every request carries WebPulseNowBot in its User-Agent. There are three, by purpose: WebPulseNowBot/1.0 (+https://webpulsenow.com/bot; site health scanner) for crawling, WebPulseNowBot/1.0 (+https://webpulsenow.com/bot; uptime check) for availability checks, and Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36 WebPulseNowBot/1.0 (+https://webpulsenow.com/bot; rendering) when a page is rendered in a real browser for screenshots and Core Web Vitals.

That last one keeps the Chrome tokens because many sites serve different markup to anything that does not look like a browser, and a measurement taken on that markup would describe a page no visitor sees. It identifies itself as WebPulseNowBot all the same.

Requests are read-only — GET and HEAD only. The crawler never submits forms, never sends cookies or credentials, and never attempts to reach anything behind a login.

2. It obeys robots.txt

Before crawling anything, WebPulseNowBot fetches /robots.txt and applies the rules for its own user-agent, falling back to the wildcard group. Disallowed paths are never requested — not for crawling and not for link checking. Crawl-delay is honoured in full — the crawler goes single-file and waits the requested interval between requests. A site that asks for a long delay gets fewer pages read within the scan’s time budget, never faster requests.

If robots.txt disallows the site root, the scan stops and reports that rather than crawling anyway.

3. How to allow it

Add an exception in robots.txt: User-agent: WebPulseNowBot — Allow: /

If a firewall or WAF is turning it away, allow the WebPulseNowBot user-agent there too. On Cloudflare that means a WAF custom rule with a Skip action matching the user-agent; under Security → Bots, "Bot Fight Mode" will challenge unverified crawlers regardless of robots.txt.

Whoever administers the site only needs to do this for that site, and only if it should be monitored. The crawler runs at a customer's request — sometimes the site's operator, often an agency monitoring it for the operator.

4. How to block it

Disallow it in robots.txt and it will stop: User-agent: WebPulseNowBot — Disallow: /

No appeal, no workaround. The crawler does not disguise its user-agent, rotate addresses, or attempt to bypass a block of any kind.

5. What load to expect

Two kinds of request, on very different rhythms. Availability checks are the bulk of the traffic and the lightest: one request per monitored site every five minutes, continuously.

Full audits are scheduled by the customer monitoring the site, daily at the finest, and are bounded: 25 to 2,500 pages per scan depending on that customer's plan, a handful of concurrent requests, dropping to one at a time whenever robots.txt asks for a crawl delay.

If a scan is causing problems for a site, the Crawl-delay directive is respected and will slow it down.

6. What it keeps, and what it never does with it

A scan produces one thing: a report for the customer who asked for it. That report holds the findings, the pages each was found on, and screenshots of destinations that were broken. It is kept for 30 to 365 days depending on that customer's plan, after which the page-level detail is deleted. A daily score rollup outlives it, so the trend line survives when the detail behind it does not.

Every finding names the URL it was read from, and nothing is repeated back without pointing at the page it came from.

What WebPulseNowBot reads is shown only to the workspace that requested the scan. It is never used to train models, never sold, and never redistributed — not to other customers, not to anyone. The crawler exists to tell one customer about a site they are responsible for.

7. Where it comes from

All requests originate from these addresses: 209.71.103.80, 2a09:8280:e618:1:0:174:dd04:0.

The same list is published as JSON at https://webpulsenow.com/bot/ips.json for anything that needs to check it automatically.

Traffic claiming this user-agent from any other address is not ours.

8. Questions or problems

If this crawler is causing trouble, or you believe it is accessing something it shouldn't, email support@webpulsenow.com and we will stop it against your domain.

Questions about this policy? Contact legal@webpulsenow.com.