The WebMeterBot crawler
WebMeterBot is the crawler of WebMeter, a free website check. Here you will find what it does on websites, how to recognise it and how to exclude it from checks.
What the crawler does
It only visits a website when a check of that website is started, typically when someone requests it on webmeter.eu. It performs only passive checks of publicly available information: it does not log in, does not try passwords or vulnerabilities and does not fill in any forms.
During one website check:
- First it downloads robots.txt and follows it according to RFC 9309. It checks it before the PageSpeed tests and before loading the page in the browser. If robots.txt disallows the home page for the crawler, the check does not run and no further request is sent to the website; the crawler does not read disallowed paths. If robots.txt cannot be loaded (server error, overload or network outage), the check does not run.
- It loads the home page once in a browser (Chromium) like a single visitor, including the images, styles and scripts the page loads itself; it loads those regardless of robots.txt. In doing so it takes a screenshot of the home page as internal evidence for the cookie findings.
- It has the website measured 3 times by Google PageSpeed Insights. These tests are run by Google from its own infrastructure, with a User-Agent that contains
Chrome-Lighthouse, not WebMeterBot. - Other requests to the website (HTTP headers, at most five other public pages and files such as sitemap.xml or security.txt, a TLS handshake for the certificate and TLS versions) are usually sent at a rate of at most 2 per second.
- Checks started from webmeter.eu do not try administration addresses (for example
/wp-login.phpor/admin/).
How to recognise the crawler
The crawler identifies itself with this User-Agent:
WebMeterBot/1.0 (+https://webmeter.eu/bot; kontakt: kontakt@webmeter.eu)
In robots.txt, address it with the token WebMeterBot. The crawler also respects rules for the older token WebAnalyzerBot from the time when the tool was called Web Analyzer.
Checks are sent from these IP addresses:
57.131.157.1732001:41d0:701:1100::e11c
A machine-readable list of the addresses is available at /bot/ips.json.
How to exclude the crawler
Add the following to the robots.txt file in the root of your website:
User-agent: WebMeterBot Disallow: /
The crawler then does not run the check and sends no request to the website apart from robots.txt. Disallowing only part of the website (for example Disallow: /admin) does not stop the check; the disallowed paths are simply not read.
Or write to kontakt@webmeter.eu and state the domain. The operator will add it to the opt-out list and the crawler will then send nothing at all to the website, not even a request for robots.txt.
You will find details on the page How to opt a website out of analysis.
Contact
The crawler is operated by WebMeter. Operator details are in the privacy policy. Please send questions and complaints about its behaviour to kontakt@webmeter.eu.