01What it is / is not
It is
- +A private research instrument run by a single operator.
- +Limited to a hand-picked set of sites: a few research projects of its operator and their competitors.
- +Occasional: runs are started by hand, not on a schedule.
- +Read-only: plain GET of robots.txt, sitemaps, /llms.txt and public HTML pages, nothing else.
It is not
- -Not a search engine: nothing is indexed or republished.
- -Not an AI training pipeline: fetched pages never train models.
- -Not a scraper-for-hire: data stays with the operator.
- -It does not execute JavaScript, submit forms, or fetch media.
02Crawl behaviour
Every line below is enforced in the crawler's code, not by policy.
- robots.txt
- Compliant with RFC 9309; re-fetched and cached for at most 24 hours.
- Concurrency
- 1 request at a time per site (with or without www), across all our runs.
- Delay
- ≥ 3 s between requests, and at least 3× your last response time, or your crawl-delay if it is larger. Exception: the sites run by the operator of Wembi itself, which trust the bot, get ≥ 0.5 sand at least your last response time; nobody else's site does.
- Backoff
- Retry-After honoured on 429 / 503; repeated refusals pause the whole site for up to 24 hours.
- Discovery
- Sitemap-first. By default only the pages listed in your sitemap, plus (unless switched off for a run) one check of same-site addresses linked from them but missing from the sitemap; links found on those extra pages are not followed. Without a sitemap, or when a run is set to follow links, links within the same site. Links marked nofollow are not followed. One request for /llms.txt per site in a whole-site crawl.
- Conditional GET
- ETag / If-Modified-Since: unchanged pages cost you a 304.
- Volume
- At most 2,000 pages per site per run.
03Verification
All requests originate from these static addresses. No cloud IP pools, no proxies.
- IPv4
159.253.19.173- IPv6
2a03:f480:1:6::8c- Signatures
- Every request also carries a Web Bot Auth signature (RFC 9421 HTTP Message Signatures, Ed25519) - a cryptographic version of the same claim above. Public key: /.well-known/http-message-signatures-directory.
04How to block
Add two lines to your robots.txt. WembiBot re-reads it within 24 hours; the new rule takes effect on the next visit.
User-agent: WembiBot Disallow: /
Prefer email? Write to hello@wembi.ru: your domain goes on a manual blocklist, confirmed within 24 hours.
Disallow everyone by default but want to explicitly allow WembiBot? Add this instead:
User-agent: WembiBot Allow: /
05Changelog
- WembiBot/1.02026-10-18The operator's own sites are crawled with a shorter pause (≥ 0.5 s). Every other site keeps the pace above, unchanged.
- WembiBot/1.02026-10-17Serves a few research projects instead of one, each a site with its competitors. Limits per site and the crawl behaviour above are unchanged.
- WembiBot/1.02026-10-16Crawls follow the sitemap by default: the sitemap's pages first, then one check of same-site addresses linked from them but missing from it. One request for /llms.txt per site in a whole-site crawl.
- WembiBot/1.02026-10-15Multi-page crawling: sitemap-first discovery, links within the same site, a per-site budget of 2,000 pages; one request at a time per site across all runs; backoff that persists across runs.
- WembiBot/1.02026-07-12Initial release: RFC 9309 compliance, conditional GET, Web Bot Auth (RFC 9421) request signing.