WembiBot - crawler passportrev 1.0 · 2026-07

WembiBot

A small, well-behaved crawler operated as a private research tool. It fetches a short, hand-curated list of public pages for niche SEO research. If it showed up in your logs, this page is its passport.

WembiBot/1.0 (+https://wembi.ru)
Match this string against the User-Agent field in your access logs.

01What it is / is not

It is

  • +A private research instrument run by a single operator.
  • +Limited to a hand-curated list tied to a single research project today.
  • +Occasional: short sessions a few times a month, then silence.
  • +Read-only: plain GET of public HTML, nothing else.

It is not

  • -Not a search engine: nothing is indexed or republished.
  • -Not an AI training pipeline: fetched pages never train models.
  • -Not a scraper-for-hire: data stays with the operator.
  • -It does not execute JavaScript, submit forms, or fetch media.

02Crawl behaviour

Every line below is enforced in the crawler's code, not by policy.

robots.txt
Compliant with RFC 9309; re-fetched and cached for at most 24 hours.
Concurrency
1 connection per host, never parallel against your server.
Delay
≥ 3 s between requests, or your crawl-delay if it is larger.
Backoff
Retry-After honoured on 429 / 503; the run pauses, never hammers.
Discovery
Homepage-only for now: one page per site per run. Sitemap-first, multi-page crawling ships in a later release.
Conditional GET
ETag / If-Modified-Since: unchanged pages cost you a 304.
Volume
1 URL per site per run today; per-site budgets ship with multi-page crawling.

03Verification

All requests originate from these static addresses. No cloud IP pools, no proxies.

IPv4
159.253.19.173
IPv6
2a03:f480:1:6::8c
Signatures
Every request also carries a Web Bot Auth signature (RFC 9421 HTTP Message Signatures, Ed25519) - a cryptographic version of the same claim above. Public key: /.well-known/http-message-signatures-directory.

04How to block

Add two lines to your robots.txt. WembiBot re-reads it within 24 hours; the new rule takes effect on the next visit.

User-agent: WembiBot
Disallow: /

Prefer email? Write to hello@wembi.ru: your domain goes on a manual blocklist, confirmed within 24 hours.

Disallow everyone by default but want to explicitly allow WembiBot? Add this instead:

User-agent: WembiBot
Allow: /

05Changelog

  • WembiBot/1.02026-07-12Initial release: RFC 9309 compliance, conditional GET, Web Bot Auth (RFC 9421) request signing.