WaafirBot

WaafirBot is the web crawler for Waafir’s Market Intelligence feature. It reads the public websites of investment firms and companies — pages such as an investment thesis, a portfolio list or a “contact us” page — so that founders and their advisors can find investors whose stated focus matches their company. It only reads pages that anyone can open without signing in.

How to recognize it

Every request WaafirBot makes carries this User-Agent:

Mozilla/5.0 (compatible; WaafirBot/1.0; +https://www.waafir.io/waafirbot)

robots.txt

WaafirBot obeys robots.txt (RFC 9309). It reads the rules for the WaafirBot user agent, or the * group when there is none, before it requests any other page on a site, and checks them again at least once a day. If your robots.txt cannot be reached because of a server error, WaafirBot treats the whole site as disallowed until it can read it. If it answers 401 or 403, WaafirBot treats the whole site as disallowed too. To block it entirely:

User-agent: WaafirBot
Disallow: /

How politely it crawls

  • At most one request every 10 seconds to any one site, or slower if your Crawl-delay asks for it (up to 60 seconds).
  • It backs off when your site returns errors. When your site asks it to slow down (HTTP 429, or a bot challenge), it stops visiting for a day, or for longer if your Retry-After header says so. After repeated errors it pauses for an hour, then four hours, then a day.
  • It visits a page about once a week, and fetches pages one at a time — it does not run scripts, submit forms or log in.
  • It does not collect the names of individual people.

Opting out

The quickest way is the robots.txt rule above. You can also email support@waafir.io with your domain: we will stop crawling it and stop showing information drawn from it.