Crawler

FESIT DomainCheck bot

How our outbound crawler identifies itself, what it fetches, and how to contact us.

What we crawl

FESIT runs short, infrequent fetches against organisation websites that are part of a domain check or sector scan: DNS already resolved, then HTTPS homepage HTML, optional headless browser for JavaScript-loaded trackers, and a small set of contact/directions paths. We do not bulk-index the web and we do not follow arbitrary links.

Identification

• User-Agent: FESIT-DomainCheck/<version> (+<this page>)
• From (HTTP header, when configured): crawler@fesit.eu
• robots.txt token: FESIT-DomainCheck (also respects User-agent: *)

robots.txt

Outbound crawls respect your robots.txt. To block only FESIT:

User-agent: FESIT-DomainCheck Disallow: /

A missing or unreadable robots.txt is treated as allow-all (common crawler practice).

Operator

Operated by Depole.io BV, Tongeren, Belgium. Product site: fesit.eu. Privacy: privacy policy.