About our crawler
TechLookup.app identifies the technologies a website is built with. To do that, our crawler (TechLookupBot) loads a site's home page once, the way a browser would, and matches what it finds against a public database of technology fingerprints.
It loads one page per site and does not download images, video or fonts. Our scheduled crawl revisits a site about once a month. A visitor looking a site up can trigger one fresh scan if our last one is older than seven days, so a busy site may see a few visits a month. It obeys robots.txt and respects an opt-out list.
robots.txt
Before it loads anything else, the crawler fetches the site's /robots.txt (HTTPS first, then plain HTTP) and follows it as RFC 9309 describes. It obeys the group for TechLookupBot (in any letter case) and, if there is none, the group for *. If that group disallows the home page, the crawler leaves without loading it. To keep TechLookupBot out while letting other crawlers in, add:
User-agent: TechLookupBot Disallow: /
We remember what a site's robots.txt says for up to 24 hours, so a change can take a day to reach us. A missing robots.txt (an error such as 404) means no restrictions. If robots.txt cannot be fetched at all, or the server answers with an error such as 503, we do not scan the site on that attempt and try again later.
robots.txt stops new scans. A report we made earlier stays on TechLookup.app, with its date; to have it removed, use the opt-out below.
Opting out
If you would rather your site was not included, the opt-out page lets you prove you control the domain with one DNS record; we then stop scanning it and delete what we hold. If you cannot change DNS, email hello@techlookup.app from an address at that domain, with the domain name.