Methodology

How TechLookup.app produces the technology data on its reports, technology pages and comparisons: what we scan, how a technology is identified, how the numbers are counted, and where they fall short.

Last reviewed 8 November 2026

How a site is scanned

Our crawler, TechLookupBot, opens a site's home page in a real Chromium browser, the same engine as Google Chrome, and lets it run its scripts as it would for a visitor. It tries HTTPS first and falls back to plain HTTP. It waits up to 15 seconds for the page to load and a few more seconds for network activity to settle. If the page shows a "checking your browser" screen, it waits up to 20 seconds for the check to pass before reading the page. Images, video and fonts are not downloaded, and one scan is stopped after 60 seconds.

The browser identifies itself as TechLookupBot/1.0 with a link to the page about our crawler, which explains its visits to site owners. Only the home page is loaded; the crawler does not follow links to other pages of the site.

Before loading the page, the crawler fetches the site's robots.txt through the same network safeguards as the page itself, HTTPS first and then plain HTTP, waiting up to 5 seconds, following up to five redirects and reading at most 500 KB. It follows the rules for TechLookupBot, or for * when there are none for it, as RFC 9309 describes. If they disallow the home page, the page is not loaded and the site is checked again about a month later; the stack we recorded before stays as it was, and is not counted as removed. A missing robots.txt (a 4xx answer such as 404) means no restrictions. A server error (5xx), a 429 "too many requests" or no answer at all means we do not scan on that attempt, which is recorded as an unreachable site and retried later. We keep each site's answer for up to 24 hours, or an hour when it could not be fetched.

A site owner who does not want their site scanned can also use the opt-out page: one DNS record proves control of the domain, after which we stop scanning it and delete what we hold.

What the scanner reads

  • the page's HTML after scripts have run, its visible text and its stylesheets;
  • the URLs of its scripts, the contents of its inline scripts, and the first 64 KB of up to 20 external scripts and stylesheets;
  • the HTTP response headers of the page and the cookies set while it loaded;
  • the addresses of the background (XHR and fetch) requests the page made;
  • its meta tags and the issuer of its TLS certificate;
  • the JavaScript variables and page elements that our fingerprints look for, such as a framework's global object;
  • the domain's DNS records: TXT, MX, NS, SOA and CNAME.

For the report we also keep the page title, description and language, the server's IP address, social profile links and email addresses published on the page, and the other sites the page links to, which we use to find new sites (see below).

How technologies are identified

What the scanner reads is matched against a catalog of fingerprints for more than 7,000 technologies in more than 100 categories. The catalog comes from webappanalyzer, the open-source community continuation of Wappalyzer's fingerprints, published under the GPL-3.0 licence. We record the exact upstream version we use, and keep our own corrections separately, each with the reason for it; most fix a fingerprint that matched sites that do not use the technology.

A fingerprint is a set of patterns: a script path, a header, a cookie name, a meta tag, a JavaScript variable and so on. Each pattern that matches adds its weight to the technology's score, capped at 100; most patterns carry full weight, and a fingerprint can give a weaker clue less. That score is a rule weight, not a measured probability, so reports do not show it as a percentage. Instead each technology carries a certainty from what matched: Confirmed when two or more independent kinds of evidence matched (say a header and a script URL), Likely when one kind did, Possible when the only match is one the fingerprint itself calls a weak clue, and Inferred when nothing matched directly and another detected technology implies it.

Where a pattern captures a version, from a file name such as jquery-3.7.1.js or a generator tag, the report shows it. A clean version number is preferred over build hashes, and if a later scan cannot read a version, the last one we saw is kept.

Some technologies are implied by others: a site running Next.js runs React, so React is added even if it left no trace of its own. Fingerprints can also require another technology to be present, or exclude one that cannot coexist with them.

For the platforms that publish one, the scanner also asks a documented endpoint after the page scan (WordPress's /wp-json/, Shopify's /cart.js, Magento's /magento_version, Ghost's and Joomla's manifests), within robots.txt. An answer is evidence of its own kind, listed as an "endpoint check", and often carries the version. An endpoint only ever confirms a technology the page already showed; it never adds one.

Every report has a "How we detected these" table listing, for each technology, its certainty, which kind of evidence matched (HTTP header, script URL, cookie, DNS record and so on, or which technology implied it) with a short excerpt of the match where we kept one, and when we first and last saw it. Excerpts never include cookie values, query strings or anything that looks like a token. When a scan could not read the page or the domain's DNS in full, the report says so above the list, so "not detected" is never read as "not used".

Technologies found only in DNS records other than CNAME, such as an email provider in the MX records or a verification record in TXT, describe services the organisation uses rather than what the website runs on. Reports list them separately, under "Services behind the domain", and not as part of the site's stack.

Which sites we scan

  • Public top-site lists. The Tranco top million, the Majestic Million, the Cisco Umbrella top million (only names that are a site's own domain, since most of that list is CDN and API hostnames) and Cloudflare Radar's top-million bucket. Each list is downloaded again once a month, on the first days of the month, so new entrants join the crawl.
  • Links between sites. When a scanned home page links to a site we do not have, the link is noted. A site is added once two different scanned sites link to it, within a daily limit.
  • Lookups. Any site a visitor looks up is scanned and joins the index.
  • The browser extension, if its user opts in. Sharing is off by default. When it is on, the extension sends the domain names of sites its user visits; our own scanner then scans them. The technologies the extension reports are never published.

Our index is still growing. Every count on this site is a count of the sites in our index, not of the whole web, and a technology popular on sites that are not on these lists will be under-counted.

How often sites are rescanned

  • Sites in the Tranco top 100,000: about every 14 days.
  • The rest of the Tranco top million: about every 60 days.
  • Sites with no Tranco rank: about every 90 days.
  • When a visitor looks a site up and our last scan is more than 7 days old, a fresh scan starts straight away.

A site that cannot be reached is retried later, with the wait doubling after each failure, up to 180 days. When a scan fails because of a problem on our side, it is retried the next day. A site whose robots.txt disallows us is checked again after about 30 days, whatever its rank.

How the numbers are counted

Websites using a technology is the number of sites in our index where the technology is currently detected. A site counts once, however many pages or clues show it. The counts are recomputed every hour. They include technologies found through DNS records, so an email provider is counted by the domains whose records point to it.

Share of a category divides the sites using a technology by the sites using any technology in that category. For example, a CMS's share is the part of the sites with any detected CMS that use it; sites with no detected CMS are not in the denominator. Shares within a category can add up to more than 100%, because one site can use two tools of the same kind.

Popularity by site rank counts the sites using a technology within the Tranco top 1,000, 10,000, 100,000 and million. These counts, and the usage-over-time charts, come from dated snapshots of the index.

The technology adoption report applies these counts to the Tranco top million, as shares of the sites whose last scan succeeded, in a snapshot that does not change after it is published.

Comparisons show each technology's count, its share of the category they share, and how many sites use both. The overlap percentage is the sites using both divided by the sites using the less common of the two.

First seen, last seen, removals and returns

First seen is the date our scanner first detected a technology on a site, not the date the site started using it. Last seen is the most recent scan that detected it.

A technology is marked as removed only after it is missing from two complete scans in a row, so a single scan that missed something does not count as a change. If it is detected again later, it is recorded as returned. Scans that hit a bot check or an error page count what they did detect, but not what they did not, and a DNS lookup that fails does not mark DNS-based technologies as removed. Each addition, removal and return is stored with its date, which is what the change history on reports shows.

How accurate detection is

We test detection against 229 sites whose real platform we know from the platform's own customer pages or from checking a documented endpoint by hand, never from another detection tool. Part of the set is held back and never used to tune a rule. The latest run, on 31 October 2026, covered the 205 sites we could capture:

  • Stores and CMS platforms: all found across 87 Shopify, WordPress, WooCommerce and Ghost sites, with no rival platform reported in their place.
  • Held-back sites: 94.5% of platforms found across 73 sites.
  • Frontend frameworks: 90.7% found across 57 sites.
  • Static site generators: 52% found across 35 sites. Jekyll often leaves no trace in the finished page.

This is a small sample, so read it as evidence, not a guarantee. A false match here means a rival platform reported on a site, not any wrong technology. Webflow, Wix, Squarespace, Magento, Drupal, analytics and payment tools are not measured yet. 24 sites could not be captured and 13 labels that looked out of date count neither way.

Known limitations

  • One page. Only the home page is scanned. A shop's checkout, a help centre on another path or a tool used only after sign-in is not seen.
  • Bot protection. Some sites show our scanner a challenge or an "access denied" page instead of the site. What we detect on it is real, often the protection service itself, but the rest of the stack is missing. Reports say so when the page we got looks like a block page.
  • No trace, no detection. Server-side software that leaves nothing in the page, its headers, cookies or DNS (a database, a back-office tool, most programming languages behind a proxy) cannot be detected.
  • False positives and negatives. A fingerprint can match a site that only mentions a product, and a site can hide or rename the clues a fingerprint relies on. A technology loaded only after a click or a consent banner may not appear in our scan.
  • Changes between scans. A report shows the site as it was on the scan date shown on it.

Reporting a wrong detection

If a report shows a technology a site does not use, or misses one it does, email hello@techlookup.app with the site's address and what is wrong. A corrected fingerprint applies to every site from its next scan.