Back to docs

CTC Reference Set

The real pages a Total CTC is ranked against.

Since 21.0, the Total CTC that cognitive_effort reports is a percentile. It ranks the page's raw cognitive transport cost for a persona among the raw costs of real pages. 0.5 is the median of the reference, 0.8 is costlier than 80% of it, and 1 is at least as costly as the costliest page in it. The reference set is that collection of real pages. This page describes how it was collected and what is in it.

What is in it

The current reference set was crawled on 2026-10-24. Every page was loaded at a 1920x1080 page size and measured twice: at viewport scope (the first screen) and at full-page scope (the whole document). A viewport result is ranked against the viewport measurements and a full-page result against the full-page ones. referenceScope in the response says which was used.

Outcome Pages
Attempted 771
Graded (measured, part of the reference) 514
Blocked (a challenge page, an error status, an access-denied page or a near-empty page, on every route tried) 95
Error (timed out, TLS certificate rejected, or failed while being measured) 32
Excluded by a frame filter (24 redirected off-site, 1 gambling content) 25
Not fetched: disallowed by the site's robots.txt 105

Only graded pages are part of the reference. A page that did not load has no measurement, so it is not counted as a cheap page.

Three sampling frames

A list of well-known sites alone would rank everything against large brands. Pages are therefore drawn from three frames, and each frame supplies about a third of the graded pages.

  • Brand. The benchmark brand list: 199 well-known sites, each with an industry label, in a seeded random order.
  • Tranco. A seeded random draw of 60 domains from each of four rank bands (1-1,000, 1,001-10,000, 10,001-100,000 and 100,001-1,000,000) of the dated Tranco top-sites list 746PX. Adult and gambling domains, and domains already in the brand frame, are excluded and redrawn.
  • Public. Government and nonprofit sites: CISA's .gov registry of federal domains (a seeded sample of 45), the state government site USAGov lists for each of the 50 states, US cities of 400,000 or more people that have an official website (from Wikidata), and notable US nonprofits with 15 or more Wikipedia sitelinks (a seeded sample of 60, from Wikidata).
Frame Sites listed Sites crawled Graded pages
Brand 199 90 171
Tranco 240 177 170
Public 191 70 173

Six page types

From each site's landing page, the crawl followed the links a first-time visitor could follow to reach up to five more pages: a form (sign-up, then login, then contact), a search (the GET the site's own search form sends, or the search address the site publishes, with the query "help"), an article, a help page and a cart. Forms were loaded but never submitted. The only exception is a search, which is a GET.

Page type Brand Tranco Public All
Landing 56 56 49 161
Form 31 25 34 90
Search 13 23 21 57
Article 36 34 36 106
Help 32 28 32 92
Cart 3 4 1 8
All 171 170 173 514

How it was crawled

  • Quotas. Sites were taken in frame order, and a frame took no new site once it had 170 graded pages. Each inner page type was capped at 40 per frame.
  • Direct first. Every page robots.txt allowed was first fetched directly. A page whose direct attempt was blocked was retried once through a residential proxy. Government and military (.gov, .mil) pages were never proxied, so the 15 pages blocked on the direct route alone are all .gov pages. 114 pages were retried through the proxy, and 28 of them loaded and were graded.
  • robots.txt honoured. robots.txt was read for every fetch. 105 pages it disallowed were never fetched, and every graded page was allowed.
  • Every attempt is published. The outcome, route (direct or proxy) and reason for each of the 771 attempted pages ship with the package, so you can see what was attempted, what failed and why. Query parameters a site minted for the crawler's own session (an OAuth state, a challenge token, a per-visit id) are removed from the published URLs.

How the rank is computed

  • Unweighted. Every graded page counts once. The quotas balance the frames and page types, and the curve does not reweight them. "Costlier than N% of the real pages we measured" therefore means exactly what it says.
  • Pooled over the 25 built-in personas. Each page is costed for all 25 built-in personas (general, accessibility and emotional), each on a first visit. That gives 514 pages x 25 personas = 12,850 costs at each scope. Every persona is ranked against the same pool. One curve per persona would rank each persona against itself, which would erase the persona differences the tool exists to report.
  • A smooth percentile. The curve is the empirical distribution of those costs, interpolated linearly between the measured values. A cost below the cheapest reference cost is 0, and one above the costliest is 1.
  • Recomputed from raw measurements. The reference stores what was measured on each page (counts, text-length histograms, geometry), not scores. The curve is rebuilt from those measurements under the current model, so a change to the model moves the yardstick with it.

The 95% interval

A different draw of real pages would place the same raw cost at a somewhat different percentile. totalCTCInterval measures how much. The pages are resampled with replacement 1,000 times, with a fixed seed so every run gives the same answer, and each resampled page keeps all 25 of its personas. The interval is the central 95% of the cost's percentile across those resamples, and it appears in the rank sentence in the form "(514 pages x 25 personas, viewport scope; 95% interval L-H%)". The interval shows how firmly the reference fixes the rank. It says nothing about whether the cost itself is right.

Bands

The band field puts the percentile into one of four bands: below 0.5 lower half, 0.5 to 0.8 upper half, 0.8 to 0.95 high, and 0.95 and above top 5%. These cut points are conventions that were chosen, not fitted to data.

What a rank does not tell you

The cognitive transport cost has not been tested against human data. A rank tells you where a page sits among real pages under this model. It does not tell you how a person experiences the page, and it is not an abandonment prediction (abandonment risk is reported separately). See Cognitive Optimal Transport for the status of the model.

The data

  • Outcome list and measurements: src/data/ctc-reference-v2-2026-10-24.ts. CTC_REFERENCE_META holds the frames, quotas and outcome counts, CTC_REFERENCE_PAGES holds every attempted page, and CTC_REFERENCE_RAW holds the graded pages' measurements.
  • Crawl code: research/ctc-reference/, which includes the frames file, the crawler and the assembler.

From the Blog