The AI Recommendation Index measures what AI assistants actually recommend when buyers ask. Every number on an Index page traces back to the process below — and every page discloses its own sample size and window. The measurement parameters on this page (models, gates, guardrails) are read live from the pipeline that collects the data, not hand-maintained copy.
Each category carries a fixed set of buyer-shaped prompts — 6–8 by design; the Statistical robustness section reads the live distribution: the questions real buyers type, like “best [category] for [audience]”, “[category] alternatives”, or “what should I use for [job]”.
Prompts are strictly neutral — they never contain product names, never steer toward or away from any vendor, and stay stable between samples so trends compare like with like.
Where prompts come from: each category’s set is written from its slot in the category taxonomy plus kit-derived buyer questions — the questions a buyer in that category actually types into an engine — and each prompt is tagged with the buying intent it represents (best-in-category, alternatives, problem-solving, …). Intent tags are published with the per-intent breakdowns on category pages.
Every prompt is hashed and versioned. Each stored run carries a stable hash of the exact prompt wording it answered, stamped alongside the extraction version that parsed it. The hash stays with the run forever, so any published number can be traced to the precise wording that produced it.
Editing prompts mid-series: prompt wording is frozen while a series runs. When a prompt does have to change (ambiguity, category drift), the new wording gets a new hash — historical runs keep the hash of the wording they actually answered, per-prompt analyses group by hash, and the reworded prompt starts a new series instead of being silently spliced into its predecessor’s. Nothing is relabeled after the fact.
Categories are measured across up to 5 engines via their official APIs, with web search enabled where the engine supports it. An AI engine is not one thing — answers depend on the exact model behind it, so the Index discloses the model measured per engine:
Each category page shows per-engine shares separately — engines disagree, and that disagreement is part of the data. Not every category runs on every engine; the engines measured are listed per category.
Sampling runs continuously, spread across the batch cycle to avoid time-of-day artifacts. At the current measurement budget, each engine samples each category roughly monthly — cadence scales with budget, and every published number discloses exactly how many sampled answers it rests on.
Published numbers aggregate a 28-day (4-week) rolling window; trends compare that window against the 28 days before it.
Categories with fewer than 20 sampled answers in the current window are marked “insufficient data” and publish no rankings at all. Just above that floor the confidence interval is very wide — read the interval, not the point estimate, and treat the ranking as provisional until the sample grows.
recommendation_share = % of sampled answers in the window that recommend the product · avg position = mean rank when answers are ordered lists
AI answers are stochastic — the same prompt to the same engine can return a different list — so published rates are estimates over repeated samples, not single draws. Each prompt-engine pair is sampled 1× per cycle — read live from the configured sampling parameters. If the weekly budget projection is exceeded mid-cycle the runner degrades the rest of that cycle to single draws, so realized sampling can be lower than the value shown here. Per engine:
An engine’s N can be lower than the global value when its per-run cost would multiply the most expensive line first; a global value of 1 means repeated sampling is switched off and every number is a single draw — whatever is running is what this section shows. Samples from one prompt-engine cycle are tagged as one sample group, so they are never double-submitted and never mistaken for independent prompts. Sample sizes on category pages count answers analyzed, so repeated samples legitimately increase them. During ramp-up, only the top 50 categories by exposure sample above 1× — every other category runs single draws until the sampling budget is proven.
Cadence and rotation: sampling runs continuously under an engine-aware rotation — categories are prioritized by their most-behind engine, and an engine that already covered a category inside the current freshness window is skipped so calls go to the laggards, not to keeping already-fresh pairs artificially fresh. At the current budget each engine reaches each category roughly monthly; cost caps are enforced before submission, and when a cycle’s projected cost would exceed the remaining budget the runner degrades to single draws for the remainder instead of silently overspending or dropping coverage.
Production-default parameters, on purpose: samples are issued with each engine’s production-default request parameters — no temperature tuning, no parameter sweeps. We measure the answer distribution real users actually get, not a distribution tuned to be convenient to measure; two samples differ only because the engine’s own randomness differs.
A trend is published only when the two windows are statistically comparable. A trend is suppressed — shown as an em-dash (“not enough comparable data”), never as a direction — when any of these fail:
Trends that pass carry a confidence grade: high, or low (any engine under 20 runs in either window — rendered muted). Share changes within ±1 percentage point are reported as flat; below 50 sampled answers one answer is worth multiple points of share, so the flat band widens to ±5pp.
The Index is a sample, not a census — so here are the questions a statistically literate reader should ask before trusting a number on it, answered from the live corpus where the answer is measurable and answered “not yet” where it isn’t.
1–8 prompts per active category (median 7), across 298 categories with a written prompt set. Prompts are neutral, intent-tagged, and stable between samples — see Prompt design.
Each run is one sampled answer: one prompt, one engine, one call, no conversational context carried between them. Runs are not all independent, though: each prompt-engine pair is sampled several times per cycle, so those runs share a single draw of the prompt. We report total answers analyzed; every statistical gate — the publishing gate below and the confidence intervals — counts independent prompt executions instead, so repeated sampling can never buy a category a narrower interval or an easier path past the gate. In the current 28-day window, categories carry 2–72 runs (median 28) — 287 categories sampled, 235 of them at or above the 20-run publishing gate. Every category page states its own sample size; nothing is published off an undisclosed n.
Up to 5 engines via their official APIs, and the exact model measured per engine is read live from the collection pipeline and listed in Engines & measured models. Model changeovers are logged and never spliced into one series — the models are part of the measurement, so a change to them is a change to the instrument.
Every ranked entry publishes a 95% Wilson score confidence interval on its recommendation share, computed on that entry’s mention-runs over the category’s sample size. Wilson rather than the normal approximation because shares here routinely sit near zero on small samples, where the normal interval leaks below 0 and undercovers.
One honest caveat, and one correction to it. Repeated samples of the same prompt are clustered, not independent, so counting them as separate observations would narrow every interval by roughly the square root of the sample count — we correct for that by computing the interval over independent prompt executions rather than raw answers. What we do not correct for: the prompt set itself is fixed, so the interval describes uncertainty given these questions, not uncertainty about which questions buyers ask. True uncertainty is therefore still somewhat wider than the stated band. Prompt randomization is on the not-yet-measured list on this page.
p = x/n · center = (p + z²/2n) / (1 + z²/n) · half = z·√( p(1−p)/n + z²/4n² ) / (1 + z²/n) · z = 1.96
Category pages render the interval next to the share, so two entries whose intervals overlap can be read as what they are: not distinguishable at this sample size. Each entry also publishes a first-pick rate — the share of its mention-runs where it led the answer’s list — because being named at all and being the top recommendation are different claims. Trends carry a separate comparability gate (see Trend guardrails).
Not yet. Prompt wording is fixed per category, which buys clean trend comparability — the same question asked over time — at the cost of measuring only that phrasing. A different phrasing of the same buying question can surface a different set of products, and the Index currently cannot tell you how much. Rotating a randomized paraphrase set alongside the fixed set, so phrasing sensitivity is itself measured, is on the roadmap.
No. Prompts carry the audience wording a category’s buyers use (“for freelancers”, “for small teams”), but there are no simulated user profiles, no memory, and no account history behind a run. What an engine recommends to a logged-in user it has learned from is outside what the Index measures.
Not measured. Runs are issued from a single region without locale hints, so every published number is one geography’s view. Engines can and do localize recommendations; the Index does not currently quantify that, and no number on it should be read as a global average.
AI answers are probabilistic. The same prompt to the same model on the same day can return a different list, which is exactly why nothing here rests on a single answer and why every share ships with an interval instead of a decimal that looks more certain than it is. Engines are also a moving target: vendors change models, retrieval and grounding behavior shift without announcement, and a share can move because the instrument moved rather than because the market did. The guardrails above exist to keep that from being published as a finding.
So: the Index measures what a set of engines recommended, for a fixed set of neutral prompts, in one region, over a 28-day window — not what “AI thinks” and not what any individual user will see. Where a number is too thin to defend, we publish an em-dash instead. The live statistics behind this section are on /data, and if any of it doesn’t hold up we would rather hear it: hello@orbator.io.
Engine vendors retire and replace models. When the measured model behind an engine changes, the Index never splices the two into one series: published aggregation is restricted to the new model’s runs, and trends for that engine are suppressed or flagged until the new model has accumulated a comparable window. The models in the Engines section above are read live and always show what is measured right now.
| Date | Engine | Change | Status |
|---|---|---|---|
| Jul 21, 2026 | CLAUDE (Anthropic) | claude-haiku-4-5 → claude-sonnet-5 | announced — takes effect when the environment override lands |
| Jul 21, 2026 | CHATGPT (OpenAI) | gpt-4o-mini → chat-latest | announced — takes effect when the environment override lands |
Every product mentioned in an answer is extracted and resolved against a canonical registry, so “HubSpot”, “Hubspot CRM” and “hubspot.com” count as one entity. Resolution matches by domain first, then by name and known aliases; cited URLs serve as a confirmation signal.
Generic phrases (“CRM software”, “the tool”) are filtered and never become entities. Newly seen entities and suspected duplicates are auto-flagged for human review; confirmed duplicates are merged with their history re-pointed, and confirmed junk is removed from the rankings.
Mention detection. A product counts as mentioned only when the answer names it or cites its domain. Names match on whole-word boundaries against the registry’s canonical names and aliases — never substrings — and names shorter than three characters are never matched in prose (their domain still counts). Dictionary-word product names additionally require corroboration — exact cased spelling, a ranked-list position, or the product’s own domain cited — per the ambiguous-name rule in Junk handling.
Position. When an answer is an ordered list, each mentioned product’s rank in it is recorded; average position and the first-pick rate derive from those ranks, and answers that aren’t lists contribute mentions without positions.
Mention classes. Every mention in every run is classified four ways, and the class decides what it may count toward:
The full four-way breakdown is published per entry, so a share can always be decomposed into what produced it. In effect since extraction v6 (Aug 11, 2026) — see the Methodology changelog for the dated rule history.
Word frequency is not a recommendation, so the pipeline enforces rules that keep junk out of published shares:
See something wrong — a misresolved product, a junk entity, a number that doesn’t hold up? Corrections: hello@orbator.io. We would rather pull a number than publish a wrong one.
Every day, each published category’s ranking is frozen as a dated, append-only record. Nothing in that archive is ever edited: a correction appends a new revision alongside the original, with a note saying what changed, so the record of the record survives. Each entry discloses whether it was captured live that day or reconstructed later from stored runs, and which extraction rules produced the sightings underneath it.
One policy decides what may leave the archive, and every public page and API endpoint serving answer content is routed through it — the rules are enforced in one place rather than restated in each surface:
Retention: the runs and citations that make up the archive are kept indefinitely — they are the record. Full answer texts are retained for at least 24 months and then reviewed.
The rules that turn answers into published numbers — extraction filters, corroboration requirements, metric definitions, sampling design — improve as we find and fix measurement bugs. Every change is disclosed here, dated, in one changelog: series breaks and metric redefinitions are announced, not discovered. Extraction-rule changes bump the extraction version; every stored sighting is stamped with the version that produced it, and a CI guard fails any version bump that lacks a dated entry below — this changelog cannot silently rot. Current extraction version: 8.
Directories and lead marketplaces are no longer reported as competitors when a local business is measured. Until now the aggregator filter only ran on categories the Index itself had flagged as local, so a scan started from a customer's own product — which carries no Index category — kept listing Houzz, Angi, Thumbtack or HomeAdvisor alongside real rivals. The filter now follows the business, not the category, and the taxonomy gained the home-services aggregators it was missing (Houzz, Angi/Angie's List, HomeAdvisor, Thumbtack, Porch, Nextdoor, BBB).
Stored sightings carry the 4-way classification as their role. Recommendation share counts explicit recommendations and bare ranked-list placements (being placed on a list is a recommendation surface); warnings ("avoid X") and passing context mentions never count. The full mention breakdown is published per entry.
Every run now records a 4-way mention classification — recommended, listed, neutral mention, or warned against — so a warning or a passing mention can never be counted as a recommendation anywhere downstream.
Published rates become estimates over repeated samples instead of single stochastic draws: N samples per prompt-engine per cycle (cost-capped, top categories first), per-entry volatility (within-cycle disagreement + across-cycle movement) computed in aggregates, and ±error bars with stability badges in the product dashboard. The methodology page gains sampling design, prompt provenance, parsing rules, and 'what we cannot see' sections, plus this changelog.
Consumer platforms and marketplaces (Yelp, TripAdvisor, OpenTable, DoorDash, …) excluded as recommendations in software categories — they are sources answers cite, not products — except in the categories where they are the ranked product (e.g. OpenTable in reservation software). Platform domains can no longer be captured by, or credit sightings to, a named entity ("Yelp API").
Every ranked entry now publishes a 95% Wilson score confidence interval on its recommendation share, computed over the category's sample size — two entries whose intervals overlap are not distinguishable at that sample size, and the pages say so.
Dictionary-word product names (ChatBot, Writer, Wave) require corroboration — exact spelling, a ranked-list position, or their own domain cited — and context-only mentions ("works for Amazon sellers") no longer count toward recommendation share.
AI assistant products (ChatGPT, GitHub Copilot, …) re-admitted as valid entities — only in categories that rank assistant products, where they belong on the leaderboard.
The v2 source/platform exclusions are enforced at the single storage chokepoint, closing paths where an excluded source could still be stored as a sighting.
Review sites, analyst firms, social platforms, and search infrastructure excluded from rankings — they are where answers look things up, not recommendations.
Entity extraction from AI answers ships with the Index: products named in answers are resolved against the canonical registry, generic phrases are filtered, no source/platform filters yet.
A methodology that only lists what it measures is marketing. These are the things this method cannot observe — actual blind spots, not caveats:
The Statistical robustness section answers the related questions (randomization, personas, geography) with the same rule: an explicit “not yet” over a vague reassurance, every time.
Orbator customers are badged on Index pages for disclosure. Customer status does not affect measurement — the prompts, sampling schedule, extraction, and ranking math are identical for every product, customer or not. Rankings cannot be bought, and no one can pay to be removed.
The Index exists because we sell AI-visibility tooling — that is exactly why the measurement itself has to be, and is, untouched by who pays us.
Every category is downloadable as CSV from its page and queryable via the free JSON API (/api/index/categories; the measurement parameters on this page at /api/index/methodology; docs at /developers). Free to use with attribution to orbator.io. Questions or corrections: hello@orbator.io.
© 2026 Orbator. All rights reserved.