BusinessMarkets, money & strategy
Research finds AI brand recommendations rarely repeat across runs, a problem for freelancers comparing AI visibility trackers
Research shows ChatGPT, Claude and Google's AI almost never return the same brand list twice, and cited sources overlap little between runs. For freelancers comparing AI visibility trials, prompt volume and engine coverage matter more than feature lists.
From the r/AISEOInsider (via last30days) article of November 1, 2026 and cited sources: In 2,961 prompts, each run 60 to 100 times, ChatGPT, Claude and Google's AI returned an identical brand list less than 1% of the time. [2]
SOURCES[2] Search Engine Journal: AI recommendations change with… searchenginejournal.com
AT Pulse research deskPublished · Updated · 4 min read · AI-assisted analysisAI-assisted, not human-edited. Methodology Corrections Who owns this site
- What happened
- In 2,961 prompts, each run 60 to 100 times, ChatGPT, Claude and Google's AI returned an identical brand list less than 1% of the time. [2]
- Original source
- r/AISEOInsider (via last30days) article, independent perspective
- Brief by
- AT Pulse research desk AI-assisted; publisher and contact on the desk page
- Studies
- arXiv 2311.09735 (Nov 2023) [3] date from arXiv ID
- When
- November 1, 2026 article published
- Covered
- Nov 3 edition
- Evidence
- 9 sources across 8 sites
- 1Still looking for answers for a AI visibility platformreddit.com
- 2Search Engine Journal: AI recommendations change with…searchenginejournal.com
- 3GEO: Generative Engine Optimization (arXiv 2311.09735)arxiv.org
- 4Quantifying Uncertainty in AI Visibility…arxiv.org
- 5Generative Engine Optimization at Scale…emergentmind.com
- 6OtterlyAI pricingotterly.ai
Reach: not measured yet
In 2,961 prompts, each run 60 to 100 times, ChatGPT, Claude and Google's AI returned an identical brand list less than 1% of the time. Order within a list was close to random. Which brands appeared was fairly stable, so how often a brand appears is a more useful measure than its rank. [2]
Repeated runs cited few of the same domains. Median overlap was about 0.29–0.31 on Gemini and about 0.50 on Perplexity. A ±2.5-point confidence interval took roughly 40–50 queries on Gemini and 150 or more on SearchGPT. That suggests a 15-prompt plan cannot reliably separate competitors only a few points apart. [4] [6]
Plans differ sharply in sampling depth and engine coverage. Otterly offers 15 prompts across four engines for $29 a month, with Claude and Gemini as paid add-ons, plus a free trial. Peec starts at $95 a month for 50 prompts, includes competitor tracking on every plan, and limits Claude and Perplexity to Enterprise. Whether Peec offers a trial is unclear: its pricing page lists none, but a third-party roundup reports a 7-day trial. [6] [7] [8]
In more than 100,000 responses across five engines, small brands appeared in 11% of answers versus 73% for global brands. The data came from a vendor's tracking platform and the study is observational, but it points to setting baselines before claiming any optimization gain. [5]
rank order is close to random, appearance frequency is steadier, and tight confidence intervals take dozens to hundreds of queries per engine. Implication: low-prompt plans and single rank positions can make random noise look like competitive change.
Executive brief
The Reddit post is a freelancer asking which AI visibility tools offer competitor tracking and free trials. The bigger issue is that the data these tools sell is very noisy. A SparkToro/Gumshoe study found that ChatGPT, Claude and Google's AI returned the same brand list less than 1% of the time across repeated runs of a prompt (Search Engine Journal). Before picking a vendor, freelancers should check sampling depth: how many prompts a plan runs and how often. No other coverage of the post exists.
Read the full section
The Reddit post is a freelancer asking which AI visibility tools offer competitor tracking and free trials. The bigger issue is that the data these tools sell is very noisy. A SparkToro/Gumshoe study found that ChatGPT, Claude and Google's AI returned the same brand list less than 1% of the time across repeated runs of a prompt (Search Engine Journal). Before picking a vendor, freelancers should check sampling depth: how many prompts a plan runs and how often. Plan features matter less. No other coverage of the post exists.
What changed and event timeline
GEO is defined academically
Princeton-led researchers introduced "Generative Engine Optimization" and GEO-bench. They reported visibility gains of up to 40%, varying by domain. The paper was revised for KDD 2024 ().
Study finds AI recommendation lists rarely repeat
Rand Fishkin and Patrick O'Donnell ran 2,961 prompts, 60–100 times each. Brand lists almost never repeated, but which brands appeared was fairly stable ().
Statistical framework for visibility measurement
Sielinski measured citation overlap between repeated runs on Perplexity, SearchGPT and Gemini. Overlap was low, and confidence intervals spanned 3–7 percentage points ().
Large observational GEO dataset
Kumar analyzed more than 100,000 responses across five engines, collected with the Ranqo platform. Small brands appeared in 11% of responses, versus 73% for global brands ().
Freelancer asks for trial-friendly tools
The linked post asks for platforms with competitor tracking and free trials, and for what has worked for other freelancers ().
Capabilities and access
- Otterly.ai: Lite costs $29/mo for 15 prompts and Standard $189/mo for 100. Free trial with month-to-month billing (Otterly pricing).
- Peec AI: Starter costs $95/mo for 50 prompts and 3 models. Every plan includes competitor tracking. Claude and Perplexity are Enterprise-only.
- No vendor reviewed here says which model version, such as GPT-5 Search, sits behind each tracked surface on lower tiers.
Read the full section
- Otterly.ai: Lite costs $29/mo for 15 prompts and Standard $189/mo for 100. Plans cover ChatGPT, AI Overviews, Perplexity and Copilot; Claude and Gemini are paid add-ons. Free trial with month-to-month billing (Otterly pricing).
- Peec AI: Starter costs $95/mo for 50 prompts and 3 models. Every plan includes competitor tracking. Claude and Perplexity are Enterprise-only. The pricing page lists no trial (Peec pricing), although a third-party list says there is a 7-day trial (Gracker).
- No vendor reviewed here says which model version, such as GPT-5 Search, sits behind each tracked surface on lower tiers.
Technical analysis for researchers and developers
- Sampling variance is the main problem. Median Jaccard overlap of cited domains between repeated runs was 0.29–0.31 on Gemini and about 0.50 on Perplexity.
- Measure frequency, not rank. Order within a list is close to random. How often a brand appears across many prompts is more stable (SEJ).
- Sentiment signals are noisy. Sentiment flipped 6.7× more often than mentions did, which mixes real change with classifier error (Kumar).
Read the full section
- Sampling variance is the main problem. Median Jaccard overlap of cited domains between repeated runs was 0.29–0.31 on Gemini and about 0.50 on Perplexity. Getting a ±2.5-point confidence interval took about 40–50 queries on Gemini and 150 or more on SearchGPT (Sielinski).
- Measure frequency, not rank. Order within a list is close to random. How often a brand appears across many prompts is more stable (SEJ).
- Sentiment signals are noisy. Sentiment flipped 6.7× more often than mentions did, which mixes real change with classifier error (Kumar).
- Implication: a 15-prompt plan probably cannot tell apart competitors that sit only a few points apart. This is an inference from Sielinski's sample sizes.
Claims and evidence
- "Up to 40%" GEO visibility lift: peer-reviewed, but measured on GEO-bench, not on live commercial engines (arXiv).
- Less than 1% list repeatability: large study, but not peer-reviewed, and participants' settings were not standardized (SEJ).
- Tier gaps (73% / 44% / 11%): the data come from a vendor platform (Ranqo), brand tiers were coded by hand, and the study is observational (Kumar).
Read the full section
- "Up to 40%" GEO visibility lift: peer-reviewed, but measured on GEO-bench, not on live commercial engines (arXiv).
- Less than 1% list repeatability: large study, but not peer-reviewed, and participants' settings were not standardized (SEJ). Gumshoe, whose CTO co-authored it, sells AI visibility tracking.
- Tier gaps (73% / 44% / 11%): the data come from a vendor platform (Ranqo), brand tiers were coded by hand, and the study is observational (Kumar).
- "Best platform" rankings: many are written by vendors that rank themselves, including Gracker and Trysight.
Context and prior work
GEO grew out of academic work on optimizing content for LLM-generated answers (arXiv). By 2026 a crowded tool market had followed: one roundup counts at least seven platforms with free trials, priced from about $20 to $99 or more per month (Gracker). Measurement research is now catching up.
Read the full section
GEO grew out of academic work on optimizing content for LLM-generated answers (arXiv). By 2026 a crowded tool market had followed: one roundup counts at least seven platforms with free trials, priced from about $20 to $99 or more per month (Gracker). Measurement research is now catching up. Its common finding is that each engine behaves like its own market, and results from one engine do not generalize to others (Kumar).
Limitations, safety and contested findings
- The three studies limit their own conclusions. The SparkToro work was not peer-reviewed. Kumar's study makes no causal claims.
- Kumar found corporate sites drew about 78% of citations (Emergent Mind).
- Fishkin's view that AI "ranking position" tools are "full of baloney" is disputed by vendors who sell ranking metrics (SEJ).
Read the full section
- The three studies limit their own conclusions. Sielinski used only three consumer topics, ChatGPT-generated queries and a nine-day window (arXiv). The SparkToro work was not peer-reviewed. Kumar's study makes no causal claims.
- The sources disagree on citations. Kumar found corporate sites drew about 78% of citations (Emergent Mind). Other summaries of the same paper say brand-owned domains were only 2.9%, but the wording and definitions are unclear.
- Fishkin's view that AI "ranking position" tools are "full of baloney" is disputed by vendors who sell ranking metrics (SEJ).
Business and practitioner implications
- Pick trials by prompt volume and engine coverage. Otterly's $29 tier covers four engines but only 15 prompts (Otterly).
- Report share-of-voice ranges to clients, not single rank positions. Competitor gaps of 3–7 points can be noise (Sielinski).
- Check which engines a plan actually tracks. Claude and Perplexity are Enterprise-only at Peec (Peec).
Read the full section
- Pick trials by prompt volume and engine coverage. Otterly's $29 tier covers four engines but only 15 prompts (Otterly).
- Report share-of-voice ranges to clients, not single rank positions. Competitor gaps of 3–7 points can be noise (Sielinski).
- Check which engines a plan actually tracks. Claude and Perplexity are Enterprise-only at Peec (Peec).
- Small brands start from low visibility. Set baselines before claiming any GEO uplift (Kumar).
The source trail.
Sources (9)
Still looking for answers for a AI visibility platform
Feed excerpt only.
reddit.com