Everyone wants to know where they rank in ChatGPT.
That question makes sense. Buyers are using AI search tools to shortlist vendors, pick products, and research consultants before they ever click a link. If you're not showing up in those conversations, you're not in the running.
The problem is the way most agencies are trying to answer it.
They're selling dashboards. Tracking tools. Baseline reports that tell you your "AI visibility score." It sounds rigorous. It's mostly theater.
Here's why.
The Uncomfortable Truth About AI Outputs
Language models don't work like search engines. When Google returns results for a query, those results are deterministic. You can run the same query a hundred times and get the same ranking order.
AI tools don't work that way. They're probabilistic. The same prompt, asked twice in the same session, can produce different answers. Ask it tomorrow and it might change again. Run it across different tools and the results can be completely unrelated.
Rand Fishkin tested this at SparkToro. He and Patrick O'Donnell from Gumshoe.ai ran nearly 3,000 prompts across ChatGPT, Claude, and Google AI to measure consistency. The finding: less than a 1-in-100 chance of getting the same brand list twice from the same prompt. Getting the same list in the same order? Under 1 in 1,000. Fishkin put it plainly: "any tool that gives a 'ranking position in AI' is full of baloney."
That's not a tracking problem. That's a fundamental property of how these models work. Anyone selling you a clean AI visibility score based on a handful of prompt runs is selling you a number they made up.
The Parenting Analogy I Keep Coming Back To
I have two kids. And one thing fatherhood has taught me is that input and output are related, but you can't predict the outcome with precision. You can be consistent with your values, your time, your attention. You can build the best environment you know how to build. But there are factors outside your control, and the child makes their own choices.
AI visibility works similarly. You can't control what a language model outputs on any given prompt. What you can control is what goes into it. The information the model is trained on. The way your brand is described, cited, and associated across the web. The consistency and clarity of the signals you're sending.
The more consistent and coherent the information you put into the system, the more likely that information shapes the output over time. Not guaranteed. Not trackable prompt by prompt. But directionally, it works.
What You Can Actually Measure
Here's what has real signal:
What your brand and competitors are publishing and getting cited for. You can track where your name appears, in what context, on which platforms. Not inside a chatbot's response. In the sources those chatbots pull from.
How you're described in the places AI tools reference. Google results, Reddit threads, review platforms, industry publications, YouTube comments. These are inputs. LLMs learn from them. Your brand reputation in these places shapes what AI tools say about you when nobody's watching a dashboard.
The gap between your content and the conversations your buyers are having. You can identify the actual likely prompts your target customer would type into ChatGPT or Perplexity, then look at how close your website and your competitors are to those topics. That gap tells you something useful.
One example that stuck with me: a prospect in the fitness industry reached out because my previous agency work had tied my brand to that space. They'd found me through associations I hadn't actively managed. Those associations came from old mentions, old partnerships, old context that was still floating around. That's what AI models pull from. Not your latest campaign.
What Most Agencies Won't Tell You
Tracking prompts and answers like you track keyword rankings is not a viable strategy. The data isn't consistent. The methodology isn't reproducible. Agencies offering precise AI visibility scores based on small prompt samples are telling you a story that the underlying technology doesn't support.
That's not a minor caveat. Pretending to have data you don't have is a short-term play. It might win a sales conversation. It loses client trust when the numbers can't be explained or replicated.
The honest version looks like this: you audit what goes into the AI, not just what comes out. You analyze your website, your competitors, and the gaps between how you're described and how your buyers search. You build the inputs. You measure the inputs. You let the outputs do what probabilistic systems do.
A Practical Starting Point
Three things worth doing now:
- Analyze your website and competitor sites against the way your target persona actually talks. Not industry jargon. The language they use in a ChatGPT prompt at 9pm when they're trying to solve a problem.
- Map where your brand is mentioned, on what platforms, and in what context. Then do the same for your top two competitors. The gap between those maps is your actual visibility gap.
- Look at what comes up in Google when someone searches your brand name. That's still one of the primary inputs for how LLMs understand who you are. A bad reputation signal in Google doesn't stay in Google.
This work is less exciting than a dashboard. It's more useful.