The number we didn't ship

The most consequential thing in AskScope's scoring module is a null.

AskScope asks AI assistants the buyer questions your customers ask — "what's the best bookkeeping service in Austin," "X vs Y," "what should I use for this" — and reports whether you were recommended, mentioned, ignored, or described wrong. The obvious thing to build on top of that is a score. One number, 0 to 100, big type, trending arrow. Every sketch we made had one, because a single number is what a dashboard wants: it sorts, it charts, it fits in an email subject line, and it makes a $9/month product feel like it knows something.

We couldn't make it honest, so it isn't there. What's there instead is a set of shares, a confidence label, and — in the specific cases where nothing was actually measured — a dashed empty ring with the words "not measured" under it. That's the least impressive screen in the product and the decision we'd defend hardest.

Why one number doesn't survive contact with the thing being measured

Ask the same buyer question twice and you can get two different answers. Ask it on two different assistants and you're not even comparing the same kind of system: some answer from a live web index and return citations, some answer from model knowledge with no retrieval at all. Wait a week and the model has been updated underneath you.

A rank implies a stable ordering that somebody, somewhere, maintains. Nobody maintains this one. Any single number we printed would have been a summary of one afternoon's dice rolls, presented with the visual authority of a credit score.

So the unit of measurement in AskScope is the run, not the answer. Each question goes out repeatedly across surfaces, and the scores are plain shares of those runs:

• Visibility — the share of runs where you appear at all.
• Recommendation — the share of runs where you're actively recommended, not just named.
• Competitor gap — the share of runs where a competitor shows up and you don't.

That last one turned out to be the number people actually feel. "You appeared in 40% of answers" is abstract. "In 6 of 10 answers, a competitor was recommended and you weren't named" is a Monday morning problem.

Zero and "I didn't look" are different facts

Here's where the null comes in. Two of the four scores can refuse to produce a value:

const citationRuns = runs.filter((r) => r.citationCapable);

const citationScore =
  citationRuns.length === 0
    ? null
    : pct(
        citationRuns.filter((r) => r.ownedCitations > 0).length,
        citationRuns.length,
      );

Citation score is the share of answers that cited your own site — but only some surfaces can cite anything. An assistant answering from model knowledge has no sources to give you. If a scan only ran on surfaces like that, the correct output is not "0% of answers cited you." It's "this wasn't measured." Accuracy works the same way: AskScope only checks whether an AI described you correctly against key facts you supplied. No key facts, no accuracy score — not a zero, not a green checkmark, nothing.

The UI honours it all the way down. The gauge component takes value: number | null, and the null branch renders a dashed empty ring instead of a filled zero. Every table cell has a "not measured" state.

“It looks, at a glance, like a dashboard with holes in it. It is a dashboard with holes in it. The holes are the honest part.”

Confidence is a function of volume and consistency

The other half of the problem is that a share of ten runs and a share of a hundred runs look identical once they're a percentage. So every score ships next to a confidence label, computed from exactly two things — how many runs there were, and how consistent they were:

const rate = runs.filter((r) => r.brandMentioned).length / n;
const inconsistent = rate > 0.25 && rate < 0.75;

if (n >= 10) return inconsistent ? "medium" : "high";
if (n >= 4) return inconsistent ? "low" : "medium";
return "low";

The comment above it in the file says what the code is for: a 40–60% mention rate across a handful of runs is a coin flip, not a signal. If the results are all over the place, the confidence gets capped no matter how many runs you did. Volume alone doesn't earn certainty.

This is the part that costs something commercially. A free scan runs five questions across two surfaces — ten answers. Ten runs can reach high confidence, but only if the outcome is consistent across all of them. Land anywhere in the middle and the first thing a new user sees is a number with "medium confidence" stamped next to it, which is a strange way to open a sales relationship. We decided we'd rather explain a hedge than defend a number we know is soft.

What this costs, honestly

It costs the screenshot. A competitor's marketing can show one confident number going up and to the right; ours shows shares, a confidence chip, and occasionally a gauge that says nothing was measured. It costs a line in the pitch, too: we can't promise a rank, so we can't promise to improve one.

We're pre-launch, so we don't yet know how that lands with people who aren't us. The honest version of this post has no user reaction in it, because there are no users yet. Our bet is that a tool which tells you when it doesn't know is easier to trust the second week than a tool that never admits it — but that's a bet, and it's the main one the product is making. We've written the constraint into the terms of service, so it's not a mood we can quietly walk back later: share of runs, confidence label, trend over time, never a guaranteed "AI rank."

If you're measuring anything an LLM does

You don't need our product to apply any of this, and if you're evaluating model output — for search visibility, for internal quality scoring, for anything — three rules have earned their place.

1. 01Report shares over repeated runs, not a position. One answer is an anecdote. The percentage across ten identical asks is a measurement, and it's the smallest honest unit you can report.
2. 02Never let "we didn't measure this" render as zero. Give your data model an explicit "not measured" and make the UI carry it all the way to the pixel. If your type is number where it should be number | null, the lie is already baked in and no amount of caveat copy fixes it.
3. 03Attach confidence to the number and compute it from something real. Run count and consistency are enough. A percentage without a sample size is decoration.

You can do all three by hand. Ask an assistant the same buyer question ten times, tally how often you're named, note how much the answers wobbled, and you have a more honest read on your AI visibility than most dashboards will give you. The reason we built AskScope is that doing that every week, across four surfaces, while keeping the raw answers so you can prove what changed, is tedious — not that the measurement is secret.