Why AI Answers Change From Run to Run, and How to Read the Data

Ask an assistant the same question twice and you may get two different lists of brands. That unsettles people new to AI visibility tracking, but it's expected, and it's manageable if you read the data correctly.

Four sources of variation

1. Sampling. Language models generate text by sampling from probabilities. Even with identical inputs, wording and sometimes content differ.
2. Retrieval. With web search enabled, the pages retrieved depend on a search index that changes daily, as well as on how the model phrases its searches.
3. Model updates. Providers update models and search systems frequently, sometimes changing behaviour noticeably overnight.
4. Personalisation. Consumer apps can use a signed-in user's history, location and settings. API calls, which Citewise AI uses, are not personalised, which makes them a steadier baseline.

Reading noisy data

• Use many prompts. One prompt flipping between "mentioned" and "not mentioned" says little. Visibility across 20 prompts is far more stable.
• Look at trends, not points. Compare several runs. A change that persists for three or more runs is far more likely to be real.
• Compare engines separately. A drop in one engine and not others usually points to that engine's update or retrieval.
• Use share of voice. If every brand's mentions fall together, the engine probably changed its answer format; your relative position may be unchanged.
• Read the answers. When a number moves, open the answers and the cited sources. The explanation is usually there.

A simple rule of thumb

Treat a change as real when it is consistent across several runs and several prompts, and you can find a plausible cause in the answers or sources. Everything else is weather, not climate.

Citewise AI charts visibility, citation rate and share of voice per run so you can see the trend at a glance. Read more in our methodology.