Monday's dashboard says your brand was cited in 23 percent of tracked answers. Tuesday says 17. Wednesday says 30. Nobody touched a page, changed a prompt or published anything. Did something happen? Most likely, no. Each of those numbers is one pass over your question list, and AI answers are not the same every time you ask. A daily check can tell you whether your tracking still works and whether something large has broken. It is a poor way to tell whether a small change is real.
This guide explains what a once-a-day run can and cannot tell you, why the number of runs matters more than the number of days, how many runs it takes to confirm a change, and a schedule that gives each job the cadence it needs. A planner spreadsheet is included.
What one daily run is
When you check AI search once a day, each question on your list gets asked once, and the tool records whether you were cited. That is a sample of one answer per question.
AI answers vary from run to run even when you do everything the same. In a study of five language models, Atil and colleagues repeated identical prompts ten times with the randomness setting at zero and found that accuracy still varied by up to about 15 percent across runs. That study used benchmark tasks, not search products, so it does not tell you how much your own citations move. It does show that "same input, same output" cannot be assumed.
For AI search specifically, a 2026 statistical paper by Sielinski sampled three generative search platforms across three consumer product topics, both daily over nine days and every ten minutes. Its abstract reports that citation distributions varied substantially across repeated samples, that many apparent differences between domains fell within the measurement noise, and that single-run metrics give a misleadingly precise picture. We read the abstract, not the full paper. The scope is small: three platforms, three topics and short windows. But the direction is clear enough to plan around.
Our own tracking piece on model variability in AI citations covers why the same question can return different sources, and the guide to AI search volatility covers what that does to your numbers over time.
How noisy is a daily number?
Here is a simulation, so you can see the noise without waiting a month for it. Suppose you track 30 questions, run each once a day, and your true chance of being cited on any question is 20 percent. Halfway through, a real improvement lifts it to 30 percent.
Figure 1. A simulated 30 days of one run per question per day. The daily readings jump around; the seven-day average follows the real change with a lag.
Two things stand out. The daily dots swing by ten points or more with nothing changing. And the real step on day 16 is hard to spot in the dots, while the seven-day average shows it, a few days late.
The sums behind that. With 30 runs and a true rate of 20 percent, one daily reading has a standard deviation of about 7 points. If you compare any two days when nothing has changed, the gap is 5 points or more about 63 percent of the time, and 10 points or more about 42 percent of the time. A seven-day average, which pools 210 runs, has a standard deviation of about 3 points.
The simulation treats every run as independent. That is a best case. Real runs on the same question are related, so the true noise can be different. Treat the figure as a picture of the mechanism, not a forecast of your numbers.
Runs matter more than days
Here is the idea most cadence debates miss. What reduces noise is the number of runs behind a number. How often you collect them only decides when you hear about it. Thirty questions run daily for a week produce 210 runs. Thirty questions run seven times in one afternoon also produce 210. Under the independence assumption the precision is the same. Daily collection simply spreads the runs across the week and gives you early warning along the way.
For a true cited rate of 20 percent, the 95 percent range around what you measure looks like this (a Wilson interval, the same method we use elsewhere in this series):
| Runs behind the number | Example result | 95% range |
|---|---|---|
| 30 (one daily run, 30 questions) | 6 cited | 9.5% to 37.3% |
| 90 (three repeats of the list) | 18 cited | 13.0% to 29.4% |
| 210 (daily for a week) | 42 cited | 15.2% to 25.9% |
| 900 (daily for 30 days) | 180 cited | 17.5% to 22.7% |
A single daily reading of 20 percent is compatible with a true rate anywhere from about 10 to 37 percent. That is why Monday's 23 and Wednesday's 30 can both come from a rate that never moved.
There are two ways to buy more runs. You can repeat each question, or you can add questions. Repeats help most when a question's answer varies from run to run. If a question returns the same answer every time, a repeat adds almost nothing, because the runs are not independent. Adding questions adds new information, but only if they are questions your buyers actually ask. In practice a modest list with a few repeats usually beats a long list asked once, but check this on your own data before relying on it. The buyer prompt library shows how to build the list.
What daily runs are good for
None of this means daily tracking is wasteful. It does three jobs well.
It is a tripwire for large changes. A move from 20 to 50 percent on a 30-question list needs only about 38 runs per period to detect. That is about two days of daily runs. If a priority page is deindexed, a product name changes or a major source drops you, a daily run notices within a day or two. Small drifts hide in the noise. Big breaks do not.
It tells you when your tracking is broken. A run that returns no data, an engine that stops responding or a list that silently shrank shows up fastest in a daily check.
It builds the history you pool later. Daily runs are not the reading. They are the raw material for the seven-day or thirty-day reading. If you collect them and then look only at yesterday's number, you throw away the part that carries the signal.
How many runs it takes to confirm a change
Say you made a change and want to know whether your cited rate really moved. You need enough runs before and enough after. The standard two-proportion calculation (95 percent confidence, 80 percent power, starting at 20 percent) gives this:
Figure 2. Runs needed in each period, before and after, for a range of changes. Starting rate 20 percent; runs treated as independent.
Reading across: a jump from 20 to 40 percent needs about 81 runs per period, so one weekly run with three repeats would do it. A jump from 20 to 30 percent needs 293 runs per period. At 30 questions a day, that is about ten days of daily runs before the change and ten after. A rise from 20 to 25 percent needs 1,094 runs per period, which is more than a month of daily runs on a 30-question list, both before and after.
The practical rule: decide how big a change you care about, and let that set the window. If a five-point move matters to you, no daily dashboard will confirm it in a week. Either lengthen the window, widen the list, add repeats, or accept that you will read it as a direction and not a result. The north-star metric piece works through how many opportunities you need before a number is readable on the sales side. The same logic applies here.
Plan your own cadence
The planner below turns your list size, repeats and starting rate into the range around a reading and the runs you need to confirm a change. Change the blue cells and the sheet recalculates. Start with your list size, the number of runs per question you can afford, and the smallest change you would act on. The sheet gives you the window that f ollows.
A cadence for each job
One schedule rarely fits every job. Here is a split that matches the cadence to what each job needs.
| Job | Cadence | What to run | How to read it |
|---|---|---|---|
| Tripwire | Daily | Your 5 to 10 highest-value questions, once each | Act only on a break that holds for two or more days |
| Trend | Weekly | The full list, with at least 3 repeats | Report the rate with its range and the run count |
| Change check | After each page change or release | The same list as before and after, with repeats, over equal windows | Compare ranges, and wait for the window the planner gives |
| Report | Monthly or quarterly | Pooled weekly runs | Quote the pooled rate, run count and range; read direction, not decimals |
Two notes. First, keep the list fixed and versioned during a period. If you swap questions mid-month, the change in your number may come from the list, not from the answers. The guide to tracking AI Overview rankings uses the same rule for a list of 20 to 40 questions. Second, a cadence costs runs, and runs cost money or time with most tools. Check what your plan includes per question before choosing a schedule, because the plan may set your cadence for you.
Four reading rules
1. Always show the denominator: "Cited 23 percent" means little. "Cited in 7 of 30 runs" lets a reader judge it.
2. Compare windows, not days: Seven days against the previous seven, or thirty against thirty. Never one day against another.
3. Wait for two periods before calling a trend: One high week can be noise. Two in a row, with ranges that do not overlap the old level, is worth a note.
4. Keep the question list in the report: A number without its list is a number nobody can reproduce. Apply it to the monthly report example in the AEO business case piece: when two readings have overlapping ranges, call it a direction and hold the words "on track" until the gap is clearer.
FAQ
Is hourly tracking worth it?
For measurement, rarely. Hourly runs add runs, but runs close together in time are likely to be more alike than runs spread over days, so they probably add less than the count suggests. They can serve as a tripwire during a launch or an outage. For trend work, add repeats and questions first.
Is weekly tracking enough?
It is enough for a trend, if each weekly run includes repeats and you report the range. It is not enough to catch a break within a day or two. Add a small daily tripwire list if a fast alert matters.
Should I average the week to smooth the noise?
Yes, in the sense of pooling the runs: add up the cited runs and divide by the total runs. That is better than averaging seven percentages, and it lets you calculate a range.
How many repeats per question do I need?
There is no fixed number. Start with three on questions whose answers vary and one on those that do not, and see whether the pooled range is narrow enough for the changes you care about. The planner shows the trade.
Do I need to track every engine daily?
No. Give each engine its own reading, and run the daily tripwire only on the engines where a break would cost you most. Pooling different engines into one number hides which one moved.
Give each job its own schedule
Start by naming the smallest change you would act on, then let that decide your window. Write it on one card: the list, the repeats, the window and the range you will accept. A daily run belongs on that card as the tripwire, not as the reading.
For a first look at how AI answers mention you today, the AI Search Visibility Checker gives you a single reading. Treat it as one run: a starting point, not a trend. When you have your list, AI Search Intelligence tracks each of your prompts individually, so you can see which questions move and which hold steady. Whichever tool you use, ask how many runs sit behind each number, and write that count next to it.