How to Track Brand Sentiment in Answer Engines

Two analysts review the same 50 AI answers about a project planning vendor. One reports 82% positive sentiment. The other reports 61%. Both are correct under their own rules. One counts lukewarm responses as positive; the other classifies them as neutral.

But the problem is the lack of a shared definition of positive sentiment. Without consistent rules, sentiment scores can say more about how answers are evaluated than how AI portrays a brand. To track brand sentiment reliably, you need clear labeling criteria, a way to validate the model applying them, and a method for interpreting the results without overstating their meaning. You also need to establish which answers mention your brand by separating citations from mentions and defining the prompts used to track AI visibility.

Sentiment belongs to a claim, not an answer

An answer about a brand rarely has one tone. It might say the pricing is quote based, the interface is easy to learn and the support is slow. Those are three claims, and a buyer reads each differently. Scoring the whole answer with one label throws that detail away and makes the score depend on which claim the scorer noticed first. Split each answer into claims and label each one.

A scoping review of 38 studies on how language models treat brands shows why definitions matter. The studies used "visibility" to mean inclusion, rank, share, sentiment or citation, and only 17 defined the construct clearly enough to separate whether a brand is included, how prominent it is and how favorably it is described.

In one study, 88 percent of the mentions of a revived brand carried a disclaimer about its changed status, which is a reminder that a mention is not an endorsement. The review is a preprint and has not been peer reviewed. It searched open sources only, used a language model for screening and charting with a second model instance as the only check, and reports that 19 of its 38 studies had authors who sell AI-visibility services or whose employer's brand was measured (Ibrahim, Research Square preprint, October 2026). That last point is why this post relies on independent sources.

Figure 1. One answer, many claims, two labels each. Tone and accuracy are separate, and each pairing calls for a different response.

Neutral claims sit between the two rows. A claim you cannot compare with your own record, such as an opinion about ease of use, gets its own accuracy state so it is not counted as accurate or wrong.

What research says about labeling sentiment

None of the studies below is about answer engines. They test how reliably people and models label sentiment and similar judgments in text, which makes them evidence about the measuring instrument you are about to build.

An Old Dominion University preprint compared six human raters, three sentiment tools and three language models on 100 posts. Human agreement was modest: a Fleiss kappa of 0.39 across the six raters on three labels (positive, negative, neutral), rising to 0.45 when the task was negative against not negative.

The three models agreed with each other more, at 0.74 on three labels. The authors conclude that models should not be treated as ground truth and that human-verified labels remain the benchmark. Their limits are 100 posts on one keyword and region, six graduate-student raters and no confidence intervals (Jayanetti et al., arXiv 2610.10318).

A peer-reviewed study in Scientific Reports compared 33 human annotators with several language models on 100 texts. The authors report that models were more consistent than humans on sentiment, that both groups did poorly on sarcasm, and that most models stayed consistent across three points in time. The text set is small, and the paper's counts of models differ between its abstract and body (Bojić et al., Scientific Reports 2025).

A third preprint tested seven models on 3,000 posts labeled for offensive language and hate speech, using 12 task designs and three runs each. Repeating the same design gave a median Fleiss kappa of 0.91. Changing the design on the same posts dropped the median Cohen's kappa to 0.76, and the estimated prevalence varied far more than sampling alone would explain: its variance was 76.7 times larger for offensive language. The topic is not sentiment, and it covers two constructs on one platform (Reiter et al., arXiv 2609.35824).

Three working rules follow, and they are our reading, not the studies' conclusions. People disagree, so you need a written codebook and a second coder. A model can be very consistent and still be sensitive to how its instruction is written. And rerunning the same labeler on the same claims does not test that sensitivity.

Write the codebook before anyone codes

A codebook is a one-page document that both coders use. Write it once, freeze it between rounds and record any change. The fields below are our recommendation.

Hedged or sarcastic claims are the hard cases, since both groups in the Scientific Reports study struggled with sarcasm. Label them neutral and flag them for the second coder. If an answer does not name the brand, log the answer and code no claims.

Because agreement was higher for negative against not negative than for three labels in the Old Dominion study, we suggest reporting negative against not negative as the headline and keeping the three labels for detail. Never put customer or personal information in a prompt, a claim or a saved answer.

Pick prompts that can produce a claim

Sentiment needs a claim to label, and a claim needs the brand named. The prompt types below elicit different amounts of it, and the notes in the right column are our expectations, not findings.

The last row matters most. A prompt that asks for downsides will produce negative claims whether or not the brand has a problem, so never pool prompt types into one sentiment score. Report each type separately. Wording matters within a type too, and how query phrasing changes which brands get cited shows how small changes shift answers. Keep each prompt fixed once you start tracking it.

Check a model labeler against two people first

Coding by hand does not scale forever, and a model can help. It earns its place only after you check it, and the check is short.

1. Have two people code 30 to 40 claims independently with the codebook. Include extra negative claims in this sample, for the reason below.
2. Calculate kappa, which is agreement beyond chance: observed agreement minus the agreement expected by chance, divided by one minus the agreement expected by chance. Calculate it for three labels and for negative against not negative.
3. Settle each disagreement and record the adjudicated label. Fix the codebook wherever disagreements cluster.
4. Run the model labeler on the same claims with a fixed written instruction and compare its labels with the adjudicated ones.
5. Reword the instruction slightly, run it again and compare. If labels move, you have found the sensitivity the Reiter study describes.
6. Freeze the model, the instruction and a version name. Repeat the check whenever any of them changes.
   
   

Kappa also depends on how common each label is. When negative claims are rare, two coders can agree on 90 percent of claims and still show a low kappa, because most of that agreement would happen by chance. Figure 2 shows the effect for binary labels.

Figure 2. A model, not a measurement. It assumes binary labels and that both coders mark the same share of claims negative.

The worksheet's invented example shows the same thing. Two coders agree on 87.5 percent of 40 double-coded claims for negative against not negative, yet kappa is 0.59, below the 0.6 placeholder minimum, because only a handful of those claims were negative. Oversampling negative claims in the double-coded set gives kappa something to measure.

Read the results by theme, prompt type and source

The numbers below come from the worksheet's invented rows: 40 saved answers, 36 of which named the brand, and 138 coded claims. Overall, 32 percent of claims are positive, 43 percent neutral and 25 percent negative, with a 95 percent range for the negative share of 19 to 33 percent. That total hides the useful pattern, which shows up by theme.

Pricing stands out: 9 of its 12 claims are negative, and 6 of those 9 differ from the master record, which suggests an old price is circulating somewhere. In the same example, negative claims cite review sites most often (11 of 35), then community pages and other third-party pages (9 each), and 22 of the 35 differ from the record. That pattern is built into the example rows and is not a finding.

By prompt type, all four answers to "ask for problems" prompts contain a negative claim and 6 of 7 answers to comparison prompts do. Every prompt type is below the worksheet's 20-answer minimum, so none of these shares should be read as a rate. The ranges assume independent draws, which is a best case, and claims from one answer are not independent of each other. The model variability post explains why the same prompt can return different answers on different days.

Act on the grid in Figure 1. Negative claims that differ from your record are facts to correct where they live: your own pages first, then the third-party sources the answer cited. The PR and comms post covers a fact sheet and a log of claims that differ from it. Negative claims that match your record are product, proof or positioning questions for the people who own those, and a rebuttal is the wrong response to a true claim. If you sell in several languages, code each market and language cell separately, as the regions and languages post describes.

Code the claims in a worksheet

The worksheet has a codebook, an answer log, a claim log, an agreement tab and a summary. Grey rows are invented examples with an invented brand, prompts and results, so replace them before you read any output. Do not enter customer or personal information anywhere in it.

1. On the Settings tab, set the minimum share of claims to double code, the minimum kappa, the minimum answers per prompt type and your prompt list.
2. On the Codebook tab, rewrite each definition for your brand and freeze it between rounds.
3. On the Answers tab, log each saved answer and whether it named the brand.
4. On the Claims tab, enter one row per claim with its theme, Coder 1's tone, Coder 2's tone for the double-coded claims, an adjudicated tone for disagreements, the accuracy label and the source type. Row checks catch missing labels, unknown answers and unresolved disagreements.
5. On the Agreement tab, read the observed agreement and kappa for three labels and for negative against not negative.
6. On the Summary tab, read tone, theme, prompt type, source type and the tone-by-accuracy grid, each with ranges where they apply.
   
   

Frequently asked questions

Is sentiment the same as a mention?

No. A mention means the answer named your brand. Sentiment is the tone of what the answer says about it, and it can only be coded for answers that name you. Count unnamed answers separately.

Can we trust a model to label sentiment?

Only after you check it against two people on your own claims. The research above suggests models can be consistent, but consistency across repeated runs does not show that labels hold when the instruction changes. Freeze the instruction and recheck when anything changes.

How many claims or answers do we need?

Count answers that name the brand within each prompt type. With 20 answers and an observed share of 30 percent, the 95 percent range runs from about 15 to 52 percent, and it narrows slowly after that. The worksheet's 20-answer minimum is a placeholder, so set your own.

What if a negative claim is true?

Then it is not a facts fix. It is a product, proof or positioning question, and the response belongs with the team that owns it. Do not publish rebuttals of true claims. We found no study showing that any content change shifts the tone of later answers.

Should we report one sentiment score for the brand?

We would not. A single score blends prompts that ask for problems with prompts that do not, and themes that run positive with themes that run negative. Report by theme and prompt type, with ranges, and keep negative against not negative as the headline.

Have two people code the same twenty claims this week

Pick one prompt that names your brand and save a few answers to it. Split the answers into claims, about twenty in all. Give two people the same one-page codebook and have each label every claim on their own, without comparing notes. Then count how many labels match, list the claims where they differ and rewrite the codebook rule that caused each difference. Do not put customer or personal information in the prompt or the saved answers.

If you want a first reading on one of those prompts before you build the log, the AI Search Visibility Checker gives you one. If you want to ask the same prompts again between coding rounds so the wording stays fixed, AI Search Intelligence tracks individual prompts.