---
title: "How to Check What ChatGPT Says About Your Product"
canonical: https://supportpages.io/guides/check-what-chatgpt-says-about-your-product
updated: 2026-08-27
---

A repeatable audit that shows what assistants currently tell people about your product, and a method for tracing each wrong answer back to whatever caused it. The design constraint shapes everything: these systems are non-deterministic, personalised by account history, and updated without notice. A single answer is an anecdote, so the audit is built to produce something better than anecdotes.

## Before you start

Use a logged-out or temporary session. Account memory, custom instructions and prior conversations change answers substantially, and an audit run inside your own working account measures your account rather than your product.

Set aside somewhere to store results verbatim - a spreadsheet is fine. The value of this exercise is entirely in comparing runs over time, which requires that you kept the exact wording.

![An audit grid of ten tasks against three assistants, with each answer marked right, vague or invented.](/images/guides/figures/check-what-chatgpt-says-about-your-product.webp)

*An hour of work, and a more honest picture of reach than any analytics dashboard.*

## Steps

1. **Build a fixed list of ten to twenty questions your support team actually receives.**
   Take them from your ticket queue and search logs rather than inventing them. Include the awkward ones: pricing edge cases, data deletion, security posture, competitor comparisons, cancellation.

2. **Run each question in a clean session, twice.**
   Two runs of the same prompt frequently differ. If you only ever see one answer, you cannot tell a systematic error from a one-off, and those need different responses.

3. **Test across at least three assistants, including Google's AI Overviews and AI Mode.**
   Retrieval and training data differ per provider, so results diverge widely. An audit of one assistant tells you about one product's behaviour on one day. Google's is the one with first-party measurement behind it: Search Console's Generative AI performance report shows which of your pages appear in its AI answers, which no other provider offers.

4. **Record the answer verbatim, with metadata.**
   Date, assistant, model name if shown, whether web access appeared to be used, and which sources were cited. Paraphrasing the answer into your notes destroys the comparison you are building.

5. **Classify every error by cause, not severity.**
   Three causes account for nearly all of them: the answer reflects an outdated fact that was true once, the answer was assembled from a third-party page, or the answer has no traceable source at all. Each needs a different fix.

6. **Follow the citations before concluding anything.**
   When sources are shown, open them. A wrong answer that cites your own documentation is your documentation being wrong, which is the most common and most fixable case.

7. **Fix the source rather than the answer.**
   Correct your own stale page, or get the outdated third-party page updated, or publish a clearer page that answers the question directly. There is no mechanism for editing what a model says. Two documented levers exist around it, both Google's: the Search Console report tells you whether your pages are being cited at all, and a reader who adds your site as a preferred source sees it badged in AI Overviews and AI Mode - Google publishes an embeddable button for that.

8. **Verify your pages are reachable.**
   If your documentation blocks the relevant crawlers or renders its body text in JavaScript, the assistant is answering from whatever else it could reach. Check that before concluding your writing is at fault.

9. **Re-run the identical question list quarterly and after major releases.**
   Changing the questions between runs is the fastest way to make the results uncomparable. Add new questions in a separate section, keep the original list frozen.

## Common mistakes

**Drawing a conclusion from one prompt in one session.** Variance between identical runs is routine. One bad answer screenshotted into a team channel usually starts a project that a second run would have made unnecessary.

**Auditing from a personal account.** Memory, custom instructions and history are all in play. The answer you see may be unavailable to anybody else.

**Assuming a wrong answer means the model invented it.** Far more often it found something real of yours - a two-year-old changelog entry, a deprecated API page, a comparison table from a launch - and reported it accurately.

**Trying to correct the model directly.** Thumbs-down feedback, telling it that it is wrong, and asking it to remember a correction do not propagate to other users' sessions.

**Reporting the result as a percentage accuracy score.** With a sample of twenty non-deterministic questions, a headline accuracy figure is a number with no method behind it, and it will be quoted long after the run that produced it.

## When this stops scaling

The manual version holds at roughly twenty questions, three assistants, quarterly - about an afternoon each time. What breaks past that is not your effort budget but the measurement itself: adding more questions increases the noise you are averaging over rather than the confidence you gain, because variance between runs is large and the systems change underneath you between audits.

This is the point where a guide of this kind would normally recommend a tool, and there is not an honest one to recommend, with a single exception: Search Console's Generative AI performance report is first-party data about one surface, and it is the only such data any provider gives you. Monitoring products in this space sample the same non-deterministic outputs you would sample by hand, and no product controls what an assistant says about you, because none of them can write to the model. The only lever that reliably matters is upstream: whether an accurate, current, fetchable page exists for each of those twenty questions. Work on that, and treat this audit as a way of finding out which pages to work on - not as a metric to move.