Can your team show where an AI answer came from? The 10-answer trace test
Imagine a board member stops the meeting on slide six and asks where one number came from. It came out of your company’s AI assistant last week, and nobody in the room can open the document behind it.
Your AI is already answering for your company, and nobody has checked what it says. Nothing has gone wrong yet; the answer is there, you just cannot see where it came from. That gap is how a A$440,000 report for an Australian government department went out in July 2025 with more than a dozen bogus references and footnotes and a quote from a court judgment that did not exist, until a university academic checked the sources and the firm agreed to repay the final instalment of its fee [1].
Short answer: take 10 answers your team got from an AI tool in the last month, split each into the claims it makes, and try to open the document and section behind every claim. Mark each claim Traced, Traced but wrong version, Not found, or Worked out. Any answer with a claim you cannot trace should not have left the building. Fix the sources and the citation rule first, then measure again.
Every Thursday we send The Revenue Signal, our weekly newsletter: one study or how-to, with the numbers and the sources. Subscribe here
..
Why is a confident AI answer not the same as a traced one?
A generative AI tool writes a fluent sentence whether or not a document supports it. The US National Institute of Standards and Technology names this risk “confabulation”: “the production of confidently stated but erroneous or false content” [2]. The tone of the answer tells you nothing about its source.
The best-known example is a courtroom. In June 2023, a federal judge in New York ordered two lawyers and their firm to pay a $5,000 penalty after a brief cited six court opinions that did not exist; the cases had come from ChatGPT [3]. One of the lawyers told the court, “I just never thought it could be made up” [3]. Nobody in that story set out to deceive anyone. The trace was simply skipped, because the answer looked finished.
That is the pattern to watch in your own company. The risk is rarely the answer nobody uses. It is the one that gets pasted into a board pack, a client email or a renewal, because it read well and arrived fast.
..
Why do I check the source before the number?
I did not start in AI. I spent more than twenty years in finance, and I ran the financial department of a government office, where every figure had to be traceable and every decision had to survive scrutiny. A number without its source document went back to whoever sent it. That was not distrust; it was the job.
AI tools have made that habit urgent again, because they produce numbers faster than anyone can check them. A claim you cannot trace is a claim you cannot trust. The test below is the finance habit, applied to AI answers, and it needs nothing but a spreadsheet and an hour.
..
How do you run the 10-answer trace test?
The test takes about an hour for 10 answers (our estimate; longer if your documents are spread across many folders). You need the answers, the documents they were based on, and one person who knows where those documents live.
..
Step 1. Collect 10 real answers (10 minutes)
Ask your team for AI answers they actually used in the last 30 days: pasted into a report, an email, a client reply or a decision. Not the ones they tried for fun. Aim for a mix: about four that contain a number, three about a contract term or policy, and three summaries of a longer document. For each, copy the question, the full answer, the date, the tool and who used it.
Tip: ask for the answers people forwarded. A forwarded answer has already been trusted, so it is the one that matters.
..
Step 2. Set up the trace sheet (10 minutes)
Open a new spreadsheet with eight columns: A Answer ID, B Claim, C Claim type (number, term, date, name or summary), D Source document, E Section or page as printed in the document, F Version or date of that document, G Result, H Notes.
Make column G a dropdown so everyone uses the same four results: Traced; Traced, wrong version; Not found; Worked out. In Google Sheets, select the column, then Data > Data validation > Add rule, choose Dropdown under Criteria, type the four values and click Done [4]. In Excel, type the four values somewhere on a second sheet, select the column, then on the Data tab click Data Validation, on the Settings tab set Allow to List, click in the Source box, select your four values and click OK [5].
..
Step 3. Split each answer into claims (15 minutes)
Give every checkable statement its own row: each number, date, name, contract term and “the policy says”. Skip opinions and advice; you cannot trace “this looks risky”, only the facts it rests on. A single paragraph of AI text often holds several claims, so expect more rows than answers.
What good looks like: someone who has never seen the answer can read column B alone and know exactly what to look for.
..
Step 4. Trace every claim (about 3 minutes a claim)
Open the document the answer cites. If it cites none, search your shared drive for the exact figure or phrase. Fill in the document, the section or page exactly as the document prints it, and the version date. Then choose the result:
Traced: the figure or the words are in the current version of the document.
Traced, wrong version: they are there, but in an older or replaced document.
Not found: you cannot find them anywhere. The AI filled a gap.
Worked out: the AI calculated the figure from documents. Write the calculation in column H and check it yourself.
Tip: a citation is a place to start, not proof. The claim still has to be in the section the answer points to.
..
Step 5. Count the result and read it (5 minutes)
Count each result with one formula per cell, for example =COUNTIF(G:G,”Not found”). Then judge each answer as a whole: an answer passes only when every one of its claims is Traced, or Worked out with a calculation you checked. One untraceable claim fails the whole answer, because that is how it would fail in front of a client.
Table 1. What each trace result tells you, and the first fix. A starting guide from our own method, not a standard.
..
Every claim names a document and a section that someone can open in under a minute.
Calculations are labelled as calculations, with the inputs shown.
“The documents do not say” is an allowed answer, and people are thanked for it.
There is one current version of each price list, policy and contract template, and an owner who retires the old one.
A written list of about 20 questions with checked answers, an answer key, is run again every month, so you see whether accuracy holds when documents change.
..
Today: ask three people for the AI answers they forwarded in the last month. Collect 10.
This week: run the trace test (about an hour). Retire every superseded document it turns up.
This month: write 20 questions with answers checked by the person who owns each document. Run them once a month.
Do not spend on: a bigger model, more documents in the tool, or a wider rollout before the 10 answers trace. More documents with no version owner add more ways to be wrong.
Short on time? Trace the three answers that contain a number. They take 20 minutes and show the pattern.
..
How Revenue Experts helps
If you would rather not run this by hand every month, this is what we built Context Studio for. Context Studio is a joint project of Revenue Experts and Ollon. It does two jobs with the same documents: it turns them into records your team reviews and approves, and it makes them searchable so people get answers they can check.
Its Document Chat cites the document and section behind each statement, lets you open the cited PDF from the answer, and shows every tool the agent used and every query it ran, so a reader can see how the answer was put together [6]. Accuracy is measured the way this test measures it: each run is compared field by field with answers a person has reviewed, and a run that scores clearly below the best earlier one is flagged before anyone relies on the change [6]. When documents have newer revisions, search uses the latest one by default, so an old price list does not compete with its replacement [6].
We say the limits first. Context Studio does not score or block an unsupported answer automatically today; the agents are instructed to cite their sources, to keep what the documents say apart from their own inferences, and to say when they do not know. It reads PDFs, including scans, today. And nothing replaces a person who owns the documents [6].
If you want to see it on your own documents, start with a demo with John Bush, our co-founder. After that, a scoped pilot on one document type, with an accuracy score measured on your own documents. Book 20 minutes with John
Of the last 10 AI answers your team forwarded, how many could you trace to a document in under a minute?
If this was useful, the next one arrives on Thursday. Subscribe to The Revenue Signal..
FAQ
If an AI answer shows citations, is it right? Not necessarily. A citation tells you where to check; the claim still has to be in that section of the current version. Trace a sample even when every answer carries citations.
How many answers do we need to check? Ten real, forwarded answers are enough to see the pattern. For a monthly check, a fixed list of about 20 questions with checked answers lets you compare one month with the next.
What if the answer came from a public tool with no company documents behind it? Trace it to the source the tool shows. If it shows none, mark every claim Not found until someone finds the document.
Can Context Studio guarantee that no answer is wrong? No, and we do not promise it. It shows the source of each statement so a person can check it, and it measures accuracy against answers your own experts have checked.
..