Five ways a summary goes wrong
A summary reads the same whether or not the document supports it. The errors that matter are small, and each one keeps the shape of a true sentence. The examples below are made up: a report on a pilot in a few stores, and an AI summary of it.
| What goes wrong | Example | How to catch it |
|---|---|---|
| A changed figure | The report says sales rose 4.2% in the pilot stores. The summary says 4.8%. | Look for every figure of the summary in the original. |
| A swapped name | The report says Dana led the pilot. The summary names the pilot lead as Sam. | Look for every name the same way. |
| A claim the document never makes | “Customers preferred the new layout.” The report says nothing about customers. | Find the sentence the line rests on. Here there is none. |
| A careful finding stated as certain | The report calls the gap in sales “too small to be conclusive”. The summary says the pilot increased sales. | Read the original sentence beside the summary line. No figure is wrong, so a check of the details finds nothing. |
| Something important left out | Two stores reported stock shortages. The summary does not mention them. | Read what the summary left out. |
Two studies show that these errors are common enough to plan for. In a study coordinated by the European Broadcasting Union and led by the BBC, published on October 22, 2025, journalists at 22 public service media organizations judged more than 3,000 answers from ChatGPT, Copilot, Gemini and Perplexity. They found that 45% of the answers had at least one significant issue, and 20% had major accuracy issues. That study is about answers to questions on the news, not summaries of a document you supply, so read it as a warning about fluent text and not as a measure of your summary.
The second study is closer to the job. In a comparison of 4,900 AI summaries of scientific texts with the originals, published in Royal Society Open Science in April 2025, Uwe Peters and Benjamin Chin-Yee tested 10 AI models. Most of them drew broader conclusions than the original texts did, even when asked for accuracy, and the AI summaries were “nearly five times more likely to contain broad generalizations” than human-written ones. No figure has to be wrong for that to happen. The summary only has to drop the words that limited the finding.
“A 12-week trial of 84 adults linked the app to a small drop in reported stress.”
Not quite. Every detail and every limit of the original is still here.
“The app reduces stress.”
Yes. A small drop in one 12-week trial of 84 adults has become a general fact. No figure is wrong, so a check of the details finds nothing to mark: Summary Check traces this line to the original sentence and passes it. Only reading the two sentences side by side shows the difference.
“In a 12-week trial, the app was linked to slightly lower reported stress.”
Not quite. The number of adults is left out, and what remains is as careful as the original.
“Researchers ran a 12-week trial with 84 adults.”
Not quite. True, and not much use, because it leaves out the result. It will not mislead anyone, though.
A wrong figure is easy to find. A finding stated more broadly than the document states it has to be read to be seen.
The five-minute check
You do not need to reread the whole document. You need to know where to read. The four steps below go from the quickest check to the slowest, and Summary Check does the looking for the first three in your browser, with no AI and no upload.
- 0–1 minFigures and namesLook for each one in the original.
- 1–3 minThe lines that carry a decisionTrace each to the sentence it came from, and read that sentence.
- 3–4 minWhat was left outSkim the figures and names the summary dropped.
- 4–5 minA corrected summaryAsk for one made from the document alone.
Timings are a guide for a summary of a page or less. A long document, or one with a lot riding on it, deserves more.
1. Check the figures and names
Mark every number, amount, date and name in the summary, and search the original for each one. A figure that is not there is the first thing to question. Two kinds are not errors but still need a look: a rounded figure, such as “about $4.2 million” for $4,236,000, and a figure the AI worked out, such as “grew by 16” where the original says “from 212 to 228”. Check the sum on the second kind. Words inside quotation marks should appear in the original exactly.
Summary Check does this when you paste both texts. It marks each detail of the summary that is not in the original, and it tells a rounded figure and a worked-out total apart from one that is simply absent. If you were given a summary with no original at all, the Fact Check List lists every claim in it that is worth verifying elsewhere.
2. Trace the lines that carry a decision
Most lines in a summary are background. A few carry weight: the recommendation, the deadline, the cost, the risk, who is responsible. For each of those, find the sentence of the original it rests on and read that sentence in full, with the one before it and the one after. You are looking for the limits the summary dropped: “in the pilot stores”, “so far”, “unless”, “too small to be conclusive”.
A line with no sentence behind it may be a fair conclusion drawn from several parts of the document, or it may be invented. Until you find its support, it is the summary’s claim and not the document’s. Summary Check shows each summary sentence beside the nearest sentence of the original and says how far the original backs it up, from “In the original” to “No match in the original”.
3. Read what was left out
A summary has to leave things out. The question is whether it left out the point. Skim the original’s headings and its last section, and look for anything framed as a risk or an exception.
Summary Check lists the amounts, dates, names and quotes that the original gives and the summary does not, starting with the ones the original leans on, such as a figure it repeats. That list shows details, so a warning with no figure or name in it will not appear there. For notes from a meeting, check that every decision in the transcript made it into the notes. The guide to AI meeting minutes covers that case.
4. Ask for a corrected summary
If you found problems, do not patch the summary by hand and move on, because the lines you did not check came from the same process. Ask the AI for a corrected version. Name the lines you could not confirm and tell it to use the document alone.
Some lines in your summary are not supported by the document. For each line below, quote the sentence of the document it rests on. If there is none, take the line out. Use only the document, and leave the rest of the summary as it is.
1. {{line_you_could_not_confirm}}
2. {{line_you_could_not_confirm}}
Summary Check writes a prompt like this for you from the points still marked, with any left-out details you chose to add back. Then check the new version the same way: a corrected summary is a new summary. If the text in question is your own writing that an AI edited, the check is a different one: see how to check what ChatGPT changed in your writing.
Ask for a summary you can check
If you are the one asking for the summary, you can make all of this faster. A summary that points at its own evidence takes minutes to check, and one that names its reader keeps what that reader needs.
Before
2/10 Needs work
- Has: What to make
- Missing: Who it's for
- Has: Background facts
- Missing: Length
- Missing: Format
- Missing: What to avoid
- Missing: What good looks like
- Missing: An example
- Missing: If unsure
- Missing: Layout
After, with what was added marked
10/10 Excellent
- Has: What to make
- Has: Who it's for
- Has: Background facts
- Has: Length
- Has: Format
- Has: What to avoid
- Has: What good looks like
- Has: An example
- Has: If unsure
- Has: Layout
The second request names the reader and the decision. It also asks for the sentence behind each point and for a list of what was left out. The marked words are what it adds.
Here is the same request with blanks, to copy. The example line shows the format only, and the AI fills in the real figures.
"""
{{document}}
"""
Summarize the document above for {{reader}}, who will use the summary to decide {{decision}}.
Use only what the document says.
- Copy every number, date, name and amount exactly as the document gives it. Do not round figures or work out new ones.
- After each key point, quote the sentence it rests on, or give its section or page.
- Keep the document's own limits. Where it says "may" or "in this sample", so does the summary.
- If the document is unclear or contradicts itself on a point, mark the point [unclear] instead of choosing a reading.
Length: one line that answers the reader's question, then up to 6 key points of one or two sentences each.
Format: a numbered list, then a section headed "Left out" that names what the summary does not cover.
Example of a key point: "Late deliveries rose from 4% to 6% in the third quarter. Source: 'Late deliveries rose from 4% to 6% in the third quarter' (section 2)."
A good summary lets the reader check any line against the document in under a minute.
The executive summary template is a ready version for a decision-maker with two minutes: it copies numbers exactly and keeps the author’s level of certainty. For a long document, the guide to summarizing long documents with AI adds page numbers and a way to split the work, and the Token Counter shows whether the text fits a model in one go. If the document names real people or clients, Safe Paste swaps those details for placeholders before the text goes to an AI.
What no automatic check can tell you
Summary Check matches words and details. It does not understand either text, and it is not a hallucination detector. Four judgments stay with you:
- Whether a matched line means the same. A sentence can share most of its words with the original and still overstate it, as “The pilot increased sales” does in the table above.
- Whether an unmatched line is wrong. A conclusion drawn from several parts of a document can be sound and still have no single sentence behind it.
- Whether the summary kept what matters. That depends on who is reading and what they will decide.
- Whether the original is right. A summary can be faithful to a document that is mistaken, and a transcript carries every mishearing into the notes.
The check can also miss a name, because it finds names by their capital letters. A second AI can help you look, and it can miss the same things, so the lines that carry a decision still need your own eyes on the original.
Key takeaways
- A summary sounds equally sure whether or not the document supports it.
- Check the figures and names first, then trace the lines that carry a decision to the sentence behind each one.
- Read what was left out, and make sure it is not the point.
- Ask for a corrected summary from the document alone, and ask future summaries to quote their evidence.
- Matching words is not matching meaning, so read the original where it matters.
Questions
How do I check if an AI summary is accurate?
Compare it with the original in a fixed order. Look for every figure and name of the summary in the original first. Then take the lines that carry a decision and read the sentence each one rests on. Skim what was left out, and ask for a corrected version made from the document alone. For a one-page summary, plan on about five minutes.
Can ChatGPT check its own summary?
It can help, and it can repeat its own mistake. The useful request is narrow: ask it to quote the exact sentence of the document behind each key point, then search the document for those quotes yourself. A quote you cannot find marks a line to take out. A reply that only says the summary was verified is not a check.
Is there a free tool that finds hallucinations in a summary?
Summary Check is free and runs in your browser. It traces each summary sentence to the nearest sentence of the original and marks numbers, dates, names and quotes that are not there. It matches words and details and does not understand meaning, so treat it as a way to find where to read. It is not a hallucination detector.
Is it safe to paste a confidential document into a summary checker?
Summary Check compares the two texts in your browser and uploads nothing. The larger question is the AI that wrote the summary, because it did receive the document: use the tool your employer approves for confidential material. For text you paste into an AI yourself, Safe Paste swaps names and account numbers for placeholders first.
How do I check an AI summary of a meeting?
Treat the transcript as the original and check the same way: names and dates first, then every decision and action against the lines where it was said. A transcript can mishear, so confirm anything surprising with someone who was there. The guide to AI meeting minutes has a prompt that asks for decisions and actions.
What if I do not have time to read the original?
You do not have to read all of it. Read the sentences behind the lines you will act on, and skim the list of what was left out. If the document is long, the Token Counter shows whether it fits an AI model in one go. If it did not fit, the summary may cover only part of it.
Prompts to use with this
- Check a contract against your standard terms before legal reviewA vendor or customer sent their own contract, and you want to know where it departs from your usual terms so your lawyer's time goes to what matters.
- Check a contract's insurance requirements against your current coverageA customer contract requires specific insurance, and you need each requirement compared with your policies before you sign or send certificates.
- Check the totals, percentages and growth rates in a report before it goes outA report, slide deck or memo full of figures is about to be sent and you want every total, percentage and change rechecked against the source numbers.
- Write a report from your notes and figuresYou need findings, numbers and notes organized into a clear report with headings and a summary.