All Things AI · Season 3 · What You Can Build · Episode 8
A long document stops being an evening. You get what matters, with the page it came from, in the time it takes to make coffee. The skill is not asking for a summary, it is asking the right question and demanding a citation for every answer.
The document you meant to read
There is a stack of them. The long report, the association documents, the disclosure package, the sixty-page agreement, the market analysis somebody sent that might contain something useful.
You skimmed it. You looked at the parts you knew to look at. You did not read the middle forty pages, and neither did anyone else in the transaction, and everyone proceeded on the assumption that nothing important was hiding in there.
Usually nothing is. The problem is that usually is doing a lot of work in that sentence, and the one time it matters is the time it really matters.
Why a summary is the wrong thing to ask for
The instinct is to paste the document in and ask for a summary, and the result is a paragraph that tells you the document is about what you already knew it was about.
This happens because a summary is an averaging operation. It produces the middle of the document, and the middle of the document is the boring part. What you actually need is the exceptions: the thing that is unusual, the thing that contradicts something else, the date that is different from the date you expected.
Those are the opposite of a summary. They are the outliers, and you have to ask for outliers specifically or you will never see them.
The questions that actually work
Every one of these is a question about exceptions rather than about content, and they are worth keeping somewhere you can copy them.
- What in this document is unusual compared to a standard one of these, and why.
- What does this document require of someone that they might not expect.
- List every date, deadline, and time period mentioned anywhere in here, with the page.
- Where does this document contradict itself, or contradict this other document.
- What is mentioned once and never explained.
- What would a careful reader want to ask about after reading this.
The rule that makes it trustworthy
Every answer comes with a page number, or you do not use the answer.
This is the entire difference between a tool you can rely on and one that will eventually embarrass you. A model asked for facts will produce facts, and when it cannot find one it will produce something plausible in the same confident register as everything else. There is no tonal difference between a real finding and an invented one.
Requiring a citation collapses that problem, because a claim with a page number is checkable in ten seconds and a claim without one is visibly a claim without one. Put the requirement in the request itself: cite the page for every statement, and say you could not find it rather than answering when it is not there.
Then spot-check. Not all of them. Two or three, on the findings that would actually change a decision. That habit takes a minute and it is what converts this from an interesting trick into something you can lean on.
Reading many documents at once
The single-document version saves you an evening. The multi-document version does something you could not previously do at all.
Twelve documents in a folder, asked where do these disagree with each other, is a question no human reasonably answers, because it requires holding twelve things in mind simultaneously and checking every combination. It is exactly the shape of problem a model is good at and a person is bad at.
That is where the real value sits. Not faster reading, but a category of question that was previously not worth asking because answering it took longer than the answer was worth.
Keeping it private
This is the capability most likely to involve a document with somebody’s name and circumstances in it, which makes it the one where where it runs matters most.
A local model handles this work well. Reading, extracting, and comparing are exactly the instruction-following tasks that open models running on your own machine are good at, and the file never leaves your laptop. For anything client-attached, that should be the default rather than the exception.
Save the paid services for documents that are already public, or for the rare case where the reasoning is genuinely harder than an open model can manage.
Where this goes wrong
Four failures, and the first two are responsible for almost all disappointment with this.
Asking for a summary. Covered above and worth repeating because it is what everyone does first, gets an underwhelming paragraph from, and quietly concludes the technology is overrated.
Accepting answers without citations. A model that cannot find something will often answer anyway, in the same voice it uses for things it did find. Without a page number you have no way to tell those apart, and you will not notice until it matters.
Trusting it on the thing you most needed. The findings that would change a decision are exactly the findings to verify by hand. This is not a limitation of the tool so much as basic practice with any source.
Forgetting it read a version. Documents get amended. A model reading last month’s copy will answer confidently about terms that have since changed, and it has no way to know a newer one exists.
What it grows
Your evenings, obviously. Less obviously, the quality of what you catch.
A document you actually process beats one you meant to. The comparison is not between a model reading it and you reading it carefully, because you were not going to read it carefully, and pretending otherwise is how the industry has operated for decades.
The comparison is between a model reading it and nobody reading it. On that comparison this is not a marginal improvement, it is the difference between having looked and not having looked.
How to build it into the week
The tempting version is to do this only when something feels important, which means it happens rarely and unevenly.
The better version is a standing habit: anything over about fifteen pages that arrives goes through the same set of questions before you file it. It takes a few minutes, it happens whether or not the document seemed important, and it means the unusual thing gets found in the documents nobody was worried about, which is where unusual things tend to be.
Keep the answers with the document. A short file of findings alongside the original is worth having a year later when somebody asks a question about it and you no longer remember.
What it costs
Free on a local model, which is where most of this work should run anyway for privacy reasons. On a paid service it is a small per-use cost, and long documents are the one category where those costs are noticeable enough to be worth thinking about. Another reason to keep this work local.
Questions agents actually type
Can AI read a long document and tell me what matters?
Yes, but not if you ask for a summary. A summary produces the average of the document, which is the part you already expected. Ask about exceptions instead: what is unusual, what contradicts something else, and what is mentioned once and never explained.
How do I stop AI making things up about a document?
Require a page number for every statement and instruct it to say it could not find something rather than answering. A claim with a citation is checkable in seconds. Then spot-check the two or three findings that would actually change a decision.
Can AI compare several documents at once?
Yes, and this is where it does something a person genuinely cannot. Asking where a set of twelve documents disagree with each other requires holding all of them in mind at once, which is the shape of problem models handle well and people handle badly.
Is it safe to use AI on client documents?
Run it on a local model and the file never leaves your computer, which removes the question entirely. Reading and extracting are exactly the tasks open models running on your own machine handle well, so there is little reason to send that material anywhere.
Season 1 is Claude Code for Real Estate Agents, filmed and on its own hub. All three seasons: AI for real estate agents.