All Things AI · Season 3 · What You Can Build · Episode 3
A capable model runs offline on a normal laptop. Nothing is uploaded, nothing is billed monthly, and no service can go down in the middle of your work. For anything involving a client document, this is the version of AI that answers the question nobody asked.
The question most agents never thought to ask
You have a document with somebody’s name, their address, their financial situation, and their circumstances in it. You paste it into a chat window because you want a summary.
Almost nobody stops at that moment to ask where it went. It went to a company, over the internet, onto their machines, under terms you agreed to without reading. Most of the large services are reasonable about this and most have settings that help. That is not the point.
The point is that there is an option where the question does not arise, and it is available on the laptop already in your bag.
What running a model locally actually means
You download a file. The file is the model. It sits on your disk like any other large file, and a small program loads it into memory and lets you talk to it.
That is the whole architecture. There is no account, no request leaving your machine, and no way for anyone to see what you asked. Turn off the internet entirely and it keeps working, which is the simplest demonstration of the difference and worth doing once just to watch it happen.
It is genuinely capable now
This is the part that changed recently. Local models used to be a curiosity that produced obviously worse output. The open models available now handle summarising, drafting, extracting, rewriting, and following detailed instructions at a level that is fine for daily work. There is still a gap to the largest paid services on the hardest reasoning tasks. For the work an agent actually does, the gap has stopped mattering.
It is permanently free
No subscription, no per-use charge, no tier that gets more expensive as you use it more. Once it is on your disk it is yours, and it will keep working in three years whether or not the company that released it still exists.
It cannot be taken away or changed underneath you
Anyone who has built a workflow on a paid service has had the experience of it behaving differently one morning because something was updated. A local model is a fixed file. It behaves the same on the four hundredth day as the first, which matters enormously once you have built anything on top of it.
What you need to run one
One hardware question and one software step, and neither is as demanding as people assume.
- Memory is the constraint, not processor speed. On a modern laptop, sixteen gigabytes runs a small model comfortably and thirty-two runs a genuinely good one. Apple machines have an advantage here because the memory is shared with the graphics side, which is exactly what a model needs.
- Disk space for the model file itself. A few gigabytes each. You will end up with two or three and it is still less than a phone backup.
- One free application to load and run them. There are several, they install like any other app, and the good ones present a list of models to download with a button next to each.
- No account, no card, and no internet after the download.
Which model to put on it
Two are worth knowing by name, and both have full episodes in Season 2.
Hermes is the one to start with if your work is instruction-shaped. Read this and produce that, follow these constraints, keep this format. It is unusually good at doing what it is told rather than improvising, which is precisely what you want from something processing documents.
GLM is the one to add second. It is stronger on long-form drafting, and having two models with different training means you can run the same request through both and compare. That costs nothing but a minute, and the second opinion is often better than either model on its own.
What it is genuinely good for
Not everything, and being clear about the boundary is what keeps this useful rather than disappointing.
- Anything with a client document in it. Summaries, extraction, comparison, first-pass reading of something long. This is the category where local is not just adequate but correct.
- Repetitive drafting where you already know the shape of the output and just need it produced many times.
- Anything you want to run in volume, because there is no meter running.
- Work you do somewhere without connectivity, which is more of an agent’s week than most software assumes.
Where this goes wrong
Three things, and the first one is the one that makes people give up in the first hour.
Downloading a model too large for the machine. It will load, it will be unbearably slow, and the conclusion you will draw is that local models are unusable. They are not. That specific one is too big. Start with a smaller one, confirm it is fast and pleasant, and move up only if you need to.
Expecting it to know current information. A local model knows what it was trained on and has no connection to anything. It cannot look something up and, worse, it will not always tell you that. Anything current comes from your files or from a service that is genuinely online.
Assuming private means unchecked. Running locally solves where the data goes. It does nothing about whether the output is correct, and a confident wrong answer generated privately is exactly as wrong as a public one.
How it fits with the paid services
This is not a choice between local and paid, and treating it as one is why people end up with the wrong tool on the wrong job.
The sensible arrangement is a split by material rather than by task. Anything with a client, an address, or a document in it goes to the local model, because that is the category where the question of where it went should never come up. Anything that needs current information, or the hardest kind of reasoning, or a capability only a large service has, goes to the paid one, with the sensitive parts stripped out first.
In practice most agents find the split lands around eighty to twenty once they stop thinking about it, and the twenty is smaller than they expected. The work that genuinely requires the largest available model is rarer than the marketing around those models suggests.
The other advantage of running both is that you stop being dependent. A service changing its pricing, its behaviour, or its terms is an inconvenience rather than a crisis, because the work that actually matters is already running on something nobody can alter.
What it grows
The range of work you are willing to put through any of this at all, which turns out to be most of the work that matters.
There is a quiet ceiling on AI use in this industry that has nothing to do with capability. Agents avoid using it on the genuinely useful material because the genuinely useful material is the sensitive material. The result is that AI gets used for social captions and avoided for the twelve-document deal file, which is exactly backwards from where the hours actually are.
Removing that hesitation changes what the tool is for. It stops being a marketing accessory and becomes something you use on the actual work.
What it costs
Nothing, permanently, after the download. No subscription and no per-use charge. The only cost is an hour of setup and, if your laptop is old, the eventual decision about memory when you next replace it.
Questions agents actually type
Can I run AI on my own computer without the internet?
Yes. The model is a file on your disk and a small application runs it locally. You can disconnect entirely and it keeps working, because nothing about the request leaves your machine.
Is a local AI model good enough for real work?
For summarising, drafting, extracting, and following detailed instructions, yes. There is still a gap to the largest paid services on the hardest reasoning problems, but for the document and drafting work an agent actually does, that gap has stopped being the deciding factor.
What computer do I need to run AI locally?
Memory is the constraint rather than processor speed. Sixteen gigabytes runs a small model comfortably and thirty-two runs a very good one. Apple machines have an advantage because memory is shared with the graphics side, which is what a model needs most.
Is it safer to run AI locally with client documents?
It removes the question of where the file went, because it did not go anywhere. It does not make the output correct, so everything still gets reviewed. Private and accurate are two different problems and only one of them is solved by running locally.
Season 1 is Claude Code for Real Estate Agents, filmed and on its own hub. All three seasons: AI for real estate agents.