How to Summarize PDFs Locally with Open-Source LLMs (No API, No Data Leaving Your Machine)

Most "summarize a PDF with AI" tutorials send your document to a cloud API. That's fine — until the document is a contract, a patient record, or anything your compliance team would rather not ship to a third party. The alternative in 2026 is genuinely good: run an open-source model locally, so the bytes never leave your machine and you pay $0 per token. Here's how to build a local PDF summarizer with Ollama and Llama 3, plus an honest look at where local wins and where it doesn't. Why local (and why not) Local wins when: Privacy/compliance — the file can't leave your infrastructure. Volume — you summarize thousands of docs and don't want a per-token bill. Offline / air-gapped environments. Local costs you: Hardware — you want a decent GPU (or Apple Silicon) for reasonable speed; CPU-only works but is slow. Quality ceiling — a 7B–8B local model is weaker at long, nuanced documents than a frontier cloud model. Ops — you own the setup, the model updates, and the tuning. If none of those first three apply to you, a cloud API or a no-code tool is probably less hassle (more on that at the end). Step 1: Install Ollama and pull a model Ollama makes running local models a one-liner. After installing it: ollama pull llama3.1:8b # smaller/faster: ollama pull llama3.2:3b # stronger, needs more VRAM: ollama pull llama3.1:70b Enter fullscreen mode Exit fullscreen mode Model choice is the main quality/speed dial. 3B is fast and fine for short docs; 8B is the sweet spot for most machines; 70B approaches cloud quality if you have the VRAM. Step 2: Extract the text Same as any pipeline — native PDFs give text directly; scanned PDFs need OCR first. from pypdf import PdfReader def extract_text(path: str) -> str: reader = PdfReader(path) return "\n".join((page.extract_text() or "") for page in reader.pages) Enter fullscreen mode Exit fullscreen mode If this comes back nearly empty, the PDF is scanned — run it through Tesseract (pytesseract) before continuing. Step 3: Chunk, then summarize with the local model Local models have context limits too, so long documents still need the map-reduce pattern: summarize each chunk, then summarize the summaries. The only difference from a cloud pipeline is the client — we call Ollama instead of a remote API. # pip install ollama pypdf import ollama MODEL = "llama3.1:8b" def chunk(text: str, size: int = 8000, overlap: int = 400): step = size - overlap return [text[i:i + size] for i in range(0, len(text), step)] def summarize(text: str) -> str: resp = ollama.chat( model=MODEL, messages=[ {"role": "system", "content": "Summarize the text into concise bullet points. Keep names, numbers, and conclusions. Do not invent anything."}, {"role": "user", "content": text}, ], options={"temperature": 0.2}, # lower = more faithful, less creative ) return resp["message"]["content"] def summarize_pdf(path: str) -> str: text = extract_text(path) if len(text.strip()) < 50: raise ValueError("Almost no text extracted — this PDF is probably scanned. OCR it first.") partials = [summarize(c) for c in chunk(text)] combined = "\n\n".join(partials) return summarize("Combine these section summaries into one structured summary:\n\n" + combined) if __name__ == "__main__": print(summarize_pdf("report.pdf")) Enter fullscreen mode Exit fullscreen mode Note the smaller chunk size (8k vs the 12k I'd use on a cloud model): local 8B models hold quality better on tighter chunks, and it keeps each call fast. Step 4: Keep it faithful Local models hallucinate more than frontier ones, so lean on the prompt and settings: Low temperature (0.2 or below) for summaries — you want fidelity, not flair. Explicit instruction not to invent facts, and to preserve numbers/names. Spot-check a few outputs against the source before trusting the pipeline on a batch. Performance reality check On an 8B model: Apple Silicon (M-series) / a modern GPU: a 30–50 page report summarizes in seconds to a couple of minutes. CPU-only: it works, but expect minutes per document — fine for a nightly batch, painful interactively. The cost, though, is the headline: after the download, summarizing 10,000 PDFs costs the same as summarizing one — electricity. That's the whole reason to go local at volume. When local isn't the right call Being honest about the trade-off, since this is where a lot of "run it locally!" posts stop: You just need a few summaries occasionally. Standing up Ollama, a model, and an extraction pipeline to summarize five PDFs is overkill. If privacy isn't the constraint, a free web tool does it in seconds — ChatPDF and NotebookLM if you don't mind an account, or PDFSummarizer.net if you want no sign-up and formats like EPUB/PPTX handled for you. One caveat that matters specifically because this article is about privacy: those are hosted tools, so your file goes to their servers — they're the convenience option, not the privacy option. If keeping data local is the whole point, stay local. You need top-tier reasoning on long, subtle documents. A frontier cloud model still edges out an 8B local one. You don't have the hardware. CPU-only 8B is slow. Below a certain machine, cloud is simply faster and cheaper in wall-clock terms. Takeaways Local summarization is real in 2026: Ollama + Llama 3 gives you offline, zero-per-token summaries. The pipeline is the same extract → chunk → map-reduce; only the model client changes. Trade-offs: privacy and cost for hardware and a quality ceiling. Keep temperature low and spot-check for hallucinations. If privacy and volume aren't your drivers, a cloud API or a free no-code tool is less work. Running models locally for document work? I'd like to hear which model/size you settled on and what hardware you're on — drop it in the comments. Tool details were accurate at the time of writing — check current limits before you rely on them.