Scrubkit: point it at a folder, get clean text + metadata back — 100% offline

If you've ever built RAG ingestion, a search index, or any "read a pile of documents" feature in .NET, you know the annoying part isn't the embeddings or the vector store. It's step zero: getting clean text out of a messy folder of PDFs, Word docs, spreadsheets, emails, and whatever else landed there — without shipping the files off to some cloud API. Scrubkit does exactly that, and nothing leaves your machine. using Scrubkit; var scrubber = new FolderScrubber(new ReadOptions { Recursion = Recursion.AllNested }); await foreach (var doc in scrubber.ReadStreamAsync(@"C:\Docs")) { if (doc.Text.Length == 0) continue; // skip metadata-only rows await index.UpsertAsync(doc.Path, doc.Text, doc.Metadata); } Enter fullscreen mode Exit fullscreen mode One call, one flat table: per file you get Text, Metadata, TypeBucket, SizeBytes, Modified, and any Warnings. No network calls, no telemetry — usable in air-gapped and regulated environments. What it reads out of the box PDF (PdfPig), Office docx/pptx/xlsx, the plain-text family (txt/md/csv/json/ xml/html/…), and image EXIF. Unknown types come back as a metadata-only row. A single unreadable file never crashes the batch — the problem shows up as a Warning on that row. Built like a real library, not a gist Multi-targeted net8.0 + netstandard2.0. 114 tests, ~99% line coverage, an 85% gate in CI (Linux + Windows). Package validation guards the public API against accidental breaking changes. Deterministic, SourceLinked builds; tag-driven versioning (MinVer); symbol packages. Fast: a BenchmarkDotNet run extracts on the order of ~12,000 files/sec (4-way parallel) over a mixed text-file corpus. Extensible without forking Adding a format is one interface: public sealed class MyExtractor : IFileExtractor { public bool CanHandle(string ext) => ext == ".xyz"; public ExtractedContent Extract(string path) => new(metadata, text); } options.Extractors.Add(new MyExtractor()); // tried before the built-ins Enter fullscreen mode Exit fullscreen mode Add-ons reference only the tiny Scrubkit.Abstractions contracts package — no PDF or image dependencies pulled in. There's already a growing family: Scrubkit.Email — .eml (MIME): headers → metadata, body → text. Scrubkit.OpenDocument — .odt / .ods / .odp from LibreOffice / OpenOffice. Scrubkit.Epub — .epub e-books. Scrubkit.Extensions.DependencyInjection — services.AddScrubkit(…) for ASP.NET Core and worker hosts. Redaction, if you want it Text is returned exactly as read. Opt into scrubbing common PII (emails, phones, Luhn-checked cards, SSNs, IPs) by supplying an IRedactor — it's entirely your call, and it's explicitly best-effort, not a compliance tool. Try it in 30 seconds dotnet add package Scrubkit # or, without installing: dotnet run --project samples/Scrubkit.Playground Enter fullscreen mode Exit fullscreen mode NuGet: https://www.nuget.org/packages/Scrubkit Site + docs: https://jjopensoftworks-blip.github.io/Scrubkit/ GitHub: https://github.com/jjopensoftworks-blip/Scrubkit — a ⭐ helps a lot. Feedback and format requests welcome. What would you point it at?