Substack Scraper in Node.js: Post Archives, Likes, Comments, and the Walls You'll Hit (2026)

Substack has quietly become one of the more useful datasets on the web. If you do competitor research for a newsletter, track a niche, or want to know which topics actually get people to hit the heart button, the numbers are sitting right there on every publication. In this post I'll build a small Node.js script that: Pages through any publication's post archive Exports title, date, author, paywall status, word count, likes, comments, and restacks to CSV Prints a quick summary (paywalled share, top posts by likes) Pulls the comment thread for a single post No headless browser and no dependencies: Node 18+ has fetch built in. Everything below was tested against live publications in October 2026, and I'll point out the quirks I ran into along the way. Before you start: stick to public data, keep request rates low, and read Substack's terms. The endpoints below are the ones Substack's own web app uses. They're not a documented public API, so they can change without notice. Your options, briefly RSS. Every publication has /feed. Great for "what's new", but you only get the most recent posts and no engagement numbers. HTML pages. Works, but you'd be parsing markup for numbers that are already available as JSON. The archive JSON (this post). The archive page loads its data from /api/v1/archive. It returns clean JSON with engagement counts, and it paginates. Step 1: Find the archive endpoint Open any publication's /archive page, scroll, and watch the network tab. You'll see requests like this: https://www.lennysnewsletter.com/api/v1/archive?sort=new&offset=0&limit=12 Enter fullscreen mode Exit fullscreen mode Each item is a post object with a lot of fields. The ones I actually use: Field What it is title, slug, canonical_url The basics post_date ISO timestamp audience everyone, only_paid, or founding type newsletter, podcast, thread wordcount Full length, even for paywalled posts reaction_count, comment_count, restacks Engagement publishedBylines Authors (guest posts have more than one) sort=top also works and gives you the most-liked posts first, which is handy if you only care about what performed. Step 2: Page through it politely Here's the whole scraper. Save it as scrape.mjs: // usage: node scrape.mjs https://www.lennysnewsletter.com [maxPosts] import { writeFile } from "node:fs/promises"; const base = new URL(process.argv[2] ?? "https://www.lennysnewsletter.com").origin; const maxPosts = Number(process.argv[3] ?? 60); const PAGE = 12; // larger values return odd page sizes, see below const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); async function getJSON(url, tries = 4) { for (let i = 0; i = 500) { const wait = Number(res.headers.get("retry-after")) * 1000 || 2 ** i * 2000; console.warn(`${res.status} on ${url}, waiting ${wait}ms`); await sleep(wait); continue; } throw new Error(`${res.status} ${res.statusText} for ${url}`); } throw new Error(`gave up on ${url}`); } async function fetchArchive(sort = "new") { const posts = []; for (let offset = 0; posts.length ({ id: p.id, date: p.post_date?.slice(0, 10), title: p.title, type: p.type, audience: p.audience, words: p.wordcount, likes: p.reaction_count, comments: p.comment_count, restacks: p.restacks, author: p.publishedBylines?.map((b) => b.name).join(" & "), url: p.canonical_url, }); const csvCell = (v) => { const s = v == null ? "" : String(v); return /[",\n]/.test(s) ? `"${s.replaceAll('"', '""')}"` : s; }; const posts = (await fetchArchive()).map(toRow); const cols = Object.keys(posts[0]); const csv = [cols.join(","), ...posts.map((r) => cols.map((c) => csvCell(r[c])).join(","))].join("\n"); await writeFile("posts.csv", csv + "\n"); Enter fullscreen mode Exit fullscreen mode Titles contain commas and quotes all the time, so the csvCell escaping isn't optional. Spreadsheet apps will happily shift every column otherwise. Step 4: A quick summary const paid = posts.filter((p) => p.audience !== "everyone"); const avg = (xs) => Math.round(xs.reduce((a, b) => a + b, 0) / (xs.length || 1)); console.log(`paywalled: ${paid.length} (${Math.round((100 * paid.length) / posts.length)}%)`); console.log(`avg likes free: ${avg(posts.filter((p) => p.audience === "everyone").map((p) => p.likes))}, paid: ${avg(paid.map((p) => p.likes))}`); for (const p of [...posts].sort((a, b) => b.likes - a.likes).slice(0, 5)) console.log(` ${p.likes} likes, ${p.comments} comments ${p.date} ${p.title}`); Enter fullscreen mode Exit fullscreen mode Running it against two very different publications shows why this is interesting: $ node scrape.mjs https://www.lennysnewsletter.com 36 paywalled: 29 (81%) avg likes free: 335, paid: 496 $ node scrape.mjs https://astralcodexten.substack.com 24 paywalled: 2 (8%) ... Enter fullscreen mode Exit fullscreen mode One is mostly paywalled with comparatively few comments per post; the other is almost entirely free, and its top posts had hundreds of comments each. Same platform, completely different playbooks, and you can see it from 30 lines of output. Step 5: Comments for a single post Comments live on a separate endpoint keyed by the post's numeric id: async function fetchComments(postId) { const url = `${base}/api/v1/post/${postId}/comments?all_comments=true&sort=best_first`; const { comments = [] } = await getJSON(url); const flat = []; const walk = (list, depth = 0) => { for (const c of list) { flat.push({ depth, name: c.name, likes: c.reaction_count, date: c.date, body: c.body }); walk(c.children ?? [], depth + 1); } }; walk(comments); return flat; } Enter fullscreen mode Exit fullscreen mode Replies are nested under children, so a small recursive walk flattens them into rows with a depth column. Threads go deep: one popular post in my test nested 28 levels. Deleted comments stay in the tree with deleted: true and a null body (so their replies keep their place), and the number you collect won't exactly match the post's comment_count. Filter deleted rows before any text analysis. Where this approach hits a wall The limit parameter is weird. Up to 20 gave me exactly what I asked for. 25 to 49 all returned 23 posts, and limit=100 returned a single post. Keep pages small (I use 12, which is what the web app uses) and paginate with offset. Paywalled posts are previews. For an only_paid post, /api/v1/posts/ returns a body_html that stops at the paywall. On one 4,153-word post I got about 2,300 words. wordcount still reports the full length, which is useful on its own. Custom domains redirect. astralcodexten.substack.com answers with a 301 to www.astralcodexten.com. fetch follows it, but if you dedupe publications by hostname, normalize after the redirect. Cross-publication search is a different beast. Everything above works per publication. Finding publications by keyword means hitting substack.com's own internal search endpoints, which are even less stable. It's undocumented. If a field disappears, your CSV quietly fills with blanks. Check that reaction_count is a number on the first page and fail loudly if it isn't. When you'd rather not maintain it If you're tracking a lot of publications, or you need search, notes, and writer profiles too, a scraping API can take over the fetching. I use ScrapeAtlas* for this. Its /v1/substack/posts endpoint takes the publication url plus optional limit (1 to 100), offset, and sort (new or top), and returns a posts array with the same field names you saw above, plus a nextOffset for the next page: async function apiPosts(pubUrl, offset = 0) { const q = new URLSearchParams({ url: pubUrl, limit: "50", offset: String(offset), sort: "new" }); const res = await fetch(`https://api.scrapeatlas.com/v1/substack/posts?${q}`, { headers: { "x-api-key": process.env.SCRAPEATLAS_API_KEY }, }); const data = await res.json(); // data.status: "success" or "partial"; data.coverage.warnings explains gaps return { posts: data.posts ?? [], next: data.nextOffset }; } Enter fullscreen mode Exit fullscreen mode Because the field names match, toRow and the CSV code work unchanged. There are also /v1/substack/publication, /v1/substack/post/comments, and /v1/substack/search endpoints if you need publication details, comment threads, or keyword search. Wrapping up A few publications, checked weekly: the archive endpoint and the script above are all you need, and it costs nothing. Keep pages small and paginate with offset; don't trust large limit values. Treat paywalled bodies as previews, and lean on wordcount and engagement fields instead. Many publications, search, or anything you need to rely on: that's where maintaining it yourself stops being fun. If the archive endpoint starts returning something unexpected, leave a comment and I'll update the post. *I build it.