My RabbitMQ library looked production-ready. It couldn't survive a single reconnect.

For a couple of years I had a RabbitMQ client library sitting in a private repo. On paper it was impressive: automatic reconnection, channel pooling, a circuit breaker, four rate-limiting strategies, gzip compression, dead letter queues, worker-thread consumers, delayed messages. Over 1,200 lines in the main class alone. I never quite trusted it enough to publish. Recently I decided to find out whether that instinct was right. It was. The single most important feature of a message-broker client — surviving a connection drop — had never worked. Not "worked poorly". Never worked. And nothing in the repo could have told me, because there wasn't a single test. This is the story of the audit, the rebuild, and the release of @pinceladasdaweb/rabbitmq. It's also a collection of lessons I wish I'd internalized years earlier, because none of these bugs were exotic. They were all sitting in plain sight, waiting for a failure path to be exercised for the first time in production. The audit: what "looks done" actually hides The library had the shape of a finished product. Rich API, detailed README, twenty example scripts. Here's a sample of what a deep read of the code actually found. The reconnection that reconnected to nothing. The connection layer dutifully detected drops, retried with exponential backoff, and re-established the TCP connection. Then... nothing. The channel pool — the thing every publish and consume goes through — was only ever created inside connect(), and the reconnection path never called it. After any network blip, every operation would throw Not connected forever, while the connection state proudly reported connected. The recovery code existed; it recreated consumers using channels captured in closures from before the drop — channels that were dead. It had clearly never been executed. The rate limiter that crashed on its own feature. The block-a-key feature stored blocked keys in a Set... and called this.blocked.set(key, Date.now()) on it. Set has no .set(). Calling blockKey() threw a TypeError, every time, since the day it was written. The circuit breaker state that didn't exist. The public API exposed getCircuitBreakerState(), which called this.#circuitBreaker.getState() — a method the circuit breaker class simply did not have. The dead-letter helper with the wrong signature. moveToDeadLetter() called the publish method with four arguments in the wrong order, so the name of the queue was published as the message body, to the wrong exchange. The library that killed your process. The constructor registered SIGINT, SIGTERM, uncaughtException and unhandledRejection handlers — and called process.exit() in them. A library. Any unhandled rejection anywhere in your application would cause my dependency to shut your process down. None of these are subtle bugs. They're the kind of thing the first test you ever write catches. Which brings me to the first lesson. Lesson 1: features are cheap; invariants are expensive. An API surface can look complete while every failure path is fiction. The happy path is maybe 20% of an infrastructure library — the rest is what happens during the bad minutes, and that's exactly the part that "it runs on my machine" never exercises. If you claim resilience, your CI has to actually break things The rebuild started with the reliability core: reconnection has to restore the channel pool, re-assert topology, and recreate every consumer on fresh channels. But I'd been burned by "code that looks like it recovers", so the real work was making the claim falsifiable. The integration suite now runs against a real RabbitMQ (Docker locally, a service container in GitHub Actions) and includes this test — the one I care most about in the entire repo: test('recovers channel pool and consumers after a forced connection drop', async () => { const reconnected = new Promise(resolve => rabbitMQ.once('reconnected', resolve)) // Ask the broker to close every AMQP connection, simulating a real outage. await forceCloseAllConnections() // DELETE /api/connections/* via the management API await reconnected const countBefore = received.length await rabbitMQ.publish(ROUTING_KEY, { after: 'reconnect' }) await waitFor(() => received.length > countBefore) assert.equal(rabbitMQ.getClusterStatus().connectionState, 'connected') }) Enter fullscreen mode Exit fullscreen mode No mocks. The broker genuinely kills the connection; the library genuinely has to come back: pool rebuilt, exchange re-asserted, consumer re-subscribed on a new channel, message flowing again. This runs on every pull request. Using the management HTTP API for the kill switch (instead of docker restart) was a small trick that paid off: it works identically on a laptop and inside a CI service container, and it's fast. Lesson 2: a resilience feature without a test that injects the failure is a rumor. The test that kills your broker connection is worth more than the thousand lines of recovery code it validates. The adversarial review: 42 candidates, 10 confirmed bugs With the core rebuilt and a test harness in place, I ran a structured review of the source: multiple independent passes over the code, each hunting from a different angle — line-by-line, lifecycle invariants, cross-module contracts, plus efficiency and design-depth sweeps. Every candidate finding then got adversarially verified against the code before I accepted it. Ten confirmed bugs came out. Three of my favorites, because each one represents a category: 1. The option that silently discarded messages. async retryOperation (operation, maxRetries = 3, delay = 1000) { for (let attempt = 1; attempt version) and bumps from that, never from package.json. Whatever half-finished state a previous run left behind, the next run self-corrects. Serialize your releases. A concurrency: { group: publish } block means rapid merges queue instead of racing. Lesson 5: release automation fails in the middle, not at the edges. Design every step to be safe to re-run, and derive state from systems of record, not from files a previous failed run may or may not have updated. Where it landed The library is now public: @pinceladasdaweb/rabbitmq. npm install @pinceladasdaweb/rabbitmq Enter fullscreen mode Exit fullscreen mode import RabbitMQ from '@pinceladasdaweb/rabbitmq' const rabbit = new RabbitMQ({ endpoints: ['localhost:5672'], exchange: { name: 'orders', type: 'direct' } }) await rabbit.connect() await rabbit.subscribe('order-processing', async (content, message) => { await handleOrder(content) // auto-ack on success, nack → DLQ on throw }) await rabbit.publish('order.created', { id: 42 }) // publisher-confirmed rabbit.on('reconnected', () => console.log('survived an outage, consumers restored')) Enter fullscreen mode Exit fullscreen mode What's in the box: automatic reconnection with full state recovery (pool, topology, consumers), publisher confirms on everything, per-key rate limiting (token bucket, leaky bucket, fixed and sliding windows), a circuit breaker that's isolated from consumer failures, transparent gzip compression, first-class dead-letter support (createQueue, moveToDeadLetter, processDeadLetterQueue), delayed messages via the official plugin, worker-thread parallel consumers with automatic respawn, dependency-ordered sequential processing, consumer lifecycle management (unsubscribe, consumerCancelled/consumerRecovered/consumerLost events), and TypeScript declarations checked in strict mode on CI. Guarding all of it: 87 automated tests — 72 unit, 15 integration against a real broker including the forced-outage scenario — running on Node 22 and 24 on every pull request, plus 22 runnable examples that double as living documentation. And in the spirit of the whole post, the honest part: this library has excellent test coverage and zero production mileage under other people's workloads. Those are different kinds of confidence, and only one of them can be built alone. If you run RabbitMQ in Node and any of this resonates, I'd genuinely love an issue, a hard question, or a war story from your queue topology: github.com/pinceladasdaweb/rabbitmq. The library I was too embarrassed to publish taught me more in this rebuild than it did in all the years it sat in a folder looking finished. Turns out "looking finished" was the bug.