TRUE Coverage: How We Achieved 90% Faster CI by Measuring What Tests Actually Do

TL;DR: Use per-test coverage data to build a reverse map (file → tests that touch it). Git diff + map lookup = run only relevant tests. 43min → 4min CI time. The Problem Everyone Has Your test suite runs for 45 minutes. You change one file. Should you: Run all 2,000 tests? (Safe but slow) Run tests with matching filenames? (Fast but broken) Manually tag tests? (Gets stale immediately) We tried all three. They all failed. The Insight: Stop Guessing, Start Measuring What if we just measured what each test actually executes? Coverage tools have done this for years: { "src/MyComponent.js": { "statements": { "0": 5, "1": 12, "2": 0 } } } Enter fullscreen mode Exit fullscreen mode Key insight: Coverage tools report per test run, not per test file. All we needed: save coverage after each individual test. The Architecture (5 Steps) Step 1: Instrument the Code // Build config (webpack/vite/rollup/etc) export default { plugins: [ process.env.COLLECT_COVERAGE && coveragePlugin({ include: 'src/**/*', exclude: ['**/*.test.*'] }) ] } Enter fullscreen mode Exit fullscreen mode Works with: Istanbul (JS), coverage.py (Python), SimpleCov (Ruby), JaCoCo (Java) Step 2: Collect Coverage Per Test // Test framework afterEach hook afterEach(async () => { const coverage = getCoverageData(); saveCoverage(`coverage-${testName}.json`, coverage); }); Enter fullscreen mode Exit fullscreen mode Step 3: Build the Reverse Map const map = {}; for (const coverageFile of allCoverageFiles()) { const testName = extractTestName(coverageFile); const coverage = JSON.parse(read(coverageFile)); map[testName] = []; for (const [sourceFile, data] of Object.entries(coverage)) { if (wasExecuted(data)) { map[testName].push(sourceFile); } } } Example output: { "tests/user-profile.test.js": [ "src/components/Profile.jsx", "src/components/Card.jsx", "src/utils/formatters.js" ] } Enter fullscreen mode Exit fullscreen mode The magic: The test told us what it executed. No parsing, no guessing. Step 4: Detect Tests CHANGED=$(git diff origin/main...HEAD --name-only) TESTS=$(lookupTestsInMap "$CHANGED") npm test $TESTS Enter fullscreen mode Exit fullscreen mode Step 5: Auto-Update npm run build-coverage-map git add coverage-map.json git commit -m "Update coverage map [skip ci]" git push Enter fullscreen mode Exit fullscreen mode The Results ┌───────────┬────────┬───────┐ │ Metric │ Before │ After │ ├───────────┼────────┼───────┤ │ Tests run │ 47 │ 4 │ ├───────────┼────────┼───────┤ │ Time │ 18 min │ 2 min │ ├───────────┼────────┼───────┤ │ Accuracy │ 15% │ 95%+ │ ├───────────┼────────┼───────┤ │ CI cost │ $12 │ $1.20 │ └───────────┴────────┴───────┘ Enter fullscreen mode Exit fullscreen mode Real PR example: Changed: 1 component CI time: 43 minutes → 4 minutes Language Examples JavaScript (Istanbul + Jest/Playwright) afterEach(async () => { const coverage = global.__coverage__; saveCoverage(coverage); }); Python (coverage.py + pytest) @pytest.fixture(autouse=True) def save_coverage(request): yield save_coverage_data() Ruby (SimpleCov + RSpec) RSpec.configure do |config| config.after(:each) { save_coverage } end Java (JaCoCo + JUnit) @AfterEach void saveCoverage() { save(getCoverage()); } Enter fullscreen mode Exit fullscreen mode The Tradeoffs What Works: ✅ Zero maintenance (auto-updates) ✅ Handles refactors naturally ✅ Incremental updates What Required Tuning: ⚠️ Build tool noise (bundlers include common code) ⚠️ Shared utilities (some files in 50+ tests) ⚠️ Git history churn (auto-commits) Solutions: Keyword filtering, threshold checks, minimum change limits The Key Insight Traditional approaches model relationships: Import graphs → static dependencies Pattern matching → naming conventions Manual tags → human intent We don't model. We measure. Try It Yourself ~500 lines in any language: Instrument (100 lines) Hook (50 lines) Parse (200 lines) Lookup (100 lines) CI (50 lines) Complexity is in filtering noise, not the pattern. Conclusion Stop guessing. Start measuring. Coverage tools have measured execution for decades. We just use them per test instead of per run. If your CI is slow, this will fix it. Not by being clever, but by being honest. Questions? Comment below or reach out on Twitter/LinkedIn