variant-confidence v0.1.0: a calibrated confidence layer for variant-effect pathogenicity scores

variant-confidence v0.1.0: a calibrated confidence layer for variant-effect pathogenicity scores State-of-the-art variant-effect models are accurate in cross-validation but their scores are poorly calibrated on temporal data. variant-confidence adds an auditable calibration layer on top of existing predictors — it does not train a new model. The problem: accuracy is not trust Protein variant-effect predictors (AlphaMissense, ESM-1v, EVE) report pathogenicity scores, but a clinician or researcher needs to know how much to trust the number, not just its rank. The gap is calibration, not accuracy: AnnotateMissense (2026) reports MCC 0.94 in cross-validation, dropping to 0.76 on temporal ClinVar, accuracy 0.8798. A raw score near 0.9 may not mean 90% probability. Acting on an uncalibrated score is a risk. What it does variant-confidence wraps an existing predictor's score and produces a calibrated, uncertainty-aware output: Probability calibration (AC1): Platt scaling or isotonic regression over a separate holdout. Selectable, not hardcoded. Conformal prediction (AC1b): coverage 1−α intervals, split or Mondrian by gene. ECE (AC2, AC9): Expected Calibration Error reported before/after calibration, with bootstrap CI and per-bin counts. Bins with too few samples are flagged as low-reliability. Leakage-free split (AC3): temporal split by ClinVar release date with gene isolation — the same gene never appears in both train and test. This is unit-tested. Missing-score handling (AC4): works with AlphaMissense or ESM-1v alone; emits an explicit warning instead of failing silently. Non-deceptive reporting (AC7): every result includes interval/ECE + method + threshold, never a bare calibrated score. Verification (clean clone, no network) Built under a three-party governance loop: implement → independent audit in a clean clone → merge approval. ruff check . → All checks passed. pytest tests/ → 28 passed in 8.90s (offline fixture). An honest bug we caught in audit The first ECE test reported a "perfect" drop from raw ECE 0.4275 to ~0 after calibration. The audit found this was degenerate: the synthetic score generator ignored the real labels and produced an independent random true_p, so the calibrator simply collapsed to the base rate (91.5% pathogenic) — ECE≈0 by construction, not by merit. AUC of the raw score vs real labels was 0.51. The fix derives true_p from the real label with noise, so the synthetic score is genuinely discriminative but miscalibrated. The acceptance criterion is now: ECE drops and AUC is preserved after calibration. This is in the committed code and the 28 passing tests. What remains open (honestly) AlphaMissense license ambiguity: the official README says CC BY 4.0, but the distributed TSV header, Ensembl VEP plugin, and EBI page state CC BY-NC-SA 4.0. The contradiction is unresolved; treat the data as restricted (non-commercial) until clarified. The software is AGPL-3.0-or-later and fully self-contained. The end-to-end join with real AlphaMissense scores is implemented, but the flagship path is covered by an offline fixture, not a live download in CI. Try it pip install variant-confidence variant-confidence --method platt --offline Enter fullscreen mode Exit fullscreen mode Stack Python >=3.10, numpy, pandas, scikit-learn AGPL-3.0-or-later Links Repo: https://github.com/amurlaniakea/variant-confidence Release v0.1.0: https://github.com/amurlaniakea/variant-confidence/releases/tag/v0.1.0 License: AGPL-3.0-or-later — Pedro Sordo Martínez (amurlaniakea@gmail.com), 2026.