Calibrated uncertainty and selective prediction for retinal disease screening — "when should a deep model abstain and route to a clinician, and how much does it actually help?"
Deep learning models for diabetic retinopathy (DR) screening routinely report 85–95% accuracy on standard benchmarks. The number that doesn't appear in those papers is the one a clinician actually needs: what fraction of confidently-wrong predictions slip through, and how do we catch them?
This repository contributes two pieces:
- A clean reproducible benchmark of five families of uncertainty quantification methods on the same backbone and same fixed splits, evaluated through the lens of selective prediction (v1.0.0).
- OACSP — Ordinal-Aware Class-Conditional Selective Prediction — a novel post-hoc abstention rule that replaces the standard global confidence threshold with per-class thresholds calibrated either to operator-specified per-class target recall or to minimize ordinal-distance-weighted expected cost (v1.1.0).
| Method | Family | What it estimates | Cost vs. baseline |
|---|---|---|---|
| Softmax confidence | Deterministic | Aleatoric ≈ epistemic, mixed | 1× (free) |
| MC Dropout | Bayesian approx. | Mean + epistemic spread across T passes | T× inference |
| Deep Ensembles | Frequentist Bayesian | Predictive variance across M independent models | M× train + inference |
| Temperature Scaling | Post-hoc calibration | Sharpens / softens softmax to match accuracy | ~0× (1-param fit) |
| Evidential Deep Learning | Subjective logic | Dirichlet over class probs, explicit uncertainty | 1× (custom loss) |
| Conformal Prediction | Distribution-free | Calibrated prediction sets with coverage guarantee | ~0× (post-hoc) |
| OACSP (this work, v1.1.0) | Selective rule | Per-class abstention thresholds with ordinal cost | 0× (post-hoc, ~370 LOC) |
Why this matters clinically: screening tools live or die on false negatives. A 95% accurate model that confidently mis-classifies the 5% of severe-DR cases it misses is worse than an 85% accurate model that abstains on its hardest 15% and surfaces them for human review. Selective prediction reframes "how accurate is the model" as "how accurate is the model on the cases it chose to answer."
✅ v1.1.0 released (
2400870, 2026-06-14). Seeresults/results.mdfor the full breakdown including §8 OACSP results. External validation on IDRiD + Messidor-2 coming next (v1.2.0).
Baseline + post-hoc uncertainty methods (v1.0.0 numbers):
| Method | Acc ↑ | QWK ↑ | ECE ↓ | NLL ↓ | AURC ↓ | Excess-AURC ↓ | Sel. Acc @ 80% coverage ↑ |
|---|---|---|---|---|---|---|---|
| Softmax (deterministic) | 0.809 | 0.865 | 0.146 | 1.023 | 0.0793 | 0.0598 | 0.882 |
| Temperature Scaling (T=2.25) | 0.809 | 0.865 | 0.055 | 0.576 | 0.0852 | 0.0657 | 0.884 |
| MC Dropout (T=30, MI signal) | 0.809 | 0.865 | 0.146 | — | 0.0756 | 0.0561 | 0.882 |
| Conformal (split, α=0.10) | 0.809 | 0.865 | — | — | 0.1112 | 0.0917 | 0.884 |
| Deep Ensembles (M=5) | – | – | – | – | – | – | – |
| Evidential DL | – | – | – | – | – | – | – |
QWK = quadratic-weighted Cohen's κ (APTOS official metric). ECE = Expected Calibration Error (15 bins). AURC = Area Under Risk–Coverage curve. Excess-AURC isolates the uncertainty signal from baseline accuracy.
Three different methods, three different wins. Temperature Scaling cuts ECE by 62% (0.146 → 0.055) with one scalar. MC Dropout with mutual information has the lowest AURC. Conformal hits its 90% coverage target almost exactly (empirical 90.2%) with average set size 1.35.
Per-class abstention rate at ≈80% overall coverage (calibrated on val, evaluated on disjoint test):
| Class | Global threshold | OACSP equalized recall | OACSP ordinal cost |
|---|---|---|---|
| 0 — No DR | 1.5% | 7.0% | 3.3% |
| 1 — Mild | 35.7% | 23.2% | 58.9% |
| 2 — Moderate | 36.7% | 12.7% | 26.0% |
| 3 — Severe | 44.8% | 10.3% | 31.0% |
| 4 — Proliferative | 29.5% | 6.8% | 20.5% |
Headline: OACSP equalized-recall reduces Severe-DR abstention by 4.4× (44.8% → 10.3%) and Proliferative-DR abstention by 4.3× (29.5% → 6.8%) versus the standard global threshold. OACSP ordinal-cost-weighted achieves selective QWK 0.9218 vs. 0.9068 for the global baseline at near-identical coverage (0.820 vs. 0.809).
The rule is post-hoc — no retraining — and operates on the saved softmax outputs. See docs/oacsp_method.md for the method writeup and the novelty quadrant table vs. prior work.
| Method | Acc ↑ | QWK ↑ | ECE ↓ | AURC ↓ | Excess-AURC ↓ | Sel. Acc @ 80% coverage ↑ |
|---|---|---|---|---|---|---|
| (planned) | – | – | – | – | – | – |
| Method | Acc ↑ | QWK ↑ | ECE ↓ | AURC ↓ | Excess-AURC ↓ |
|---|---|---|---|---|---|
| (planned) | – | – | – | – | – |
| Reference | Backbone | Dataset | Reported QWK | Notes |
|---|---|---|---|---|
| APTOS 2019 winner (kaggle, 2019) | Inception-ResNet v2 ensemble | APTOS (test) | 0.936 | TTA + 4× ensemble, no uncertainty / no abstention |
| Krause et al. 2018 (Google) | Inception-v4 | Proprietary | – | Closed dataset; not directly comparable |
| (this work, v1.1.0) | EfficientNet-B0 + selective + OACSP | APTOS (internal test) | 0.865 baseline, 0.9218 selective QWK with OACSP at 82% coverage | Single-backbone for clean ablations |
The point of this work is not to chase the absolute best QWK number — that race is already over. The contribution is the selective-prediction comparison on a fixed backbone, plus the OACSP novel abstention rule (v1.1.0).
| Dataset | Role | Size | Classes | License | Notes |
|---|---|---|---|---|---|
| APTOS 2019 | Train / val / internal test | 3,662 images, 5-class DR | 5 (No DR → Proliferative) | Kaggle competition (free) | Indian population, smartphone fundus camera |
| Messidor-2 | External validation #1 | 1,748 images | Re-mapped to 5-class DR | Free with registration | French, multiple camera vendors → real distribution shift |
| IDRiD | External validation #2 | 516 images | 5-class DR + DME grading | Free | Indian Diabetic Retinopathy Image Dataset |
All datasets are public and require either Kaggle competition acceptance or a free registration. No private or hospital data. This is a deliberate design choice — every result is reproducible by anyone with a Kaggle account.
Stratified train/val/test splits are computed once and the CSVs committed to data/splits/aptos2019/, guaranteeing identical splits across every uncertainty method.
┌───────────────────────┐
│ Fundus image (RGB) │
└───────────┬───────────┘
│
▼
┌───────────────────────┐
│ Backbone (EffNet-B0) │
│ ImageNet-pretrained │
└───────────┬───────────┘
▼
┌───────────────────────┐
│ Dropout (p=0.3) │ ← kept active at inference
│ │ for MC Dropout sampling
└───────────┬───────────┘
▼
┌───────────────────────┐
│ Linear → 5 classes │
└───────────┬───────────┘
▼
┌─────────────────────┴─────────────────────┐
│ │
▼ ▼
┌──────────────────┐ ┌──────────────────────────┐
│ Softmax / Temp. │ │ MC Dropout / Ensembles / │
│ Scaling / │ │ Evidential / Conformal │
│ Conformal │ │ → predictive + epistemic │
└────────┬─────────┘ └────────────┬─────────────┘
│ │
└─────────────────┬───────────────────────┘
▼
┌──────────────────────┐
│ Selective Prediction │
│ AURC, risk–coverage │
│ selective accuracy │
└──────────────────────┘
Single backbone, multiple uncertainty heads — this is the design choice that makes the comparison clean. Implementations live in:
src/models/backbone.py—RetinalClassifier(timm backbone + dropout-friendly head)src/uncertainty/— one module per uncertainty methodsrc/selective/risk_coverage.py— AURC, excess-AURC, selective accuracy (modality-agnostic)src/evaluation/— calibration metrics + reliability diagrams
git clone https://github.com/ShahnawazKakarh/retinal-selective-prediction.git
cd retinal-selective-prediction
python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pre-commit install
cp .env.example .env # then fill in WANDB_API_KEY (optional)# See docs/kaggle_setup.md for the full walkthrough.
# Short version: open notebooks/kaggle_train_baseline.py on Kaggle,
# attach the APTOS 2019 dataset, set Internet=On + GPU T4, run all.# Make stratified splits (one-time)
python scripts/prepare_aptos_splits.py \
--train-csv /path/to/aptos/train.csv \
--output-dir data/splits/aptos2019
# Train baseline
python scripts/train.py \
--config configs/baseline.yaml \
--splits-dir data/splits/aptos2019 \
--images-dir /path/to/aptos/train_imagespython scripts/evaluate.py \
--run-dir experiments/runs/baseline_efficientnet_b0_aptos \
--splits-dir data/splits/aptos2019 \
--images-dir /path/to/aptos/train_images
# → metrics.json, predictions.csv, confusion_matrix.png, reliability_diagram.png# MC Dropout — T stochastic forward passes
python scripts/run_mc_dropout.py \
--run-dir experiments/runs/baseline_efficientnet_b0_aptos \
--splits-dir data/splits/aptos2019 \
--images-dir /path/to/aptos/train_images \
--n-samples 30More methods land as the project progresses — see Roadmap.
retinal-selective-prediction/
├── configs/ # one YAML per reproducible experiment
├── src/
│ ├── data/ # APTOS dataset, stratified splits, augmentations
│ ├── models/ # backbones with dropout hooks
│ ├── uncertainty/ # MC dropout, ensembles, temp scaling, EDL, conformal
│ ├── selective/ # risk–coverage, AURC, selective policies
│ ├── training/ # training loop, AMP, cosine+warmup LR, W&B
│ ├── evaluation/ # ECE, reliability diagram, confusion matrices
│ └── utils/ # seeding, config loading
├── scripts/ # CLI entry points (train, evaluate, run_*)
├── notebooks/ # Kaggle / Colab notebooks (jupytext .py)
├── data/splits/ # committed CSVs — same splits across all runs
├── experiments/runs/ # run outputs (gitignored)
├── results/ # paper-ready tables + plots
├── docs/ # Kaggle setup, dataset notes
└── tests/ # pytest
Full writeup of scientific framing, prior work, novelty positioning, and the v1.0.0 → v1.1.0 → v2.0.0 plan lives in docs/research_notes.md.
A few design decisions called out so they don't surprise future-me or reviewers:
- Single backbone for all uncertainty methods. Mixing backbones across methods is the silent killer in this literature — any AURC difference becomes "is it the uncertainty method or the model?" Fixed backbone makes the comparison genuinely apples-to-apples.
- Splits committed to git. Every method reads the exact same train.csv / val.csv / test.csv. No accidental data shuffling between experiments.
- Excess-AURC is the headline. Raw AURC is dominated by overall accuracy — a stronger model has lower AURC almost regardless of uncertainty quality. Excess-AURC subtracts the oracle's AURC and isolates "how good is the uncertainty signal."
- External validation is non-negotiable. Messidor-2 + IDRiD are pre-committed as held-out test sets — never used for hyperparameter selection, never used for calibration. They exist solely to answer "does the method generalize?"
- Negative results stay in. If MC Dropout loses to plain softmax confidence on this backbone (it sometimes does), the paper says so. The contribution is the honest comparison, not a flattering hero method.
- Repo scaffolding (gitignore, pre-commit secret scanning, deps, pyproject)
- APTOS dataset loader, stratified split utility, train/eval augmentations
- EfficientNet-B0 backbone with MC-Dropout-friendly head
- Training loop (AMP, cosine+warmup LR, early stopping on QWK, W&B logging)
- Deterministic evaluation (ECE, reliability diagram, confusion matrix)
- Kaggle notebook for baseline training
- MC Dropout uncertainty + risk–coverage / AURC selective analysis
- Baseline trained — val QWK 0.889, test QWK 0.865
- Temperature Scaling — ECE 0.146 → 0.055
- MC Dropout (T=30) — best AURC across all methods
- Conformal Prediction (Split + APS) — 90.2% empirical coverage @ α=0.10
- v1.0.0 released to Zenodo with DOI (June 13, 2026)
- OACSP — Ordinal-Aware Class-Conditional Selective Prediction (novel)
- Proper val/test split fix for OACSP calibration
- v1.1.0 released to Zenodo with DOI (June 14, 2026)
- v1.2.0: Multi-seed runs (seeds 7, 137) for mean ± SD
- v1.2.0: IDRiD external validation
- v1.2.0: OACSP transfer test on IDRiD
- v2.0.0: Deep Ensembles (M=5 independent seeds)
- v2.0.0: Evidential Deep Learning (Dirichlet head + custom loss)
- v2.0.0: Messidor-2 external validation
- v2.0.0: Equity audit — do abstention rates correlate with image-quality proxies?
- Paper draft — target IEEE J-BHI submission late 2026
This project is published as versioned Zenodo releases, each with its own DOI. The Concept DOI on Zenodo resolves to the latest version; the version-specific DOI pins to an exact artifact. The same project also progresses toward an IEEE J-BHI submission for v2.0.0.
| Version | Scope | Status | Version DOI |
|---|---|---|---|
| v1.0.0 | Single-seed benchmark on APTOS 2019 (4 uncertainty methods) | 🟢 Released | 10.5281/zenodo.20681524 |
| v1.1.0 | + OACSP (novel post-hoc abstention rule) + proper val/test split | 🟢 Released | 10.5281/zenodo.20695855 |
| v1.2.0 | + Multi-seed variance + IDRiD external validation + OACSP transfer test | 🟡 Planned | |
| v2.0.0 | + Deep Ensembles + Evidential DL + Messidor-2 + equity audit | 🔵 Target: IEEE J-BHI |
Concept DOI (always resolves to the latest version): 10.5281/zenodo.20681415
Step-by-step Zenodo release & ORCID flow: docs/zenodo_release.md
Kaggle reproducibility runbook: docs/kaggle_setup.md
OACSP method writeup: docs/oacsp_method.md
Built technical report PDF source: report/ (run python report/build.py to rebuild)
Machine-readable citation: CITATION.cff
If you use this benchmark in your work, please cite it using the CITATION.cff metadata or the BibTeX below:
@misc{khan2026retinalselective,
author = {Khan, Muhammad Shahnawaz},
title = {Selective Prediction with Calibrated Uncertainty
for Diabetic Retinopathy Screening:
A Single-Backbone Benchmark on APTOS 2019
with Ordinal-Aware Class-Conditional Selective Prediction},
year = {2026},
version = {1.1.0},
publisher = {Zenodo},
doi = {10.5281/zenodo.20681415},
url = {https://github.com/ShahnawazKakarh/retinal-selective-prediction}
}The Zenodo Concept DOI 10.5281/zenodo.20681415 always resolves to the latest version. v1.1.0 specifically is at 10.5281/zenodo.20695855; v1.0.0 specifically is at 10.5281/zenodo.20681524.
Core papers this work builds on:
- Gal & Ghahramani, Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, ICML 2016
- Lakshminarayanan, Pritzel & Blundell, Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles, NeurIPS 2017
- Guo, Pleiss, Sun & Weinberger, On Calibration of Modern Neural Networks, ICML 2017
- Geifman & El-Yaniv, Selective Classification for Deep Neural Networks, NeurIPS 2017
- Sensoy, Kaplan & Kandemir, Evidential Deep Learning to Quantify Classification Uncertainty, NeurIPS 2018
- Angelopoulos & Bates, A Gentle Introduction to Conformal Prediction, FnT ML 2023
- Tan & Le, EfficientNet: Rethinking Model Scaling for CNNs, ICML 2019
If this code is useful for your work, please cite it via CITATION.cff.
This is research code. It is not a medical device. It is not validated for clinical use. Outputs must never be used to make screening or diagnostic decisions in a clinical workflow. The point of selective prediction research is precisely to acknowledge that ML models, deployed naively, fail in clinically dangerous ways.
Bug reports, suggestions, and pull requests are welcome. Please read CONTRIBUTING.md for development setup, code style, and how to add a new uncertainty method or dataset. For substantial changes, open an issue first so we can align.
MIT © Muhammad Shahnawaz Khan
| 🌐 Website | www.skakarh.com |
| ✍️ Blog | skakarh.com/blog |
| 🛠️ Services | skakarh.com/services |
| linkedin.com/in/skakarh | |
| 📦 More projects | github.com/ShahnawazKakarh |