Project 04 / 10 · Forecast & ML
Hospital Cost Prediction
End-to-end ML pipeline predicting patient costs on 2.5M real NY state records — SHAP-explained, uncertainty-aware, built for clinical finance decision-support.
01 / Context
The problem
The system estimates inpatient cost from demographic and clinical variables while exposing uncertainty and the drivers behind each prediction, making the output useful for finance decisions rather than just reporting a point estimate.
02 / System workflow
From input to output
- 01SPARCS records
- 02Leak-safe preprocessing
- 03Model benchmark
- 04SHAP + intervals
- 05API + dashboard
03 / Architecture
Inside the system
A scikit-learn ColumnTransformer handles imputation, scaling, and encoding before Ridge, Random Forest, XGBoost, LightGBM, and a PyTorch MLP are benchmarked.
Optuna tunes candidate models and MLflow records parameters, metrics, and artifacts; the selected model is served by FastAPI.
A six-tab Streamlit interface supports interactive prediction and analysis, with Docker Compose coordinating the API, dashboard, and MLflow services.
04 / Engineering decisions
What shaped the build
Decision 01
Benchmark before selection
Five model families are compared against a linear baseline rather than presenting one model in isolation.
Decision 02
Explain uncertainty
SHAP explains feature influence and quantile regression returns calibrated 80% prediction intervals.
Decision 03
Separate model generations
The repository reports v1 full-data and v2 sampled-pipeline metrics separately because filtering changes the RMSE scale.
05 / Evidence
Measured, not implied
0.876
LightGBM R²
Best v2 result on a 150K-row spaced sample with 18 features.
33%
RMSE reduction
LightGBM improvement over the Ridge baseline.
2.5M
Records analyzed
Real New York SPARCS discharge records used for cost-driver analysis.
06 / Delivery
What shipped
- 01FastAPI provides single and batch prediction, health, metrics, and model-list endpoints.
- 02The response includes predicted cost, selected model, feature list, and confidence interval bounds.
- 03Docker Compose supports one-command deployment of inference, dashboard, and experiment tracking.
Constraint documented
Errors rise sharply for stays above $50K, and 74% of these rare high-cost cases are under-predicted. The system therefore returns wider uncertainty for high-acuity cases.