Background: Machine learning (ML) algorithms can accurately identify patients with cancer at risk of short-term mortality and facilitate timely conversations about treatment and end-of-life preferences. We developed, validated, and implemented a ML algorithm to predict mortality in a general oncology setting, using electronic health record (EHR) data prior to a clinic visit.

Methods: Our cohort consisted of patients aged ≥18 years who had an encounter in outpatient oncology practices within a large academic health system between February 1st and July 1st, 2016. We randomly split the sample into training (70%) and validation (30%) cohorts at the patient-encounter level. We trained three ML algorithms to predict 180-day mortality and describe performance in the holdout validation cohort. From October 2018 to February 2019, we used the best-performing algorithm to generate weekly lists of high-risk patients at a single community oncology practice and studied the impact on rates of documented serious illness conversations (SICs).

Results: Among 62,377 encounters used to train the algorithms, 7.4% involved a patient who died within 180 days. Gradient boosting and/or random forest outperformed logistic regression in all metrics (Table), and the gradient boosting model had superior discrimination and calibration. In the gradient boosting model, observed 180-day mortality was 45.5% (95% CI 39.0-52.3%) in the high-risk group vs. 3.3% (95% CI 2.9-3.7%) in the low-risk group. In a survey of oncology clinicians, 59% of patients flagged as high-risk were appropriate for a serious illness conversation in the upcoming week (response rate 52%). Five months after implementing the intervention, average monthly documented SICs increased by 23% (31.7 to 39).

Conclusions: A ML algorithm based on EHR data accurately identified patients with cancer at risk of short-term mortality, was concordant with oncologists’ assessments, and was associated with more SICs.

Table

	Area under the curve	Negative predictive value	Positive predictive value	Sensitivity	Specificity
Logistic regression	0.86	0.96	0.44	0.22	0.99
Random forest	0.86	0.96	0.48	0.19	0.99
Gradient boosting	0.87	0.97	0.44	0.27	0.98