Roche Diagnostics, Indianapolis, IN
David Morgenstern , Eran Netanel Choman
Background: Application of ML-based risk prediction models to lung cancer screening cohorts have been shown to increase screening efficiency. However, ML-based models may be vulnerable to sexual and racial bias arising from historical bias in health care access as well as biased training data. Demonstrating fairness in the predictions of ML-based models is a prerequisite to their acceptance by clinicians and patients. We assessed the clinical performance of LungFlag based on sex and race, two key demographic subgroups at risk for disparate outcomes due to bias. Methods: The LungFlag machine learning model is a retrospectively validated ML model intended to identify individuals who are at elevated risk for lung cancer and should be counseled regarding lung cancer screening. LungFlag uses existing routine outpatient lab measurements, smoking history, comorbidities, and demographic data to flag high-risk individuals. We compare performance of LungFlag between sexes and between races and calculate the sensitivity at the overall positivity rate of 3%. We chose a case-control design based on a large US-based community and outpatient dataset including 39,135 case patients with NSCLC and 212,454 contemporaneous NSCLC-free controls. We included ever-smokers, ages 45-80, with available lab measurements from 3-12 months before diagnosis, and minimal follow-up of 24 months. Sub-populations with less than 1% representation from the total population were excluded. Results: The comparison between the sub-populations presented by sensitivity and specificity indexes is detailed in the table. No statistically significant difference was demonstrated in the sensitivity of the model to flag individuals that were diagnosed with NSCLC on multiple sub-populations. Conclusions: The LungFlag model demonstrated fairness with respect to sex and race based on similar clinical sensitivity in a large, community-based retrospective dataset. Further assessment in prospective studies and in additional racial sub-populations is recommended to support this conclusion.
Demographic Parameter | Sub-Population | N Cases | N Controls | Specificity @ 3% Positivity Rate (95% CI) | Sensitivity @ 3% Positivity Rate (95% CI) | Odds Ratio @ 3% Positivity Rate (95% CI) |
---|---|---|---|---|---|---|
Sex | Female | 19,555 | 112,720 | 97.2 [97.1 - 97.3] | 23.1 [22.6 - 23.7] | 10.4 [9.8 – 11.2] |
Male | 19,580 | 99,734 | 96.8 [96.7 - 96.9] | 23.4 [22.8 - 24.0] | 9.2 [8.7 – 9.9] | |
Race | White | 33,724 | 179,387 | 96.9 [96.8 - 97.0] | 23.5 [23.1 - 24.0] | 9.6 [9.1 – 10.2] |
Black | 4,058 | 19,554 | 97.1 [96.8 - 97.3] | 23.7 [22.5 - 25.0] | 10.4 [8.8 – 12.0] |
Disclaimer
This material on this page is ©2024 American Society of Clinical Oncology, all rights reserved. Licensing available upon request. For more information, please contact licensing@asco.org
Abstract Disclosures
2023 ASCO Annual Meeting
First Author: Yosuke Kagawa
2024 ASCO Annual Meeting
First Author: Michael K. Gould
2022 ASCO Quality Care Symposium
First Author: Santiago Olaechea
2024 ASCO Quality Care Symposium
First Author: Tamara Hamlish