Background: We previously reported the implementation of a machine learning (ML) model for mortality prediction that was integrated into a CDSS encouraging clinicians to have a SIC with at-risk cancer patients. The clinical utility of a ML model can change after implementation due to fluctuations in the organization’s patient population and clinical practices. It is important to establish a workflow to monitor and continually reinforce ML-powered CDSS to ensure that it continues to benefit patients. We report a workgroup structure that incorporates data driven evaluation of ML model performance and feedback from CDSS end users to optimize the acceptability of the CDSS. Methods: The workflow was piloted in the gastrointestinal (GI) oncology clinic from 11/2021-5/2022. A workgroup including members of the implementation team and end-users of the CDSS met monthly to review 1) a dashboard that displays model performance, 2) an electronic health record (EHR) report that summarizes use of the CDSS, 3) feedback from end users regarding their opinion of the CDSS and any barriers to implementation. We evaluated the accuracy of model predictions among subgroups as defined by mortality and unplanned hospital admissions or ED visit rates. Fisher’s Exact Test was used to identify differences between categorical variables. Numeric values including incidence rate ratios (IRRs) adjusted for age, sex, race, and gender with 95% confidence intervals (CIs) were calculated using Poisson regression. Results: 119 patients were evaluated by the model and 50 (42%) were assessed as high-risk. In the high-risk group, the oncology team evaluated 39 (78%) patients for appropriateness of a SIC; SIC was completed with 5 (10%) patients. During workgroup meetings, physicians shared that some of the high-risk predictions were for patients undergoing curative intent therapy. 0 out of 24 patients who received curative treatment died and 5 out of 26 patients who receive palliative treatment died. The log-rank p-value of 0.03 indicates that the survival distribution differs significantly over time between two groups. The adjusted IRR for unplanned hospital visits (palliative vs curative) was 2.55 (1.3-5.0). Adjusted mean hospital visits per month were 0.34 (0.21-0.51) vs 0.13 (0.06-0.21). Conclusions: The workgroup format is a feasible method to continuously review acceptability of a ML-powered CDSS. It may evaluate critical feedback from end users in a holistic manner that can augment a data driven evaluation of the model performance. The data implies that patients undergoing curative therapy have a decreased risk for mortality and unplanned hospital admissions or ED visits. The CDSS may be optimized by excluding these patients; however, longer follow up of this sub-population is needed to confirm that they have no additional risk factors.