Background: Recent efforts exploring the utility of quantitative imaging (radiomic) biomarkers to predict response to immune checkpoint inhibitors (ICI) have shown promise to provide a more accurate and scalable method than biopsy-based PD-L1 IHC and tumor mutation burden. One of the challenges of evaluating ICI biomarkers on retrospective real-world data is the inherent association between a physician’s treatment choice and PD-L1 expression status. This work investigates radiomics-based multi-modal biomarkers within patient cohorts receiving first-line therapy (ICI monotherapy, ICI plus chemotherapy, and chemotherapy), and in cohorts of all patients receiving ICI (all-lines) subdivided by PD-L1 expression. Methods: Using a large multi-institutional real-world dataset, we analyzed radiomic characteristics of 6,295 primary and metastatic lesions from 1,206 stage IV NSCLC patients treated with PD-(L)1 ICIs from nine institutions across the US and Europe. Patients with unavailable imaging follow-up or with EGFR/ALK oncogenic driver mutations were excluded from analysis, resulting in a total dataset of 791 subjects randomly assigned to training (N = 541) and validation sets (N = 250). Radiomic data was extracted from baseline CT scans capturing tumor heterogeneity, spicularity, and burden in the lung, lymph nodes, and liver. A multi-modal ensemble classifier combining demographic features, PD-L1 TPS, and radiomic data was developed to predict response to ICI therapy per RECIST 1.1 criteria. The model's performance was evaluated in terms of the area under the receiver operating characteristic curve (ROC-AUC) and compared to PD-L1 IHC using the two-tailed DeLong test. Results: In first-line cohorts, the model identified responders with an ROC-AUC of 0.83 (0.72-0.94, P = 0.005; N = 80, 81% PD-L1 > = 50%) in ICI monotherapy, 0.65 (0.47-0.83, P = 0.28; N = 78, 71% PD-L1 < 50%) in ICI plus chemotherapy, and 0.38 (0.002-0.76, P = 0.53; N = 37) in chemotherapy. In all-lines cohorts subdivided into PD-L1 high/low/negative expression categories, the model identified responders with an ROC-AUC of 0.72 (0.57-0.87, P = 0.005, N = 102), 0.74 (0.58-0.89, P = 0.003, N = 71), and 0.60 (0.46-0.74, P = 0.16, N = 77), respectively. Conclusions: Our study demonstrates that multi-modal models can predict ICI response with high performance. The stronger performance in patients receiving ICI therapy alone indicates the model’s predictive, rather than prognostic, power. Furthermore, the model demonstrated good performance in identifying ICI responders within patient sub-cohorts defined by PD-L1 expression status. These insights may be used to inform clinical decision-making, such as escalation or de-escalation of concurrent chemotherapy in stage IV NSCLC patients. In our future work, we will investigate these questions in larger cohorts and prospective studies.