Background: Psychosocial risk factors such as anxiety and depression can affect immune responses and cancer-related outcomes. Accordingly, we investigated whether an existing anxiety or depression diagnosis influenced immune checkpoint inhibitor (ICI) safety or efficacy. Methods: In our multicenter retrospective study across all cancers, we identified 913 patients who began anti-PD-1 or anti-PD-L1 monotherapy [PD(L)-1] between 2011-2018 with follow up until January 2021. Depression and anxiety disorder diagnosis at the time of treatment initiation were abstracted from medical charts. Immune-related adverse events (irAEs) were graded according to CTCAE v4.03. Overall survival (OS) and time to treatment failure (TTF) were recorded. Multivariable logistic regression models and proportional hazard models were used to test relationships between anxiety (or depression) diagnosis and irAEs (any grade vs. none), OS, and TTF, respectively, while adjusting for race, gender, age, and other relevant clinical variables associated with each outcome (p < 0.1) in the univariate analysis (e.g., performance status, autoimmune disease, and number of metastatic sites). Results: At the time of treatment initiation, 11% (n = 102), 12% (n = 106), and 4% (n = 41) of patients in the overall PD(L)-1 ICI cohort had an anxiety diagnosis, depression diagnosis, or both, respectively. The overall incidence of any grade irAEs was 32% (n = 295), including 44% (n = 45) and 37% (n = 39) among those with anxiety and depression history, respectively. Patients with an anxiety diagnosis had a greater likelihood of experiencing an irAE [Odds ratio (OR) = 1.80; 95% CI = 1.16 to 2.80, p = 0.009]. There was not a statistically-significant relationship between depression diagnosis and irAEs (univariable OR = 1.25; 95% CI = 0.82 to 1.91, p = 0.295). In the subset of patients with lung cancer (n = 443) receiving PD(L)-1 ICI, the rates of any grade irAEs, anxiety, and depression diagnosis were 32% (n = 143), 10% (n = 44), and 12% (n = 52), respectively. In multivariable analysis, anxiety diagnosis was associated with better OS (HR = 0.61; 95% CI = 0.39 to 0.97; p = 0.038) and TTF (HR = 0.68; 95% CI = 0.47 to 0.98; p = 0.038). The link between anxiety diagnosis and irAEs was weaker in this subsample (OR = 1.46; 95% CI = 0.74 to 2.87, p = 0.274). No significant associations between depression diagnosis and any grade irAEs (OR = 0.83; 95% CI = 0.44 to 1.57, p = 0.573), OS (HR = 1.06; 95% CI = 0.73 to 1.55, p = 0.755), or TTF (HR = 1.13; 95% CI = 0.83 to 1.54, p = 0.433) were observed. Conclusions: Our results suggest that baseline psychosocial risk factors, especially anxiety disorders, may influence ICI efficacy and safety. Prospective studies are warranted to better understand the relationship between psychosocial risk factors and ICI outcomes. The study also suggests a possible role for targeted supportive care in influencing cancer-related outcomes for those with psychosocial risk factors.