Background: Gastric cancer (GC) is the fifth most incident and the fourth deadliest cancer worldwide, and currently immune checkpoint inhibitors have been approved for the treatment of patients with recurrent/metastatic GC. Studies have reported that the inactivation of chromatin remodeling complex may be a promising biomarker for immune checkpoint inhibitors (ICI) treatment. The aim of this study was to investigate the predictive value of chromatin remodeling genes on the response of ICIs in GC. Methods: We analyzed next-generation sequencing (NGS) data from three cohorts: the Chinese cohort (from GenetronHealth, n = 1013), the TCGA cohort (n = 434) and the ICI cohort (from the Memorial Sloan Kettering Cancer Center, n = 39). The abundance of tumor immune infiltrates was estimated by the tumor immune estimation resource (TIMER) 2.0 algorithm using gene expression data, and the Kaplan–Meier curves were used to determine survival differences. Results: In this study, we defined a set of chromatin remodeling genes containing PBRM1, ARID1A, ARID1B and ARID2. The loss-of-function mutations of these chromatin remodeling genes (CRG^LOF), including frameshift, nonsense and splice were identified in 176/1013 (17.4%) patients with GC in the Chinese cohort, which was significantly lower than those in the TCGA cohort (17.4% vs 27.6%, p = 0.00001). In both cohorts, patients with CRG^LOF mutations exhibited higher tumor mutational burden (TMB) (all p < 0.001) and PD-L1 expression (p < 0.001 in the TCGA; p = 0.037 in the Chinese) than those without CRG^LOF mutations. Next, we further analyzed the differences in tumor infiltrating lymphocytes (TILs) between CRG^LOF and non-CRG^LOF patients in the TCGA cohort, and the results showed that TILs level of CD8+ T cells (p < 0.001), Dendritic cells (p = 0.0012) and Neutrophil cells (p < 0.001) were all significantly higher in CRG^LOF samples than non-CRG^LOF ones. In addition, survival analysis found that CRG^LOF mutation conferred no significant effect on patient outcome in the TCGA cohort, while in the ICI cohort, GC patients with CFG^LOF mutations had significantly better prognosis after immunotherapy than patients with non-CRG^LOF mutations(p = 0.042). Conclusions: Loss of chromatin remodeling subunits may impact response to immune checkpoint therapy and may predict patient outcomes in GC. Further study in larger cohorts of immunotherapy-treated patients and functional characterization of the CRG^LOF mutations in the context of the tumor-immune microenvironment were needed.