Background: Microsatellite stable (MSS) metastatic colorectal cancer (mCRC) has been associated with immunotherapy resistance. However, the impact of metastatic diseases sites on clinical outcome with checkpoint inhibition has not be adequately investigated. Methods: Following IRB approval (14361), we performed a retrospective study to assess the response and progression-free survival (PFS) to PD-1 or PD-L1 based therapy in patients with MSS mCRC who progressed following standard chemotherapy and targeted therapy. Patients were included if they received an FDA approved anti-PD1 or anti-PD-L1 agent, alone or in combination with other investigational agents. Exclusion criteria included concurrent cytotoxic therapy. Response and PFS were determined by RECIST guidelines. Results: 97 patients with refractory MSS mCRC satisfied the inclusion criteria were evaluable for analysis. 57, 17, 13, and 10 received nivolumab, atezolizumab, pembrolizumab, and durvalumab, respectively. 47, 8, 10, 10, 17 received concurrent VEGFR, MEK, CTLA-4, radiation, or other targeted therapies. Among tested variables, including age, gender, primary tumor location, ECOG status, RAS, BRAF, APC, TP53, TMB, and sites of metastasis, only ECOG status (0 vs. 1) and metastatic disease to the liver were associated with disease control (partial response and stable disease; p = 0.005 and < 0.001 respectively). Disease control rates in patients without liver metastases (n = 43) was 56% (95% CI: 40-71%), compared with 2% (95% CI: 0.1-10%) in patients with liver metastases (n = 54) (p < 0.001). Also, liver involvement was predictive for PFS based on multivariate Cox regression model (p < 0.001). The median PFS in patients with liver involvement was 1.5 vs 4.5 months for no liver involvement (HR = 4.41, p = < 0.001). Amongst patients without liver metastases, 35% (15/43) remained without progression at 6 months. These patients were characterized by lung metastases (n = 8) and/or lymph node metastases (n = 4). Conclusions: The presence of liver metastases is MSS mCRC is predictive for lack of benefit from PD-1/PD-L1 inhibitors. Patients without liver metastases can derive a meaningful clinical benefit, with 35% being progression-free at 6 months. Future PD-1/PD-L1 drug development should consider metastatic sites of disease in study design and development.