Background: Immune checkpoint inhibitors (ICIs), which are currently used in the standard second-line treatment for non-small cell lung cancer (NSCLC), have largely improved the prognosis of advanced NSCLC. However, patients treated with ICI monotherapy have low response rates; hence, there is an urgent need to expand the population responding to immunotherapy treatment. This study aimed to explore the efficacy and safety of the combination of ICI with chemotherapy or anti-angiogenic therapy compared to ICI monotherapy in patients previously treated for advanced NSCLC. Methods: Data was collected from previously treated patients with NSCLC who further received ICI monotherapy or combination therapy. The distribution of clinical variables was assessed using the chi-square test or Fisher's exact test. The relationship between descriptive variables and survival was described using Kaplan–Meier curves and compared by log-rank test. Cox proportional-hazards models were used to calculate hazard ratios (HR) and 95% confidence intervals (CI) to confirm survival differences. Results: A total of 145 patients were included in this study, of which 63 were in the ICI monotherapy group, 57 were in the ICI with chemotherapy group, and 25 were in the ICI with anti-angiogenic therapy group. Compared with the ICI monotherapy group, the ICI with chemotherapy group had significantly higher ORR (31.6% vs. 11.1%, P = 0.006), DCR (84.2% vs. 61.9%, P= 0.006), PFS (mPFS: 6.37 vs. 3.47 months, P< 0.0001; HR = 0.42, 95% CI: 0.29–0.63, P< 0.0001), and OS (mOS: 18.60 vs. 8.47 months, P< 0.0001; HR = 0.40, 95% CI: 0.25–0.64, P= 0.0001). In addition, the ICI with anti-angiogenic therapy group also had significantly elevated DCR (88% vs. 61.9%, P= 0.02), PFS (mPFS: 8.17 vs. 3.47 months, P< 0.0001; HR = 0.30, 95% CI: 0.17–0.53, P< 0.0001), and OS (mOS: 19.20 vs. 8.47 months, P= 0.006; HR = 0.44, 95% CI: 0.24–0.80, P= 0.007) compared with the ICI monotherapy group. There was no significant difference between the ICI with chemotherapy group and the ICI with anti-angiogenic therapy group. Meanwhile, the addition of chemotherapy or anti-angiogenics did not increase immune-related adverse events and had a good safety profile. Conclusions: For previously treated patients with advanced NSCLC, treatment with ICI in combination with chemotherapy led to enhanced ORR, DCR, PFS, and OS compared to treatment with ICI monotherapy. Moreover, treatment with ICI combined with anti-angiogenic therapy led to enhanced DCR, PFS, and OS compared to treatment with ICI monotherapy. The combination of immunotherapies is a promising second-line treatment option for some of the patients in this study, and the findings of this study necessitate the need for further exploration.