Background: Little is known about risk and protective factors for dementia in cancer survivors. With increasing longevity due to improved cancer therapy, it’s prudent to study factors contributing to cognitive impairment in this patient population. Methods: We performed multivariable cox regression analyses to assess established risk factors for all-cause dementia (age, education, APOE4-status, social and physical activity engagement) in breast and prostate cancer survivors. Patients 50+ years old with pathology-confirmed cancer diagnosis at Mayo Clinic were identified from a longitudinal, population-based cohort – Mayo Clinic Study of Aging. Analyses were controlled for age, APOE4-status, education, and cancer treatment. Results: We included a total of 209 patients with breast cancer (mean age 75.4 years, 24% APOE4-positive, median follow-up 6.8 years) and 352 with prostate cancer (mean age 77 years, 26% APOE4-positive, median follow-up 6.4 years). 31 patients with breast cancer (15%) and 66 patients with prostate cancer (20%) were diagnosed with dementia at last follow-up. Older age was associated with increased risk for dementia (5-year increase – breast: hazard ratio (HR) 1.9 [95%CI 1.3-2.7, p = 0.001]; prostate: HR 2.4 [95%CI 1.8-3.2, p < 0.0001]). APOE4-positivity was associated with increased dementia risk in breast (HR 3.6 [95%CI 1.6-7.8, p < 0.002]), but not prostate cancer (HR 1.2 [95%CI 0.7-2.0, p = 0.5]). More education was associated with lower dementia risk in prostate cancer (education year: HR 0.91 [95%CI 0.84-0.98, p < 0.02]; education years > 12: HR 0.26 [95%CI 0.1-0.6, p < 0.001]). Physical activity engagement in mid-life was associated with lower dementia risk (prostate: HR 0.56 [95%CI 0.4-0.8, p < 0.005]; breast: HR 0.48 [95%CI 0.3-0.9, p < 0.02]). Conclusions: In this longitudinal, population-based study we found that some risk and protective factors for dementia in cancer survivors vary by cancer type. While advanced age is an expected dementia risk factor, there were mixed results for APOE4-positivity (increased risk only in breast cancer) and education (only protective in prostate cancer). Our results support further study into factors influencing dementia risk by cancer type in hopes of identifying preventative strategies.