Background: Clinical trials suggest that adjuvant radiotherapy (RT) may be omitted in women aged ≥65 with early-stage, hormone-receptor positive breast cancer provided completion of 5 years of endocrine therapy (ET). However, at the time of RT consult, it is often unknown whether the patient will start ET, or will start ET but not complete 5 years, either of which, if known in advance, would alter RT recommendations. We studied a cohort of patients who would have been eligible for these trials to examine which factors were related to declining or discontinuing ET. Methods: Using a prospectively maintained institutional database, we identified patients age ≥65 who underwent surgery between 2010 and 2017 with stage I breast cancer, hormone receptor positive. Missing data were replaced using multiple imputation. Recurrence statistics were calculated using Kaplan-Meier analysis. Multivariate (MVA) logistic regression was used to assess which factors were associated with not starting or discontinuing ET. Results: We identified 590 patients who met eligibility criteria. 453 (76.8%) patients started ET, of whom 315 (69.5%) completed at least 5 years, 84 (18.5%) stopped ET before 5 years, and 54 patients (11.9%) were lost to follow up. Median follow up was 67.2 months. Among the 137 patients who did not start ET, the most common reasons were osteopenia or osteoporosis (27.7%), arthritis (18.2%), and declined consultation with medical oncology (15.3%). On MVA, not starting ET was significantly associated with older age (OR 1.10, p < 0.0001), osteopenia (OR 2.27, p = 0.005) or osteoporosis (OR 3.82, p = 0.009), or not undergoing radiation (OR 1.92, p = 0.006) or chemotherapy (OR 6.45, p = 0.02). The most common reason for stopping ET was joint pain (50.0%), with other reasons including hot flashes (10.7%) and fatigue (7.1%). Not completing 5 years of ET was significantly associated with older age (OR 1.09, p = 0.002) and prior/current smoking history (OR 1.75, p = 0.037), while patients whose marital status was single or never married were significantly more likely to complete 5 years of ET (OR 0.26, p = 0.019). Recurrence-free survival was lower among the group which never started ET (p < 0.0001) or which stopped ET before 5 years (p < 0.0001) as compared to those who completed 5 years, with 5-year recurrence-free survival of 97.5% (95% CI: 95.6-99.3) for patients who completed 5 years of ET, compared to 88.7% (95% CI: 82.5-95.4) for those who did not undergo ET and 87.6% (95% CI: 80.4-95.6) for those who started ET but did not complete 5 years. Conclusions: Patients who are older, have osteopenia/osteoporosis, or who have declined prior treatment such as chemotherapy or RT are more likely not to start ET, while patients who are older or prior/current smokers are more likely to discontinue ET after starting and single / never married patients are more likely to complete ET. These factors may be used to guide discussion of omission of adjuvant radiotherapy.