Background: The benefit of adjuvant chemotherapy with oxaliplatin for elderly patients with colorectal cancer (CRC) remains controversial. Our study could not demonstrate the benefit of adding oxaliplatin for elderly CRC patients aged over 70 years as with the previous clinical trials. Here, we evaluated the prognosis after recurrence in the elderly patients who received adjuvant chemotherapy. Methods: This retrospective study included patients aged over 70 years who were diagnosed with high-risk stage II and stage III CRC and received adjuvant chemotherapy in our two hospitals (The Jikei University Hospital and Katsushika Medical Center) between January 2010 and December 2019. The patients were divided into two groups; patients who received fluoropyrimidine monotherapy were included in Fp group and those who received fluoropyrimidine plus oxaliplatin were included in Fp+OX group. Moreover, we evaluated patient characteristics, treatment, and survival in the patients with recurrence and compared them between two groups. Results: A total of 127 patients received adjuvant chemotherapy; 75 patients in Fp group and 52 patients in Fp+OX group. With a median follow-up time of 64.5 months, the 5-years disease-free survival and 5-years overall survival in Fp group/Fp+OX group were 70.6/67.1% (hazard ratio [HR] 1.10, 95% confidence interval [CI] 0.61–1.98) and 89.0/71.8% (HR 1.48, 95% CI 0.76–2.88), respectively. The benefit of adding oxaliplatin was not observed. Among them, 15 patients in Fp group and 14 patients in Fp+OX group were relapsed; median age was 78 (range: 72-83) and 78 (range: 72-86) years, male was 67% and 64%, PS of 0/1 was 67/33% and 86/14%, primary tumor site of right/left was 47/53% and 57/43%, and RAS mutation was 20% and 29%, respectively. Median relapse-free survival was 17.7 and 14.3 months (p = 0.453). There were no significant differences in treatment after recurrence; aggressive treatment (surgery/chemotherapy) was 73% (33/40%) and 86% (21/65%), best supportive care was 7% each, and unknown was 20% and 7%, respectively. However, the median overall survival from the date of relapse was significantly worse in Fp+OX group than Fp group (45.0 and 14.4 months, p = 0.011). Conclusions: Recurrence after receiving adjuvant oxaliplatin combination therapy was considered as one of the poor prognostic factors. It might attenuate the benefit of adding oxaliplatin in adjuvant setting in elderly patients.