Background: The standard treatment for unresectable recurrent rectal cancer (URRC) is chemotherapy (CTx) regardless of local recurrence (LR). Approximately 80% of patients with LR experience LR-related pain, making these patients difficult to treat, especially the timing of radiotherapy (RT). However, the impact of LR and the appropriate timing of palliative RT on prognosis in patients with URRC remains unclear. Methods: This retrospective study included patients with URRC who had an ECOG Performance Status (PS) of 0–2 and received fluoropyrimidines plus oxaliplatin and/or irinotecan as first-line CTx between September 2006 and December 2020 at a single institution. Patients were divided into two groups: patients with LR (LR+) and patients without LR (LR−). Progression-free survival (PFS), overall survival (OS), and post-progression survival (PPS) after first-line CTx were compared between the two groups. The association between receiving palliative RT for LR-related symptoms until disease progression after first-line CTx and the proportion of patients receiving subsequent second-line CTx was also evaluated in the LR+ group. Multivariate Cox analysis, including variables with p < 0.10 in univariate analyses, was performed for PPS. Results: The LR+ and LR− groups included 49 and 99 patients, respectively. No significant differences in the baseline characteristics were detected between the two groups, except liver metastasis, which occurred less frequently in the LR+ group. OS and PPS were shorter in the LR+ group compared with the LR- group (median OS, 33.4 vs. 48.2 months; hazard ratio [HR], 1.52; p = 0.05; median PPS, 17.2 vs. 23.5 months; HR, 1.77; p = 0.01). PFS was similar between the two groups (median PFS, 13.0 vs. 15.8 months; HR, 1.18; p = 0.39). Fewer patients in the LR+ group received second-line CTx compared with the LR- group (68 vs. 83%, p = 0.06). The proportions of patients with PS ≥ 1 (78 vs. 52%, p = 0.007) and PS deterioration at progression after first-line CTx (59 vs. 27 %, p = 0.002) were higher in the LR+ group compared with the LR group-. The most common cause of PS deterioration in the LR+ group was LR-related pain (77%). Among patients with LR-related pain, the proportion of patients receiving subsequent second-line CTx was numerically lower in patients who did not receive palliative RT than in patients who did receive palliative RT until disease progression after first-line CTx (75 vs. 54%, p = 0.23). Multivariate analysis revealed that receiving subsequent second-line CTx was an independent prognostic factor for PPS in LR+ patients (adjusted HR, 4.77; p = 0.006). Conclusions: The presence of LR is associated with poor prognosis in URRC, which may be due to worsening PS caused by LR-related symptoms and the lower proportion of subsequent CTx. Thus, palliative RT should be recommended more actively when LR becomes symptomatic.