Background: We present a large retrospective analysis comparing rectal toxicity following high dose rate (HDR) brachytherapy as monotherapy relative to dose-escalated external beam radiotherapy (EBRT) for patients with localized prostate cancer. Methods: 2683 patients treated with HDR or EBRT between 1994 and 2017 were included. 545 (20.3%) received HDR and 2138 (79.7%) EBRT. HDR fractionation was 38 Gy/4 fractions (n=321), 24 Gy/2 (n=96), or 27 Gy/2 (n=128). EBRT patients received a median dose of 75.6 Gy in 1.8 Gy fractions [range 70.2-82.8 Gy], using either 3D conformal or intensity modulated radiotherapy (IMRT). All EBRT patients underwent 3D image guidance via an off-line adaptive process. Treatment was directed to prostate only (n=780) or prostate and seminal vesicles (n=1351). No nodal therapy was given. Target volume for HDR patients included the prostate with no expansion. Acute and chronic gastrointestinal (GI) toxicity was defined as occurring ≤ 6 and > 6 months, respectively, after radiotherapy and was graded per CTCAE version 3.0. Toxicity variables were analyzed with χ² test. Results: Median follow-up was 7.5 years (7.4 years for EBRT and 7.9 years for HDR). 69.1% of EBRT patients received IMRT with the remainder treated using 3D conformal technique. Compared to EBRT, HDR was associated with decreased rates of acute grade ≥ 2 diarrhea (0.7% vs. 4.5%, p < 0.001), rectal pain/tenesmus (0.6% vs. 7.9%, p < 0.001), and rectal bleeding (0% vs. 1.6%, p=0.001). Rates of chronic grade ≥ 2 rectal bleeding (1.3% vs. 8.7%, p < 0.001) and radiation proctitis (0.9% vs. 3.3%, p=0.001) favored HDR over EBRT. Rates of any chronic rectal toxicity grade ≥ 2 were 2.4% vs. 10.5% (p < 0.001) for HDR vs. EBRT, respectively. For the 1478 EBRT patients treated with IMRT, acute and chronic rates of any grade ≥ 2GI toxicity were 4.2% and 5.6%, respectively, compared to 1.5% (p=0.002) and 2.4% (p=0.002), respectively, for HDR patients. Conclusions: In appropriately selected patients with localized prostate cancer undergoing definitive radiation therapy, HDR brachytherapy as monotherapy is an effective strategy for reducing acute and chronic rectal toxicity.