Background: We evaluated whether the intermediate-risk factors of percentage of positive biopsies (PPB), clinical tumor category, and prostate-specific antigen (PSA) level, in addition to age, were associated with the risk of prostate cancer-specific mortality (PCSM) among men with Gleason 3+4 prostate cancer treated with brachytherapy (BT) alone or BT and a short course of androgen deprivation therapy (ADT). Methods: We conducted a prospective cohort study of 1920 consecutively treated men with Gleason 3+4 adenocarcinoma of the prostate who received BT or BT and a median of 4 months of ADT between 10/14/1997 and 5/28/2013. Separate multivariable Fine and Gray competing risks regression models among men treated with BT or BT and ADT were used to assess whether PPB, cT2b-T2c, and PSA of 10.1-20.0 ng/ml, in addition to age greater than the median of 70 years, were associated with the risk of PCSM after adjustment for comorbidity. Results: After a median follow-up of 7.8 years (interquartile range 5.2-10.4 years), 284 men (14.8%) had died, including 31 (10.9% of deaths) from PC of which 18 (58.1%) and 13 (41.9%) occurred in men treated with BT or BT and ADT, respectively. For men treated with BT alone, increasing PPB, PSA of 10.1-20.0 vs 4.0-10.0 ng/mL, and age >70 vs ≤70 years were significantly associated with an increased risk of PCSM (adjusted hazard ratio [AHR] 1.015 95% confidence interval [CI] 1.000-1.031, P=0.048; AHR 5.55, 95% CI 2.01-15.29, P<0.001; and AHR 3.66, 95% CI 1.16-11.56, P=0.03, respectively). The respective results for men treated with BT and ADT were AHR 1.009, 95% CI 0.987-1.031, P=0.44; AHR 4.17, 95% CI 1.29-13.50, P=0.02; and AHR 3.74, 95% CI 0.87-16.05, P=0.08. The clinical tumor category was not significantly associated with the risk of PCSM. Conclusions: Among men with biopsy Gleason score 3+4 PC, both age >70 years and PSA of 10.1-20.0 ng/ml were significantly associated with an increased risk of PCSM following BT, and adding 4 months of ADT may not be sufficient to mitigate this risk. Advanced imaging and targeted biopsy of suspicious areas should be considered to personalize treatment in order to minimize the risk of PCSM in these men.