Background: Androgen deprivation therapy (ADT) is a standard of care for high-risk prostate cancer, but treatment tolerance is variable. Prior work has demonstrated the correlation between body composition (BC) and clinical outcomes in prostate cancer. Specifically, high visceral fat density has been associated with fat depletion phenomenon and poor prognosis in prostate cancer. However, the interaction of long-term ADT tolerance and body fat composition is less studied. We investigated if BC could predict for outcomes and treatment tolerance in patients with high-risk prostate cancer with planned ADT. Methods: An IRB-approved retrospective review was conducted at a tertiary care center of patients with high-risk (T3a or prostate-specific antigen [PSA] > 20 ng/mL or Gleason score 8-10 or N/M+) prostate cancer who received definitive external beam radiation therapy (RT) from 2006 to 2013. A previously validated, fully automated deep learning BC analysis pipeline was performed on RT simulation scans to compute BC at the top of L3 slice, including total skeletal muscle (SM), subcutaneous fat (SF), and visceral fat (VF) surface area (cm²) and average CT density (Hounsfield Units (HU)); results were manually validated by experts. BC was stratified by median value. Adult Comorbidity Evaluation-27 (ACE) was used to measure co-morbidity. Long-term ADT was defined as > 2 years, tolerance was defined as unplanned discontinuation > 3-month difference in intended/actual duration of ADT. The association between BC markers, oncologic outcomes, and treatment tolerance was analyzed using univariable Cox regression and chi-square test. Results: A total of 207 men were analyzed with a median follow up time of 10.8 years (range 0.7-17.3y). Median age was 65 (range 42-83), with 61 (29.4%) patients classified as high-risk, 134 (64.7%) very-high-risk, and 12 (5.8%) N+/M+ at diagnosis. High VF density was associated with worse overall survival (OS) (HR 1.71, 95%CI 1.09-2.68, p = 0.0204) but not cancer-specific survival (CSS) (p = 0.08) or biochemical-relapse free survival (bRFS) (p = 0.97). SM and SF density, as well as area of SM, VF, SF, and total fat were not associated with outcomes. N/M stage was associated with bRFS (p = 0.0139), and N/M stage (p = 0.0101) and higher ACE score (p = 0.0218) were associated with OS. Among 88 (42.5%) patients planned for long-term ADT use, 24 (27%) patients discontinued ADT prior to duration, of which 15 (17%) patients discontinued due to toxicity. BC markers did not correlate with tolerance to long-term ADT (p = 0.17). Tolerance to long-term ADT was not associated with bRFS or OS. Conclusions: High VF density is associated with worse OS but not bRFS or CSS in high-risk prostate cancer patients, and not associated with adipose area or ADT tolerance. VF density may be a biomarker of underlying metabolic health in prostate cancer patients independent of disease, and a potential area of intervention.