Background: Improving body image and self-esteem are top ASCO priorities in the survivorship care of men with prostate cancer (PCa). Body image and global self-esteem, influenced by physical self-worth, are negatively affected by PCa treatment. We investigate whether exercise can improve physical self-worth in men treated for PCa and if improving self-worth is associated with changes in quality of life (QoL) and mental health. Methods: We performed a secondary analysis of a phase II randomized controlled trial comparing the effects of Exercise for Cancer Patients(EXCAP), a structured, 6-week, home-based exercise intervention, to usual care (UC) in men with non-metastatic PCa receiving radiation or Androgen Deprivation Therapy (ADT). The Physical Self-Perception Profile (PSPP), a valid 30-item questionnaire where higher scores indicate greater physical self-worth, was assessed at pre- and post-intervention. Changes between arms were compared using ANCOVA. Spearman correlations were calculated for pre/post-intervention change scores for PSPP and QoL, depression, and anxiety as measured by the Functional Assessment of Cancer Therapy (FACT), Center of Epidemiologic Studies Depression (CES-D) scale, and State-Trait Anxiety Inventory (STAI), respectively. Results: Fifty-eight men were randomized; average age was 67.1 years. Physical self-worth at baseline moderated the effect of the intervention. Compared to UC, EXCAP improved physical self-worth in those with baseline PSPP scores above the median (p < 0.04). Exercisers with baseline PSPP scores in the top quartile demonstrated a more significant improvement over UC (p < 0.01). Improvements in physical self-worth were associated with improved QoL (r = 0.29, p = 0.04), depression (r = -0.28, p = 0.04) and anxiety (r = -0.30, p = 0.03). Conclusions: Exercise significantly improves physical self-worth in men with PCa on radiation or ADT, and greater physical self-worth is associated with improved QoL, depression and anxiety. Those with higher baseline physical self-worth derived the most benefit from exercise. Exercise should be prescribed to boost self-esteem and body image in men receiving radiation or ADT for PCa. Clinical trial information: NCT00815672