Background: A lower risk of breast cancer among physically active women has been frequently reported, but the risk in women using hormone replacement therapy (HRT) appears to be higher. We quantified the association between physical activity and breast cancer, and we examined the influence that HRT use and other risk factors had on this association. Methods: After a systematic literature search, prospective studies were meta-analysed using random-effect models. As physical activity assessment and reporting of results were heterogeneous across studies, breast cancer risk associated with the highest level of physical activity was compared with the lowest level of physical activity. Dose-response analyses were also conducted with studies reporting physical activity either in hours/week or in hours of metabolic equivalent per week (MET-h/week). Results: The literature search identified 38 independent prospective studies published between 1987 and 2014 that included 4,183,888 women of which 116,304 breast cancer cases. Compared to the lowest level of physical activity, the highest level was associated with a summary relative risk (SRR) of 0.88 (95% CI (0.85, 0.90)) for all breast cancer, 0.89 (0.83, 0.95) for ER+/PR+ breast cancer and 0.80 (0.69, 0.92) for ER-/PR- breast cancer. Risk reductions were not influenced by the type of physical activity (occupational or non-occupational), the place of residence, adiposity, and menopausal status. Risk reductions increased with increasing amounts of physical activity, without threshold effect. In six studies that examined the influence of physical activity according to HRT use, the SRR was 0.78 (95% CI (0.70, 0.87)) in women who never used HRT and 0.97 (95% CI (0.88, 1.07)) in women who ever used HRT, without heterogeneity between studies. A sustained change from being physically inactive to engaging in 4 to 7 hours/week of mainly vigorous physical activity could lead to a 31% (95%CI (22, 40)) risk reduction in women who never used HRT. Conclusions: Increasing physical activity is associated with meaningful reductions in the risk of breast cancer. However in women who ever used HRT, the preventative effect of physical activity seems to be cancelled out.