Background: Sexual dysfunction is common among breast cancer survivors, but poorly characterized in sexually inactive women. We aim to better understand sexual desire in sexually inactive breast cancer survivors. Methods: We performed a cross-sectional survey of breast cancer survivors between 2014 and 2016. Women with a history of breast cancer were approached at a surveillance visit, offered enrollment, and signed consent. They completed the validated 19-question Female Sexual Function Index (FSFI) survey. The FSFI defines sexual inactivity as no sexual activity in the four weeks prior to survey completion. Fisher’s exact test was used for categorical data analysis. Firth’s bias-reduced logistic regression adjusted for age and menopausal status. Results: Of 585 respondents, 305 (53.1%) were age 40 to 59 and 241 (42.0%) were age 60 to 79 when surveyed. Four hundred and twenty-seven (73.0%) were post-menopausal and 285 (48.7%) were sexually inactive. Sexual inactivity was positively correlated with increasing age (p < 0.001) and not being in a current relationship (p < 0.0001). Four hundred and six (69.4%) respondents underwent lumpectomy and 179 (30.6%) underwent mastectomy with most having reconstruction, n = 129 (72.1%). Neither type of surgery or receipt of chemotherapy, endocrine therapy, or radiation were significantly different for sexually inactive and active respondents after controlling for age and menopausal status. 23.6% of sexually inactive respondents reported sexual desire half the time or more. 32.4% of sexually inactive respondents reported feeling sexual desire “a few times (less than half the time)” (versus 28.8% of sexually active respondents, p = 0.005) and 44% reported feeling sexual desire “almost never or never” (versus 8.0% of sexually active respondents, p < 0.001). 31.7% of sexually inactive respondents rated their degree of sexual desire as “low” (versus 25.0% of sexually active respondents, p = 0.04) and 40.9% rated it as “very low or none at all” (versus 7.0% of sexually active respondents, p < 0.001). 27.5% of sexually inactive respondents experienced moderately high sexual desire. Sexually inactive respondents were more likely to report being dissatisfied with their overall sexual life with 23.5% choosing the rating “moderately dissatisfied” (versus 8.8% of sexually active respondents, p < 0.001) and 28.1% choosing “very dissatisfied” (versus 5.8% of sexually active respondents, p < 0.001). Conclusions: Sexual inactivity is associated with lower rates of self-reported sexual desire and lower satisfaction with overall sexual life. However, nearly a quarter of sexually inactive respondents still reported significant sexual desire. This suggests patients classified as sexually inactive may not be content or indifferent to this lack of sexual activity or desire. Sexual inactivity in breast cancer survivors warrants intervention studies to improve sexual satisfaction.