Background: Alternative payment models for oncology seek to improve quality and reduce spending. Yet the ability to measure high-quality care across oncology practices remains uncertain. We characterized quality of care for oncology practices using registry and claims-based measures of processes, utilization, end-of-life care, and survival and assessed correlations of practice-level performance across measure type and cancers. Methods: Using SEER-Medicare data, we studied individuals with newly diagnosed lung (N = 95,635), breast (N = 78,736), or colorectal (CRC, N = 51,385) cancers in 2010-2015 treated in oncology practices with ≥20 patients (502, 492, and 347 practices, respectively). We measured receipt of guideline-recommended treatment and surveillance (processes), hospitalizations or emergency department visits during 6-month chemotherapy episodes (utilization), care intensity in the last month of life (EOL), and 12-month survival (lung and CRC only). We calculated summary process, utilization, and EOL measures for each patient (number of measures met divided by the number for which the patient was eligible). We used hierarchical linear models with practice-level random effects to estimate summary measures and survival for each practice. We calculated practice-level reliability (a measurement’s reproducibility) for each measure based on the between-measure variance, within-measure variance, and sample size. Results: Few practices had ≥20 patients eligible for most measures (38%, 37%, and 31% of practices had ≥20 patients for any lung, breast, and CRC measures, respectively). Measure reliability was low. Only 13%, 7%, and 20% of measures for lung, breast, and CRC, respectively, had a median reliability across practices ≥0.7. Among practices with ≥20 patients with summary measures of each type within cancer, correlations across measure types were low (all correlation coefficients (r)≤0.21 except a weak correlation of the CRC process summary measure with 1-year CRC survival, r = 0.38, p < 0.001). Summary process measures were minimally or not correlated across cancer type (lung, breast, CRC; all correlation coefficients ≤0.16). Conclusions: Claims-based measures of care processes, utilization, EOL care, and survival are limited by small numbers of fee-for-service Medicare patients across practices, even after pooling 6 years of data. Measures have poor reliability and are poorly correlated across measure or cancer type. Additional research is needed to identify reliable quality measures for practice-level alternate payment models.