Background: Caregiver support for HNC patients affects clinical outcomes including survival. Psychological distress in caregivers may impair the quality of support they provide. The impact of patient symptom burden and caregiver tasks on caregiver psychological distress is unknown. Methods: Patient symptom burden was assessed with the Vanderbilt Head and Neck Symptom Survey 2.0 (VHNSS 2.0; 10 domains, 3 single items). Caregiver task burden was assessed with the Caregiver Task Inventory (CTI; 11 domains), and quantified as task number and task difficulty/distress. Psychological distress was measured with the Profile of Mood States short form (POMS-SF). Two-step clustering analysis was used to independently generate clusters of caregiver distress, caregiver task burden, and patient symptom burden. Chi-Square and logistic regressions were used to test for associations of the resultant clusters of task burden and patient symptoms with caregiver distress. Results: 89 HNC patient-caregiver dyads were included. Patients were mostly male (77%) and Caucasian (88%). Median time since diagnosis was 3.7 months (IQR 2–7); 90% received combined modality therapy. Caregivers were mostly Caucasian (92%), female (85%) and spouses (80%). We found two caregiver clusters of psychological distress (40% mod-high, 60% low), and two clusters of caregiver task scores (40% mod-high, 60% low). Similarly, two clusters of patient symptom burden were found: 51% mod-high, 49% low. Caregivers with mod-high task scores were more likely than low to report mod-high levels of psychological distress (71% vs. 24%, p < 0.001). Patients with mod-high symptom burden were more likely than low to have caregivers with mod-high psychological distress (55% vs. 23%, p = 0.005). No effect modification of patient symptom burden on the association between caregiver task and caregiver psychological distress was seen (p > 0.05). Conclusions: Psychological distress in HNC caregivers is associated more strongly with caregiver task scores than patient symptoms. Further work to define the caregiver and task characteristics that lead to psychological distress should inform future interventions to support caregivers and patients.