Background: Immune cell (IC) infiltrates in primary tumors (PT) have been identified as prognostic markers in head and neck squamous cell carcinoma (HNSCC). IC densities may differ among PT, lymph node metastasis (LNM) and recurrent disease (RD) and by primary disease site (oral cavity- OC, oropharynx- OP, hypopharynx- HP, larynx- L). Here, we compare CD3 and CD20 IC densities in PT, LNM and RD in paired samples from different disease sites and determine the prognostic impact of IC infiltrates. Methods: Tissue microarrays with 425 PT, 198 LNM and 46 RD samples--each in triplicate--were stained immunohistochemically for CD3 and CD20 in the same slide. Immune cell densities per mm² were determined using a digital image analysis software (QuPath). Individual means were calculated from triplicates of each sample. IC infiltrates from different sample types (PT, LNM, RD) and primary tumor sites were compared using Kruskal-Wallis and Mann-Whitney-U tests. Paired samples were compared using Wilcoxon signed rank test. IC densities were classified as CD3 high/low and CD20 high/low for each primary tumor site using the individual median as a cut-off. Overall survival (OS) was calculated using the Kaplan-Meier method. P-values for each hypothesis were corrected using a false discovery rate of 5%. Results: CD3 and CD20 IC densities differed significantly by sample type (both p<0.0001) and primary site (CD3: p=0.012, CD20: p=0.0017). CD3 and CD20 densities were significantly lower in PT compared to LNM or in RD compared to PT and LNM. Paired samples (n=172) revealed a significantly higher CD3 and CD20 density (both p<0.0005) in LNM compared to PT, but no significant differences between PT and RD (n=28, p>0.05). CD3 densities were significantly higher than CD20 densities in all sample types. CD3_high patients had the best prognosis in all sites except for OC (q<0.05) independent of CD20 status. In OC, CD3 density was not prognostic, but CD3_low/CD20_high patients had the worst OS compared to CD3_low/CD20_low and CD3_high/CD20_high or even CD3_high/CD20_low patients (p=0.018) who had the best prognosis. Conclusions: IC densities of CD3 and CD20 vary by sample type and primary site. Except for OC, in all sites the prognostic impact is determined by CD3_high, whereas in OSCC only the combination of CD3 and CD20 IC densities achieves a good prognostic value. Interestingly, CD3_low/CD20_high patients have the worst overall survival in OC patients. Further work is needed to understand the interaction of B- and T cell infiltrates in the tumor, especially in OC.