Background: Expression of inhibitory immune checkpoints (ICPs) within tumors has emerged as an important barrier for effective anti-tumor immunity. Antibody-mediated blockade of ICPs can lead to durable responses in patients. Interestingly, only a small subset of tumor infiltrating lymphocytes (TILs) express these checkpoints and there is a need to better understand the characteristics of this subset. We undertook this study to understand characteristics of TILs within melanoma. Methods: We used single cell mass cytometry, gene expression profiling of purified T cell subsets, T cell receptor (TCR) sequencing as well as functional studies to understand the characteristics of TILs in melanoma patients (n=50). We also performed exome sequencing of tumor cells in some patients. Results: We find that TILs are functionally and phenotypically distinct from circulating T cells. They express higher levels of inhibitory ICPs (PD-1, TIM-3) and secrete less IL2, IFNg and TNFa than T cells in circulation. Expression of vascular endothelial growth factor within tumors correlated with reduced T cell infiltration. Expression of ICPs (PD-1, TIM-3, PD-L1) were enriched in T cells with a phenotype and expression profile of tissue resident memory T (TRM) cells with most cells expressing multiple checkpoints. Within the myeloid compartment, ICPs were predominantly expressed on CD14+CD16+ subset. TCR sequencing revealed that individual melanoma metastases revealed that the top clones within each of the lesions have distinct TCRs. Concurrent TCR and tumor exome sequencing of individual metastases in the same patient revealed that inter-lesional diversity of TCRs exceeded differences in mutation/neoantigen load in tumor cells. Conclusions: Our findings suggest that TRM cells and CD16+ myeloid cells may be the major target of ICP blockade within tumors. The ability to activate, and retain T_RM cells may be an important determinant of the T cell content of the tumor microenvironment and should be a goal for future vaccines. Importantly, our study illustrates inter-lesional diversity of TCRs within individual metastases which may differentially impact the outcome of immune therapy at each site.