Background: The diagnosis of a malignancy is typically informed by clinical presentation and tumor tissue features including cell morphology, immunohistochemistry, and molecular markers. Additionally, multi-omic approaches (Abraham 2021 Transl Oncol) and deep learning models using digital pathology (da Silva 2021 J Pathol) have augmented expert pathologists and led to improved diagnoses but are often not employed together on the same patient. The opportunity exists for a truly multimodal, multi-omic machine learning classifier that comprehensively assesses all aspects of a tumor from the molecular underpinnings to the morphological and histological phenotypic presentation to provide the most accurate diagnosis while at the same time providing predictive biomarker data from the same specimen. Methods: Whole transcriptome data from 220,246 tumor profiles, large panel and whole exome data from over 170,000 tumor profiles, and digital pathology features from over 50,000 tumors were used to construct a multi lineage classifier. The classifier was trained on 256 Oncotree classifications corresponding to established WHO diagnoses where a tumor of each class has been observed at least 30 times in our dataset. The dataset was split 50% for training and the other 50% for testing, UMAP was employed for dimensionality reduction, and ensemble models were used for making the final calls. The performance of the classifier was evaluated by comparing with traditional pathologist-directed diagnostic workup. Results: Tumor lineage classifiers predicted the correct classifications where the primary site was known with accuracies ranging between 97% and 100% when using the 32 highest level Oncotree categories corresponding to human tissues. Accuracy on the most granular Oncotree categories varied with many between 90 and 95%. When applied to Cancer of Unknown Primary (CUP) cases (n = 3589), an unequivocal Oncotree classification could be obtained over 90% of the time. Conclusions: Combining multi-omic and digital pathology information into a multimodal artificial intelligence platform can provide comprehensive information to pathologists to aid in diagnosis. This tool can be used to meet an unmet clinical need to define the lineage of CUP cases, which when coupled with biomarker data, will provide an opportunity to examine whether this information can be used to improve the outcomes of patient with CUP.