Background: [F18]DCFPyL (PyL) is a PSMA targeted imaging agent that provides whole-body staging of prostate cancer. Image analysis of the primary tumor using deep learning algorithms might offer additional insight into disease biology, including co-existing metastatic disease. We developed convolutional neural network (CNN) models using inputs from the entire prostate on PyL images along with auto-segmented hot spots within the of primary prostate cancer tumor to predict synchronous metastases and compared the models against established models built from clinico-pathologic data. Methods: 91 Veterans with de novo prostate cancer were imaged with PyL PSMA PET/CT for initial staging (46% with metastatic disease). The PyL images of the prostate were analyzed using aPROMISE, which autosegments, localizes, and quantifies disease on PSMA PET. The segmentations of the prostate were used to map the PyL PET image of the prostate. Both the entire prostate, as well as aPROMISE defined hot spots were used as inputs for the CNN, where, according to attention map analysis, the hotspot information helps the network understand the location and extent of tumors. The CNN model architecture was based on SqueezeNet v2. The dataset was split into training, validation and test sets using stratified random sampling. The area under ROC curve (AUC) was computed to determine the performance of the model in predicting the presence of metastases and the test predictions were compared with ground truth (M1). Training was repeated 50 times and the best performing experiment was identified. Prediction scores from UCSF-CAPRA and UCLA PSMA risk calculator were used as comparators. Results: The best CNN model had an AUC of 0.89 for prediction of metastatic disease (median 0.72, ICR 0.64 and 0.8). For comparison, the AUC from the UCSF-CAPRA score and UCLA-PSMA risk calculator (any upstaging on PET), which rely on clinicopathologic information, had AUCs of 0.729 and 0.754 in this dataset, respectively. Conclusions: The CNN based model using PyL imaging demonstrates that synchronous metastases can be predicted from intraprostatic PyL uptake patterns alone with a predictive accuracy in this dataset that is at least comparable to published prediction models based on clinicopathologic features. This study raises the hypothesis that PyL CNN-based models could be developed to prognosticate metastatic progression.