¹⁸F-flutemetamol is a positron emission tomography (PET) tracer for in vivo amyloid imaging. The ability to classify amyloid scans in a binary manner as ‘normal’ versus ‘Alzheimer-like’, is of high clinical relevance. We evaluated whether a supervised machine learning technique, support vector machines (SVM), can replicate the assignments made by visual readers blind to the clinical diagnosis, which image components have highest diagnostic value according to SVM and how ¹⁸F-flutemetamol-based classification using SVM relates to structural MRI-based classification using SVM within the same subjects. By means of SVM with a linear kernel, we analyzed ¹⁸F-flutemetamol scans and volumetric MRI scans from 72 cases from the ¹⁸F-flutemetamol phase 2 study (27 clinically probable Alzheimer's disease (AD), 20 amnestic mild cognitive impairment (MCI), 25 controls). In a leave-one-out approach, we trained the ¹⁸F-flutemetamol based classifier by means of the visual reads and tested whether the classifier was able to reproduce the assignment based on visual reads and which voxels had the highest feature weights. The ¹⁸F-flutemetamol based classifier was able to replicate the assignments obtained by visual reads with 100% accuracy. The voxels with highest feature weights were in the striatum, precuneus, cingulate and middle frontal gyrus. Second, to determine concordance between the gray matter volume- and the ¹⁸F-flutemetamol-based classification, we trained the classifier with the clinical diagnosis as gold standard. Overall sensitivity of the ¹⁸F-flutemetamol- and the gray matter volume-based classifiers were identical (85.2%), albeit with discordant classification in three cases. Specificity of the ¹⁸F-flutemetamol based classifier was 92% compared to 68% for MRI. In the MCI group, the ¹⁸F-flutemetamol based classifier distinguished more reliably between converters and non-converters than the gray matter-based classifier. The visual read-based binary classification of ¹⁸F-flutemetamol scans can be replicated using SVM. In this sample the specificity of ¹⁸F-flutemetamol based SVM for distinguishing AD from controls is higher than that of gray matter volume-based SVM.