We develop a machine learning approach to content-based image categorization and retrieval. We represent images by histograms of their spectral components associated with a bank of filters and assume that a training database of labeled images–that contains representative samples from each class–is available. We employ a linear dimension reduction technique, referred to as Optimal Factor Analysis, to identify and split off “optimal” low-dimensional factors of the features to solve a given semantic classification or indexing problem. This content-based categorization technique is used to structure databases of images for retrieval according to the likelihood of each class given a query image.