A new rubber-band straightening transform (RBST) and texture analysis were developed for classification of masses on mammograms as malignant or benign. A database of 168 mammograms containing biopsy-proven malignant and benign breast masses was digitized at a pixel size of 100 micrometer by 100 micrometer. Regions of interest (ROIs) containing a single mass from each mammogram were extracted by an experienced radiologist. A clustering algorithm was employed for automatic segmentation of each ROI into a mass object and background tissue. A 40-pixel-wide region of the gray-scale image surrounding the mass was transformed onto the Cartesian plane using RBST. The first row of the transformed image contained the pixels along the mass edge. The j^th row of the transformed image contained the pixel values at a distance of j pixels from the edge in the normal direction to the edge contour. In the RBST image, mass margins appeared as approximately horizontal edges, and spiculations appeared as approximately vertical lines. Thus, edge sharpness and possible spiculations could be better characterized with directional texture measures. Texture features were extracted from spatial gray-level dependence matrices calculated from the RBST images. Linear discriminant analysis was used for classification, and receiver operating characteristic (ROC) analysis was used to evaluate the classification accuracy. The area A_z under the test ROC curve reached 0.87 using cross-validation and 0.89 using the leave-one-case-out method.