Deep learning techniques have demonstrated their ability to facilitate medical image diagnostics by offering more precise and accurate predictions. Convolutional neural network (CNN) architectures have been employed for a decade as the primary approach to enable automated diagnosis. On the other hand, recently proposed vision transformers (ViTs) based architectures have shown success in various computer vision tasks. However, their efficacy in medical image classification tasks remains largely unexplored due to their requirement for large datasets. Nevertheless, significant performance gains can be achieved by leveraging self-supervised learning techniques through pretraining. This paper analyzes performance of self- supervised pretrained networks in medical image classification tasks. Results on colon histopathology images revealed that CNN based architectures are more effective when trained from scratch, while pretrained models could achieve similar levels of performance with limited data.