Studies on contextual abnormality detection and localization for images and videos are presented in this work. The task of detecting abnormalities becomes challenging while considering the context in the scene. Some object which is normal in one scenario may be considered as abnormal in another. We present conceptually simple, flexible and a general framework, by incorporating instance segmentation, skip-gram with negative sampling and isolation forest for detecting and localizing contextual abnormality in images and videos. The skip-gram-based model is generally used for word2vec in natural language processing for finding the similarity between words. In this work, we extended them to detect the object-based abnormality in the images and video. Then we introduce the voting technique, which overcomes the variable-length feature vector issues; the decision of normal or abnormal object is based on this technique by considering the output from the isolation forest. We consider the anomalous events as scenarios having a different distribution from the normal settings such as a less frequently seen object in a given combination, the increase in the number of specific objects category, the object’s presence at unseen distance and occupancy of the out-of-vocabulary object. We observed that the proposed framework works in the proximity of multiple object categories and camera motion in the natural capture videos.