Due to the large amount of visual information generated daily, proposals that automatically analyze and process data are becoming increasingly necessary. This work focuses on the detection of anomalous objects in video sequences captured by PTZ (pan-tilt-zoom) cameras, considering as anomalies the objects which belong to categories that should not appear in a specific scene (e.g. pedestrians on a highway). There is a lack in the previous literature of a principled approach for the control of PTZ cameras that takes advantage of the recent developments in deep learning-based object detection. Our proposal aims to fill this gap by offering a probabilistic framework where the guidance of PTZ cameras is accommodated. The proposed methodology involves three different modules. An object detection stage, where deep learning networks are used to detect the objects that appear in the scene; an anomalous detection module, where a mixture of Dirichlet distributions is considered to detect automatically, and without supervised training, those detected objects which are likely to be anomalous; and finally, a PTZ camera controller which allows to follow and focus on the object considered as the most probably anomalous in the scene. The experimental results show the performance and viability of our proposal, which outperforms several competitors from qualitative and quantitative points of view.