Plant or crop diseases are dominant factors that influences crop quality and quantity. Therefore segmentation, classification, and detection of diseased symptoms at an early stage of infection are very essential. Deep learning technology tackles these tasks jointly in precision agriculture. In this paper, we propose an effective deep learning framework for automatic plant disease detection and segmentation, DPD-DS, using an improved pixel-wise mask-region-based convolution neural network (CNN). The proposed DPD-DS introduces a light head region convolution neural network (R-CNN) with the intention of saving the memory space and computational cost. It increases the detection accuracy and computational speed by adjusting the proportions of the Anchor in the RPN network and modifying the structure of the backbone. To test the feasibility and robustness of the proposed approach, DPD-DS model is compared with the existing state-of-art models. The experimental results convey that the proposed method attains the best results in precision, recall, and mean average precision (mAP), which is comparatively better than existing methods. Furthermore, the proposed framework detection time is significantly decreased by more than two times, improving the model’s efficiency to detect plant leaf diseases.