Improving diamond growth to achieve high-quality materials for applications like optics and quantum detectors is a key objective. Leveraging machine learning, especially with its challenges such as complex, time-dependent data features and the sheer data volume per growth cycle, presents a promising solution. The extraction of accurate spatial features from images for real-time diamond growth monitoring is challenging due to data's complex and sparse nature. This paper evaluates various feature extraction methods in diamond growth, introducing a novel deep learning approach for precise geometric feature segmentation. It utilizes a human-in-the-loop system for efficient data annotation, significantly cutting down labeling time and costs. Our approach, using the DeeplabV3plus model, showcases high efficiency in feature classification, with accuracies up to 96.31% for pocket holders, 98.60% for diamond tops, and 91.64% for sides, demonstrating deep learning's potential in handling complex dataset features effectively.