To accurately and efficiently distinguish the stem end and the blossom end of a navel orange from its black spots, we propose a feature skyscraper detector (FSD) with low computational cost, compact architecture and high detection accuracy. The main part of the detector is inspired from small object that the stem (blossom) end is complex and the black spot is densely distributed, so we design the feature skyscraper networks (FSN) based on dense connectivity. In particular, FSN is distinguished from regular feature pyramids, and which provides more intensive detection of high-level features. Then we design the backbone of the FSD based on attention mechanism and dense block for better feature extraction to the FSN. In addition, the architecture of the detector is also added Swish to further improve the accuracy. And we create a dataset in Pascal VOC format annotated three types of detection targets the stem end, the blossom end and the black spot. Experimental results on our orange dataset confirm that the FSD has competitive results to the state-of-the-art one-stage detectors like SSD, DSOD, YOLOv2, YOLOv3, RFB and FSSD, and it achieves 87.479% mAP at 131 FPS with only 5.812M parameters.