We have proposed a novel scheme called Bridge Apparent Damage Detection System (BADDS) based on the network model of preliminary screening and fine recognition enabled by Faster Region Convolutional Neural Network (Faster R-CNN), which effectively improves the accuracy as well as reduces the omission factor. The average precision (AP) of honeycomb pitting, crack and salt out can reach up to 90.10%, 90.81% and 99.11%, which are increased by 23.89%, 21.04% and 35.43% respectively, compared with those of Faster R-CNN. On the other hand, the omission factor of them in BADDS are 6.26%, 3.72% and 12.17%, which are decreased by 10.81%, 9.94% and 11.27% respectively. The system performance is evaluated by Precision-Recall (P-R) curve.