Mammogram is the primary imaging technique for breast cancer screening, the leading type of cancer in women worldwide. While the clinical effectiveness of mammogram has been well demonstrated, the mammographic characteristics of breast masses are quite complex. As a result, radiologists certified for reading mammography are lacking, which limits the accessibility of mammography for more population. In this paper, we propose a Computer Aided Detection (CADe) method to automatically detect masses in mammography. Our method combines Faster R-CNN with Feature Pyramid Network, Focal Loss, NonLocal Neural Network to achieve the optimal mass detection performance. We comprehensively evaluated our method and baseline methods on three public datasets combined, namely the Digital Database for Screening Mammography (DDSM), INbreast, and Breast Cancer Digital repository (BCD). Our results demonstrate that our method outperforms the baselines by a large margin, reporting an Average Precision of 0.805, and a recall of 0.977, respectively.