The decomposition of magnetic dipole signal in orthonormal basis functions (OBFs) is widely used in underwater target detection. Traditionally, only the squared sum of three coefficients is constructed as a magnetic anomaly detector, but the space distribution characteristic of coefficients is not fully utilized. In this letter, the constraint of coefficients is analyzed, and their space distribution constraint that located on an ellipsoid surface is obtained. Moreover, when the magnetization direction of target is nearly horizontal position, the constraint relationship of coefficients in the signal space can be compressed into an elliptical plane. Then, we project the decomposed coefficients of the measured signal onto the elliptical plane to further reduce the noise by about 1/3, and the squared sum of renewed coefficients is constructed as a detector for anomaly judgment. The simulation is conducted, compared with the traditional OBF method, and it can achieve an incremental detection probability of about 6% in the case of low signal-to-noise ratio.