The imaging geometry of a real imaging system always deviates from the ideal projection models. This deviation is treated as distortion, which is always estimated along with other parameters when calibrating a camera. Although the perspective projection model is chosen in most existing distortion estimation methods, it can't be used to describe a fish-eye imaging system. In this paper, we proposed a method to estimate the distortion of the fish-eye cameras in which the stereographic projection model is being used. Images of spheres and sets of parallel lines are utilized to estimate the distortion parameters. The method is validated on synthetic, simulated, and real images. Experimental results show that our method is robust to noise, and a subpixel accuracy is achieved. By applying estimated parameters, undistorted fisheye images can be treated as a stereographic projection of the world, which will be highly convenient for computer vision tasks.