Ultra-wideband (UWB) impulse radar plays an important role in contactless vital sign (VS) detection. The VS can be extracted remotely by acquiring the oscillations in the human chest. Unfortunately, it is usually challenging to identify VS due to the low signal-to-noise ratio (SNR) only based on the traditional fast Fourier transform (FFT) especially in complicated conditions. To extract VS accurately, this paper presents a new scheme by analyzing the skewness characteristic of the received UWB impulses, which are modulated by life activities. The distance from the human subject to the radar antenna can be calculated by performing the discrete short-time Fourier transform (DSFT) on skewness. The frequency of human respiratory movement can be estimated based on the developed ensemble empirical mode decomposition (EEMD)-based accumulation technique by canceling out the harmonics effectively. The performance of the developed detection method is tested with several experiments carried out in different environments.