The influence of different boundary conditions on the spectral properties of the incompressible Navier-Stokes equations is investigated. By using the Fourier-Laplace transform technique, we determine the spectra, extract the decay rate in time, and investigate the dispersion relation. In contrast to an infinite domain, where only diffusion affects the convergence, we show that also the propagation speed influence the rate of convergence to steady state for a bounded domain. Once the continuous equations are analyzed, we discretize using high-order finite-difference operators on summation-by-parts form and demonstrate that the continuous analysis carries over to the discrete setting. The theoretical results are verified by numerical experiments, where we highlight the necessity of high accuracy for a correct description of time-dependent phenomena.