In this paper, uniformly unconditionally stable first and second order finite difference schemes are developed for kinetic transport equations in the diffusive scaling. We first derive an approximate evolution equation for the macroscopic density, from the formal solution of the distribution function, which is then discretized by following characteristics for the transport part with a backward finite difference semi-Lagrangian approach, while the diffusive part is discretized implicitly. After the macroscopic density is available, the distribution function can be efficiently solved even with a fully implicit time discretization since all discrete velocities are decoupled, resulting in a low-dimensional linear system from spatial discretizations at each discrete velocity. Both first and second order discretizations in space and in time are considered. The resulting schemes can be shown to be asymptotic preserving (AP) in the diffusive limit. Uniformly unconditional stabilities are verified from a Fourier analysis based on eigenvalues of corresponding amplification matrices. Numerical experiments, including high-dimensional problems, have demonstrated the corresponding orders of accuracy both in space and in time, uniform stability, AP property, and good performance of our proposed approach.