Bundle adjustment is a least squares method-based algorithm for minimizing the global reprojection error and has provided an effective solution for structure from motion (SfM). The Levenberg–Marquardt algorithm provides a feasible and convenient way for bundle adjustment and creates a system of linear equations which are normal equations. For the special sparsity structure of an augmented Hessian matrix, a Schur complement trick is introduced to reduce computation complexity. However, the general sparse matrix storage formats are not optimized for the augmented Hessian matrix and consume too much computation time. According to the Schur complement trick, this paper divides the arrow-like augmented Hessian matrix into a structure matrix, a camera matrix and an observation matrix, and then proposes a new compressed matrix sequence (CMS) method to reduce time complexity for matrix operations. Under the definition of CMS, all of the matrices are stored in a dense form to accelerate the matrix operations, of which the matrix operations are redefined as well. CMS costs little computation time to build sparse matrices or access sparse matrices. The experimental results show that CMS achieves a significant speedup over general sparse matrix storage formats. Also, CMS being insensitive to the data input is more stable.