In this paper, an improved computation method of maximal Lyapunov character exponent (mLCE) is proposed to analyze the nonlinear dynamic behaviors in the V²C-mode-controlled buck converter. In this method, a compensation algorithm of the Jacobi matrix at the non-differentiable point is given after analyzing the geometric relationship between the two subsystems on both sides of the switching surface. Besides, according to the fact that the chaos control can be realized by increasing the correlation of system state variables, the coupling strength ε is introduced to the system of V²C-mode-controlled buck converter and optimized by particle swarm optimization (PSO), in order to eliminate the recognized chaotic behaviors. Finally, the effectiveness of the given method is validated by a simulation and an experimental setup.