In the realm of power systems, the increasing involvement of residential users in load forecasting applications has heightened concerns about data privacy. Specifically, the load data can inadvertently reveal the daily routines of residential users, thereby posing a risk to their property security. While federated learning (FL) has been employed to safeguard user privacy by enabling model training without the exchange of raw data, these FL models have shown vulnerabilities to emerging attack techniques, such as deep leakage from gradients and poisoning attacks. To counteract these, we initially employ a secure-aggregation (SecAgg) algorithm that leverages multiparty computation cryptographic techniques to mitigate the risk of gradient leakage. However, the introduction of SecAgg necessitates the deployment of additional subcenter servers for executing the MPC protocol, thereby escalating computational complexity and reducing system robustness, especially in scenarios where one or more subcenters are unavailable. To address these challenges, we introduce a Markovian switching-based distributed training framework, the convergence of which is substantiated through rigorous theoretical analysis. The distributed Markovian switching (DMS) topology shows strong robustness toward the poisoning attacks as well. Case studies employing real-world power system load data validate the efficacy of our proposed algorithm. It not only significantly minimizes communication complexity but also maintains accuracy levels comparable to traditional FL methods, thereby enhancing the scalability of our load forecasting algorithm.