Higher polygenic risk score for schizophrenia (szPGRS) has been associated with lower cognitive function and might be a predictor of decline in brain structure in apparently healthy populations. Age-related declines in structural brain connectivity—measured using white matter diffusion MRI —are evident from cross-sectional data. Yet, it remains unclear how graph theoretical metrics of the structural connectome change over time, and whether szPGRS is associated with differences in ageing-related changes in human brain connectivity. Here, we studied a large, relatively healthy, same-year-of-birth, older age cohort over a period of 3 years (age ∼ 73 years, N = 731; age ∼76 years, N = 488). From their brain scans we derived tract-averaged fractional anisotropy (FA) and mean diffusivity (MD), and network topology properties. We investigated the cross-sectional and longitudinal associations between these structural brain variables and szPGRS. Higher szPGRS showed significant associations with longitudinal increases in MD in the splenium (β = 0.132, p_FDR = 0.040), arcuate (β = 0.291, p_FDR = 0.040), anterior thalamic radiations (β = 0.215, p_FDR = 0.040) and cingulum (β = 0.165, p_FDR = 0.040). Significant declines over time were observed in graph theory metrics for FA-weighted networks, such as mean edge weight (β = −0.039, p_FDR = 0.048) and strength (β = −0.027, p_FDR = 0.048). No significant associations were found between szPGRS and graph theory metrics. These results are consistent with the hypothesis that szPGRS confers risk for ageing-related degradation of some aspects of structural connectivity.