Anorexia nervosa (AN) is a serious eating disorder characterized by self-starvation and extreme weight loss. Pseudoatrophic brain changes are often readily visible in individual brain scans, and AN may be a valuable model disorder to study structural neuroplasticity. Structural magnetic resonance imaging studies have found reduced gray matter volume and cortical thinning in acutely underweight patients to normalize following successful treatment. However, some well-controlled studies have found regionally greater gray matter and persistence of structural alterations following long-term recovery. Findings from diffusion tensor imaging studies of white matter integrity and connectivity are also inconsistent. Furthermore, despite the severity of AN, the number of existing structural neuroimaging studies is still relatively low, and our knowledge of the underlying cellular and molecular mechanisms for macrostructural brain changes is rudimentary. We critically review the current state of structural neuroimaging in AN and discuss the potential neurobiological basis of structural brain alterations in the disorder, highlighting impediments to progress, recent developments, and promising future directions. In particular, we argue for the utility of more standardized data collection, adopting a connectomics approach to understanding brain network architecture, employing advanced magnetic resonance imaging methods that quantify biomarkers of brain tissue microstructure, integrating data from multiple imaging modalities, strategic longitudinal observation during weight restoration, and large-scale data pooling. Our overarching objective is to motivate carefully controlled research of brain structure in eating disorders, which will ultimately help predict therapeutic response and improve treatment.