Contemporary understanding of human visual spatial attention rests on the hypothesis of a top-down control sending from cortical regions carrying higher-level functions to sensory regions. Evidence has been gathered through functional Magnetic Resonance Imaging (fMRI) experiments. The Frontal Eye Field (FEF) and IntraParietal Sulcus (IPS) are candidates proposed to form the frontoparietal attention network for top-down control. In this work we examined the influence patterns between frontoparietal network and Visual Occipital Cortex (VOC) using a statistical measure, Granger Causality (GC), with fMRI data acquired from subjects participated in a covert attention task. We found a directional asymmetry in GC between FEF/IPS and VOC, and further identified retinotopically specific control patterns in top-down GC. This work may lead to deeper understanding of goal-directed attention, as well as the application of GC to analyzing higher-level cognitive functions in healthy functioning human brain.