Area V5 or Middle Temporal (MT) area of the primate brain is said to be involved in visual motion perception. Physiological studies indicate that the neurons in MT respond selectively to the direction of moving stimuli. However in response to the complex stimuli containing multiple oriented components, a set of MT neurons are selective to the direction of the component motion whereas the other set of MT neurons are selectively respond to the direction of the whole pattern motion. This paper discusses a two layer LISSOM model (Laterally Interconnected Synergetically Self-Organizing Map) which is analogous to neurons in the cortical areas V1 as well as MT. The adaptive Hebbian learning technique has been used to train the network with sequences of moving square stimuli and observed the following: i) afferent weight connections of V1 neurons are tuned as orientation detectors and ii) neurons at MT is tuned to the whole pattern motion. Lateral connections in each layer mediate the competition between the neurons which results in a topographic map. The results from the two layer LISSOM model was found to be in-line with that of well known experimental studies results.