Face recognition in video is being actively studied as a covert method of human identification in surveillance systems. Identifying human faces in video is a difficult problem due to the presence of large variations in facial pose and lighting, and poor image resolution. However, by taking advantage of the diversity of the information contained in video, the performance of a face recognition system can be enhanced. In this work we explore (a) the adaptive use of multiple face matchers in order to enhance the performance of face recognition in video, and (b) the possibility of appropriately populating the database (gallery) in order to succinctly capture intra class variations. To extract the dynamic information in video, the facial poses in various frames are explicitly estimated using active appearance model (AAM) and a factorization based 3D face reconstruction technique. We also estimate the motion blur using discrete cosine transformation (DCT). Our experimental results on 204 subjects in CMU's face-in-action (FIA) database show that the proposed recognition method provides consistent improvements in the matching performance using three different face matchers.