This paper presents an approach to enhance the performance of appearance and shape based feature fusion for face recognition system using Principal Component Analysis and Hidden Markov Model. Though the traditional face recognition system is very sensitive to the face parameter variations, the proposed feature fusion based facial recognition system is found to be stance and performs well for improving the robustness and naturalness of human-computer-interaction. Active Appearance Model and Shape Model have been used to extract the appearance and shape based facial features from the facial images. Feature fusion is performed and combined feature vector is created using these two types of features. To reduce the dimensionality of the feature vector, Principal component Analysis method has been used. Hidden Markov Model has been used for learning and classification purpose. In experimental result, appearance based, shape based and combined appearance and shape based output are reported and shows the superiority of the proposed facial recognition system. © 2012 IEEE.