In recent years, social robots are widely research due to the advancement of electronics hardware and deep learning algorithms breakthrough. Hand gesture recognition playing an important role in human-robot interaction. However, to deploy state-of-art of deep learning methods for hand gesture recognition, researchers have to associate all hand gestures with their respective meanings (label). Re-train the deep learning model is required if a new gesture added for recognition. In addition, the meaning of hand gestures often depending on the environmental conditions. In this paper, we propose a framework that learns and recognizes hand gesture meaning based on surrounding objects. The proposed method consists of two sub frameworks: i) A deep learning model, Mask RCNN to detect and extract features of objects and hand gestures; ii) An incremental learning model, Declarative Memory Recurrent Neural Model to incrementally learn hand gestures meaning based on surrounding objects. The effectiveness of our proposed method is validated through several experiments.