In this paper we develop a gestural communication perception system for a social robot companion that is able to autonomously learn novel gestures on-the-fly. The system constantly tracks human gestural activities with a camera and evaluates the performed gestures under an open-set assumption. This allows for the identification of unknown gestures. Once detected, the system stores motion sequences of the novel gesture class and employs a dialogue interaction with the human to automatically label the unknown gesture. Subsequently, the gestural model is updated, grounding the unknown gesture through dialog interaction. In our experiment, we evaluate a neural network with varying threshold values for the open gesture recognition with unknown detection. Results show that the general classifier reaches an accuracy of more than 83%, and an f1-score of 0.79 in an open-ended scenario. The method is furthermore tested in a first in-lab interaction setting, which shows the system usability and its potential for future personalized human-robot gestural communication.