Sentiment analysis is a field of research that analyzes people’s attitudes, psychology, emotions about a particular issue. In recent years, deep learning techniques have yielded remarkable success in sentiment analysis. One of the paramount factors that affect the effectiveness of the deep learning model in sentiment analysis is the representation of the words in the input documents. In this paper, first, we have evaluated the effectiveness of some recent word representation in sentiment analysis for Vietnamese reviews. Second, we have proposed a new feature representing a document named BERT-POS feature. This feature has both contextual semantic and syntactic information of the sentences in a comment. Thus, it helps to improve the accuracy of the sentiment analysis problem. Third, we have conducted several extensive experiments to prove the effectiveness of our proposed feature.