Classical Chinese sentence segmentation for tomb biographies of Tang dynasty
CL Liu, Y Chang - arXiv preprint arXiv:1908.10606, 2019 - arxiv.org
CL Liu, Y Chang
arXiv preprint arXiv:1908.10606, 2019•arxiv.orgTomb biographies of the Tang dynasty provide invaluable information about Chinese
history. The original biographies are classical Chinese texts which contain neither word
boundaries nor sentence boundaries. Relying on three published books of tomb
biographies of the Tang dynasty, we investigated the effectiveness of employing machine-
learning methods for algorithmically identifying the pauses and terminals of sentences in the
biographies. We consider the segmentation task as a classification problem. Chinese …
history. The original biographies are classical Chinese texts which contain neither word
boundaries nor sentence boundaries. Relying on three published books of tomb
biographies of the Tang dynasty, we investigated the effectiveness of employing machine-
learning methods for algorithmically identifying the pauses and terminals of sentences in the
biographies. We consider the segmentation task as a classification problem. Chinese …
Tomb biographies of the Tang dynasty provide invaluable information about Chinese history. The original biographies are classical Chinese texts which contain neither word boundaries nor sentence boundaries. Relying on three published books of tomb biographies of the Tang dynasty, we investigated the effectiveness of employing machine-learning methods for algorithmically identifying the pauses and terminals of sentences in the biographies. We consider the segmentation task as a classification problem. Chinese characters that are and are not followed by a punctuation mark are classified into two categories. We applied a machine-learning-based mechanism, the conditional random fields (CRF), to classify the characters (and words) in the texts, and we studied the contributions of selected types of lexical information to the resulting quality of the segmentation recommendations. This proposal presented at the DH 2018 conference discussed some of the basic experiments and their evaluations. By considering the contextual information and employing the heuristics provided by experts of Chinese literature, we achieved F1 measures that were better than 80%. More complex experiments that employ deep neural networks helped us further improve the results in recent work.
arxiv.org