Protein-Protein Interaction Article Classification Using a Convolutional Recurrent Neural Network with Pre-trained Word Embeddings

1 Introduction

Extraction of protein-protein interaction (PPI) information from the literature is of utmost importance for biomedicine, since the understanding of disease, pharmacological and other processes requires the analysis of networks formed by these relations. Several databases maintain manually curated protein-protein interaction data but, since the primary source for identifying PPIs is the scientific literature, keeping these databases up-to-date is a demanding and expensive task. The use of named-entity recognition (NER) and relation extraction methods in assisted curation workflows has been shown to expedite this work [1], [2]. An important step in such workflows is document prioritization or triage, in order to select articles that are more likely to contain relevant information.

Retrieval and extraction of PPI related information has been a major focus of recent shared evaluations in the biomedical domain, namely in the BioCreative challenges. Lan et al. [3] compared the use of bag-of-words (BoW), interaction trigger words and protein named entities (NEs) features in a support vector machine (SVM) classifier, applied to the BioCreative-II PPI task data. Their best result, when using a single classifier, was obtained with a feature set containing BoW features and protein NEs co-occurring with interaction trigger words (F-score of 77 %). Abi-Haidar et al. [4] tested three classifiers in the same data set: SVM, variable trigonometric threshold classifier (VTT), and a nearest neighbor classifier with singular value decomposition (SVD) applied for feature selection. They reported a top F-score of 78 % using the VTT classifier with a feature set of 650 discriminating words.

Several other works have also addressed the problem of document prioritization for protein-protein interactions. Sumoela and Andrade [5] proposed a classification and ranking model to evaluate the entire MEDLINE database, the largest repository of scientific literature in the life sciences, with respect to any topic of interest. Their method is based on selecting words that commonly convey meaning, namely nouns, verbs, and adjectives, and relies on the different frequencies of these discriminating words between a set of relevant articles and a reference set. This approach is also behind the MedlineRanker web-service [6], which allows to retrieve a list of articles ranked by similarity to a training set defined by the user. One possibility, as referred by the authors, is to use a list of document identifiers obtained from a PPI database, therefore getting as result other articles related to that same topic. Marcotte et al. [7] proposed a log likelihood scoring function to identify articles discussing PPIs, using a feature set composed of 83 discriminating words selected from a training set of 260 MEDLINE abstracts involving yeast proteins. They reported an accuracy of 77 %, with a recall around 55 %, when articles with a log likelihood score of 5 or higher were selected.

The BioCreative III PPI Article Classification Task (ACT) counted with 52 submissions from ten participating teams [8]. Most teams applied some sort of machine learning technique, the best results being obtained using support vector machines, Maximum Entropy or Large Margin classifiers. The top performing teams used various levels of lexical analysis, including part-of-speech (PoS) tagging and named entity recognition (NER), and the best team overall also used dependency parsing to extract the textual features used for classification. Additionally, various teams used the manually assigned MeSH terms, which are indexing terms that provide information regarding the article’s subject. The best AUC iP/R (area under the interpolated precision-recall curve) was 0.680 and the highest MCC (Matthew’s correlation coefficient) was 0.553, with an accuracy of 89.2 % and an F-score of 61.4 % [9].

In recent works, deep learning classifiers have been applied successfully to many tasks, most notably when applied to image data but also in many text and natural language processing problems. Similarly to traditional neural networks, deep learning classifiers are composed of processing units arranged in data transformation layers that apply simple non-linear functions to obtain different levels of representation of the input data [10]. In deep learning, however, a larger number of layers and/or units per layer are used, which allows the method to learn more complex classification functions. Another great advantage of such methods is that they eliminate the feature engineering effort that is required in traditional machine learning [10].

For text based tasks, the input data needs to be encoded in a way that can be used by the deep network classifier. This can be achieved by representing each word as a vector of a relatively small dimension. This way, each document is represented by a sequence of word vectors which are fed directly to the network. Word embeddings is a technique that derives such vector representations of words from large unannotated corpora, representing words with similar semantics by vectors that are close to one another in the vector space [11]. The use of this representation together with deep learning techniques have led to improved results in different NLP tasks, including word sense disambiguation [12], text classification [13], and named entity recognition [14].

Convolutional neural networks (CNN) are one of the most popular network architectures used in deep learning, having been extensively applied in image recognition and classification problems with very good performance. Various works also demonstrate their application in text classification tasks [13], [15], [16]. Nonetheless, the sequential nature of natural texts can be better modeled by recurrent neural networks (RNN), which contain a feedback loop that allows the network to use information regarding the previous state. Long short-term memory (LSTM) networks are a special type of RNN in which a set of information gates is introduced that allow these networks to learn long-term dependencies while avoiding the vanishing gradient problem [17], [18], [19].

An important consideration when defining a deep neural network for a given classification problem is related to selecting the network topology, namely type and number of layers and number of units in each layer, and model parameters such as activation function in each layer, loss function and optimizer algorithm.

Another important aspect is related to overfitting, which means that the network is capable of learning the “best” representation for the data used in training but is not able to generalize to unseen data. Various strategies have been proposed and are commonly employed to address this problem, namely early stopping, dropout, and regularization. The first is based on stopping the training when the value of the loss function, measured in an held-out part of the training data, stops decreasing. Dropout freezes the weights of a fraction of the units in the previous layer. This means that after a training epoch the weights of those units are not changed, so that the network is forced to assign importance to other features and does not focus on some parts of the feature space. Regularization is a common strategy used for trying to avoid overfitting. We used L2 regularization, which increases the value of the loss function when larger weights are used and thus benefits the distribution of weights across more features.

In this work we applied deep learning methods, namely convolutional recurrent neural networks, for prioritization of MEDLINE articles containing protein-protein interaction information. The paper is organized as follows: the next section describes the methods and data used, followed by the presentation of results, and finally the conclusions.

2 Methods

This section describes the data and methods used. Text processing and classification tasks were implemented in Python, using the Scikit-learn machine-learning library [20] and Keras [21]. Word embedding models were calculated with the Word2Vec [22] implementation in the gensim framework [23].

2.2 Model

The implemented network, illustrated in Figure 1, is composed of an embedding layer, a dropout layer, a convolutional layer followed by average pooling, a long-short term memory (LSTM) layer, and a dense layer with sigmoid activation function. The structure of the network was selected empirically by repeating various tests on the training and development sets.

Figure 1:

Deep network architecture.

The embedding layer represents the sequence of words in the documents by a sequence of embedding vectors. We used Word2Vec to create embedding models for 15 million abstracts from the full MEDLINE, containing around 775 thousand distinct words. We tested various models, with windows of 5, 20, and 50 and vector sizes of 100 and 300. Preliminary results showed that the embedding vectors of size 300 always gave better results. In the following experiments we used the model with 300 features calculated for window sizes of 20 and 50, as these gave consistently better results in preliminary tests than the other models.

The embedding layer is followed by a dropout layer in order to reduce over-fitting and thus improve generalization. The final network uses a dropout rate of 0.1.

A 1-dimensional convolutional layer is used to extract higher order features from the sequence of word vectors. We used a layer with 128 filters with kernel size set to 3 and rectified linear unit (ReLU) activation function. This is followed by average pooling over a window of 3.

The LSTM layer was configured with 128 units. Dropout was also used in this layer to prevent over-fitting, with a rate of 0.2. The output of the LSTM is then connected to the output layer. This consists of two neurons that apply a sigmoid activation function to obtain probability outputs in the range of 0–1 for each class. L2 regularization was used in this layer, with parameter set to 0.01.

Parameters such as the activation function in the convolution layer and in the final layer, number of convolution filters, the number of units in the LSTM, the dropout rates, regularization and optimizer function, were selected by performing a grid search over the parameters. For this, and to limit the computational cost, we selected 20 % of the development set (400 documents) and applied five-fold cross-validation for each parameter combination. In each fold, 80 % of the data were used for training and 20 % were used for evaluating the classification performance. In this process, we used a maximum of 10 training epochs with early stopping. Of the training data in each fold, 10 % were left out as validation data and the training was stopped if the loss in this validation data stopped decreasing.

Table 1 shows the parameter combinations tested, with the selected combination shown in bold.

Table 1:

Network parameter combinations tested using grid search.

Parameter	Values
Number of units in LSTM	96, 128
Number of filters in convolutional layer	96, 128
Droupout rate after embedding layer	0, 0.1, 0.2, 0.3, 0.4, 0.5
Droupout rate in LSTM	0, 0.1, 0.2, 0.3, 0.4, 0.5
Activation in convolutional layer	ReLu, hyperbolic tangent (tanh)
Activation in final layer	sigmoid, softmax
Optimizer algorithm	Adam, RMSprop
Regularization parameter	0, 0.01, 0.02

1. Values in bold were selected as the best combination.

3 Results and Conclusions

Table 2 shows the results obtained on the test set of the BioCreative III article classification task. Two embedding models with different window sizes (20 and 50) were compared. The network parameters were set as described in the previous section.

For this task, the state-of-the-art results were achieved during the BioCreative III challenge. Comparing to the best results, our classifier shows an improvement in terms of area under the precision-recall curve (0.715 vs. 0.680) and Matthew’s correlation coefficient (0.600 vs. 0.551), with a similar accuracy (0.894 vs. 0.892).

We present results for the prioritization of scientific articles containing information regarding protein-protein interactions, following a classification based ranking approach. We evaluated the combination of convolutional and recurrent neural networks, and show that these methods improve the classification and ranking performance over the previous state-of-the-art results. Importantly, these improvements were achieved without the need for features such as grammar relations extracted from full dependency parsing results, MeSH index terms, or the results of named-entity recognition systems.

On the other hand, setting of network parameters has an important impact on the final classification performance, but this can be achieved through grid search, as done in this work. The initialization of network weights may also have an impact, and weight initialization strategies are an interesting direction for improving the results of this approach.