Classification and Comparison via Neural Networks

3.1. Proposed Solution: A Combined Neural Network Architecture

Our neural network architecture is inspired by siamese networks (Bromley et al., 1994), which are extensively used for regressing similarity between two inputs. A siamese network contains two identical base networks, followed by a module predicting the similarity between the inputs of the base networks. In this work, we extend the generic application of siamese networks to regress both absolute and comparison labels simultaneously. To do so, we adopt the combined neural network architecture given in Fig. 1.

Our architecture is based on the assumption that absolute and comparison labels depend on the same latent variables. Formally, there exists a function $f_b:{ℝ}^d\to {ℝ}^p$ representing the coupling between absolute and comparison labels. We interpret f_b(x_i) as a latent vector informative for both absolute and comparison label predictions involving i. Therefore, given items (i, j), the absolute label y_i is regressed from f_b(x_i) and the associated comparison label y_(i,j) is regressed from f_b(x_i) and f_b(x_j). We refer to f_b as the base network, which has the same structure and parameters for all items i ∈ {1,…, N}.

Given f_b, our combined neural network comprises two sub-networks: the absolute network f_a and the comparison network f_c. The absolute network $f_a:{ℝ}^d\to ℝ$ receives x_i as input and outputs ${\widehat{y}}_i$, the absolute label prediction. Formally, for all (i,y_i) ∈ D_a:

where ∘ denotes function composition, $f_b:{ℝ}^d\to {ℝ}^p$ is the base network, and $h_a:{ℝ}^p\to ℝ$ is an additional network linking latent features to absolute labels. The comparison network $f_c:{ℝ}^d\to ℝ$ is trained via a so-called siamese architecture. Given two inputs x_i and x_j, the corresponding comparison label prediction ${\widehat{y}}_{(i,j)}$ is given by:

where $f_b:{ℝ}^d\to {ℝ}^p$ is again the base network and $h_c:{ℝ}^p\to ℝ$ is an additional network linking latent features to a one-dimensional score f_c = h_c ∘ f_b. We denote the parametrization of neural network functions f_b, h_a, and h_c w.r.t. their weights as f_b(·; W_b), h_a(·; W_a), and h_c(·; W_c), respectively. Note that, as a result, fa(·) = fa(·; Wa, W_b) and f_c(·) = f_c(·; Wc, W_b). Given both Da and Dc, we learn f_b(·; W_b), h_a(·; W_a), and h_c(·; W_c) via a minimization of the form:

We train our combined neural network via stochastic gradient descent over Eq. (3). Here, L_a is a loss function for the absolute network and L_c is a loss function for the comparison network; we instantiate both below, in Section 3.2. The balance parameter α ∈ [0, 1] establishes a trade-off between the absolute label set D_a and the comparison label set D_c. When α = 0, our model is restricted to the comparison/siamese network setting. When α =1, our model is restricted to the absolute/classical neural network setting.

This architecture is generic and flexible. In particular, the special case where h_a = h_c naturally captures the relationship between binary absolute and comparison labels described in the beginning of Section 3. Indeed, under this design, the event y_i,j = +1 becomes more likely when i has a higher propensity to receive the absolute label y_i = +1, compared to j. Nevertheless, we opt for the more generic scenario in which h_a and h_c may differ; joint prediction of absolute and comparison labels is thus more flexible. Finally, our design naturally extends to multi-class absolute labels, multiple labelers, and rankings. We present and apply an extension to multi-class absolute labels collected from multiple labelers in our experiments (c.f. ROP in Section 4). We discuss how to generalize to multi-class absolute labels in Appendix A, and how to generalize to rankings via the so-called Plackett-Luce model (Luce, 2012) in Appendix B.

3.2. Loss Functions

We train f_a using classical loss functions L_a, such as cross-entropy, hinge loss, and mean-square loss. To train f_c via L_c, we use two linear comparison models: Bradley-Terry and Thurstone. In contrast to cross-entropy and hinge losses, linear comparison models consider the order between compared items. The main advantage of these models is the capability of learning a score for each item, even though the labelers provide only binary comparative information, i.e., y_(i,j) ϵ { − 1, +1} (Cattelan, 2012). This score is then used to predict not only the absolute label of the item, but also the rank of the item among all items in the dataset.

Formally, linear models for pairwise comparisons assert that every item i in a labeled comparison dataset D_c is parametrized by a (non-random) score s_i (Bradley & Terry, 1952; Imrey, 1998). Then, all labelling events in D_c are independent of each other, and their marginal probabilities are:

where F is a cumulative distribution function (c.d.f.). When F is the standard normal c.d.f., Eq. (4) becomes the Thurstone model, yet when F is the logistic c.d.f., Eq. (4) becomes the Bradley-Terry model (Cattelan, 2012). We extend the linear comparison model in Eq. (4) by introducing a generic representation of the score s_i. In particular, we assume that $s_i=f_c\left({\mathit{x}}_i\right),,\forall i\in \{1,\dots ,N\}$, where f_c is the neural network described in Eq. (2). Then, given x_i, x_j, and y_{(i j)} under the Bradley-Terry model, the (negative log-likelihood) comparison loss becomes:

where ${\widehat{y}}_{(i,j)}=f_c\left({\mathit{x}}_i\right)-f_c\left({\mathit{x}}_j\right)$. Similarly, under the Thurstone model, the (negative log-likelihood) comparison loss becomes:

Datasets

We evaluate our model on four real-life datasets, GIFGIF Happiness, GIFGIF Pleasure, ROP, and FAC, summarized in Table 1a.

GIFGIF is an MIT Media Lab project aiming to understand the emotional content of animated GIF images (MIT Media Lab). Labelers are provided with two images and are asked to identify which image better represents a given emotion. A labeler can choose either image, or use a third option: neither. We discard the outcomes that resulted in neither. This dataset does not contain absolute labels. We manually label N = 1371 GIF images within the category happiness, labeling image i with the absolute label y_i = +1 if regarded as happy and y_i = −1 if sad. There are 9617 pairwise comparisons among these 1371 images, where y_(i,j) = +1 if image i is regarded as happier than j, and y_(i,j) = −1, o.w. We repeat the same procedure for the emotion category pleasure and manually label N = 1125 GIF images within this category with y_i = +1 if the subject of the image i is pleased and y_i = −1 o.w. There are 5667 pairwise comparisons among these 1125 images.

Retinopathy of Prematurity (ROP) is a retinal disease occuring in premature infants; it is a leading cause of childhood blindness (Gole et al., 2005). The ROP dataset (Brown et al., 2018) contains two sets of images: a small set of N = 100 retinal images, which we denote by S, and a larger set of N = 5561 images that we denote by H. Images in both sets receive Reference Standard Diagnostic (RSD) labels (Ryan et al., 2014): To create an RSD label for a retinal image, a committee of three experts labels the image as ‘Plus’, ‘Pre-plus’, or ‘Normal’ based on the existence of ROP, indicating severe, intermediate level, and no ROP, respectively. These ordered categorical labels constitute our absolute labels. In addition, five experts independently label all 4950 pairwise comparisons of the images in set S only. Note that some pairs are labeled more than once by different experts. For each pair (i, j), the comparison label is y_(i,j) = +1 if image i’s ROP severity is higher. Note that only images in S contain both absolute and comparison labels. We use S for training and reserve H for testing purposes.

The Filter Aesthetic Comparison (FAC) dataset (Sun et al., 2017) contains 1280 unfiltered images pertaining to 8 different categories. Twenty-two different image filters are applied to each image and resulting images are labelled by Amazon Mechanical Turk users. Filtered image pairs are labelled such that the comparison label y_(i,j) = +1 if filtered image i has better quality than filtered image j, and y_(i,j) = −1 o.w. Each filtered image appears in three comparison pairs. At the same time, absolute labels are generated by the creators of FAC dataset as follows. For each pair (i, j) such that y_(i,j) = +1, i and j receive scores +1 and −1, respectively. Hence, each filtered image receives a score in [−3, +3]. Subsequently, image i receiving a +3 (−3) score is assigned the label y_i = +1 (y_i = −1); images that do not receive +3 or −3 score are discarded. Thus, the absolute label y_i = +1 if the filtered image i has high quality and y_i = −1, o.w. We choose one of the categories with N = 3520 filtered images as our dataset, since comparison labels only exist for filtered image pairs within the same category. All in all, there are 3520 binary absolute labels and 4964 pairwise comparison labels in this dataset.

Network Architectures

We use GoogLeNet (Szegedy et al., 2015) without the last two fully connected layers as our base network f_b. We resize all images, so that ${\mathit{x}}_i\in {ℝ}^{3\times 224\times 224}$. The corresponding base network output f_b(x_i) has p = 1024 dimensions ∀_i ϵ {1,2, ⋯, N}. To leverage the well-known transfer learning properties of neural networks trained on images (Bengio et al., 2013), we initialize the layers from GoogLeNet with weights pre-trained on the ImageNet dataset (Deng et al., 2009). We choose h_a and h_c as single fully connected layers with sigmoid activations. We add L2 regularizers with a regularization parameter A to all layers and train our combined neural network end-to-end with stochastic gradient descent.¹ We instantiate our loss functions and hyperparameters as shown in Table 1b. Specifically for ROP, we first pre-process retinal images with a pre-trained U-Net (Ronneberger et al., 2015) to convert the colorful images into black and white masks for retinal vessels (Brown et al., 2018). Recall that the special case where h_a = h_c, combining cross-entropy as L_a and Bradley-Terry loss as L_c, naturally captures the relationship between absolute and comparison labels (c.f. Section 3), at least in binary classification. In our experiments, we implement both h_a ≠ h_c and h_a ≠ h_c (c.f. Section 4.2).

Experiment Setup

We allocate 60% of the total of N images in the GIFGIF Happiness, GIFGIF Pleasure, ROP S, and FAC datasets for training, 20% for validation, and 20% for test. Thus, images in the training set are not paired with images in the validation or test sets for the generation of comparison labels. For ROP, we augment the test set by adding the 5561 images in set H and predict the existence of ROP during test time, i.e. ${\widehat{y}}_i=-1$ if image i is predicted to be ‘Normal’ and ${\widehat{y}}_i=+1$, o.w. Recall that H contains only absolute labels and no comparisons (c.f. Table 1a).

Given α ϵ [0, 1], for each combination of the loss functions L_a, L_c, and hyperparameters λ, L.R. shown in Table 1b, we train our architecture over the training set. Then, we evaluate the resulting models on the validation set to determine the optimal loss functions and hyperparameters. We predict the absolute and comparison labels in the test set using the corresponding optimal models. When α ϵ (0, 1), functions f_b, h_a, and h_c are trained (and tested) on both datasets D_a and D_c. Note that when α ϵ = 0.0, by Eq. (3), we only learn f_b and h_c, as dataset D_a is not used in training; in this case, we set h_a = h_c to predict the absolute labels on the validation and test sets. Similarly, when α = 1.0, we only learn f_b and h_a, and set h_c = h_a to predict comparisons.

Competing Methods

We implement four alternative methods which incorporate absolute and comparison labels: Sun et al. (2017)’s method, logistic regression, a support vector machine (SVM), and an ensemble method combining the predictions of logistic regression and SVM. We use Sun et al. (2017)’s norm-comparison loss function L_c, introduced in detail in Appendix C, combined with the best performing L_a for each dataset (c.f. Table 1b). In both logistic and SVM methods, we first use GoogLeNet pre-trained on the ImageNet dataset (Deng et al., 2009) to map input images to features. Then, we train a model via logistic regression and an SVM, respectively, on these features. Training procedures for these methods are reported in more detail in Appendix D. Finally, we implement a soft voting ensemble method combining logistic regression and SVM (Friedman et al., 2001). More precisely, we train another logistic classifier on the pairs of logistic regression and SVM predictions.

Metrics

To evaluate the performance of our model on validation and test sets, we use Area Under the ROC Curve (AUC), accuracy (Acc.), F1-score (F1), and Area Under the Precision-Recall Curve (PRAUC) on both absolute labels D_a and comparison labels D_c. We report these metrics along with the 95% confidence interval, which we compute as 1.96 × σ_a, where ${\sigma }_A^2$ is the variance. Variance for AUC and PRAUC are computed by:

where A is the AUC or PRAUC, P_x = A/(2 − A), P_y = 2A²/(1 + A), and m, n are the number of positive and negative samples, respectively (Hanley & McNeil, 1982). Variance for accuracy and F1-score are computed by:

for A being the accuracy or F1-score. Finally, when comparing the prediction performances of different methods, we assess the statistical significance of relative improvements. We do so via the p-value of one-sided Welch T-test for unequal variances (Sawilowsky, 2002), which is computed by:

where Φ is the standard normal cumulative distribution function (c.d.f.), and A_l and A_s are the metric values with larger and smaller magnitudes, respectively.

4.1. Prediction Performance

Tables 2a and 2b show the prediction performance of our model trained on GIFGIF Happiness, GIFGIF Pleasure, ROP, and FAC datasets. For each a, we optimize the hyperparameters on the validation set following the procedure explained in Experiment Setup. For AUC, Acc., F1, and PRAUC on both D_a and D_c, we evaluate all eight metrics on the test set, for α = 0.0 (comparison labels only), α =1.0 (absolute labels only), and best performing α ∈ [0, 1]. Optimal hyperparameters are reported in Appendix E (c.f. Tables E.4 to E.11).

Tables 2a and 2b indicate that, on all datasets and metrics, comparisons significantly enhance the predictions of both label types. Compared to training on absolute labels alone (α = 1.0), combining absolute and comparison labels (α ∈ (0,1)) or training only on comparison labels (α = 0) considerably improves absolute label prediction, by 8% – 35% AUC, 14% – 25% Acc., 12% – 40% F1, and 14% – 55% PRAUC across all datasets. For all of these improvements over training with α = 1.0, we compute the p-value of Welch T-test. In all cases, p-values are above 0.99. Interestingly, in several cases indicated on Table 2a by *, comparisons lead to better predictions of absolute labels even when the latter are ignored during training. This indicates the relative advantage of incorporating comparison labels into training. Adding absolute labels to comparisons also improves comparison prediction performance, albeit modestly (by at most 6% AUC/0.923 p-value, 5% Acc./0.877 p-value, 3% F1/0.793 p-value, and 6% PRAUC/0.913 p-value).

Our performance over the ROP dataset is especially striking. Recall that its training set contains only 80 images and its test set contains 5561 images, unseen during training. Despite this extreme imbalance, by incorporating comparisons into training, we successfully train and optimize a neural network involving 5,974,577 parameters, on 80 images (60 for training, 20 for validation)! Specifically, our combined model attains an absolute label prediction performance of 0.914 AUC, 0.883 Acc., 0.64 F1, and 0.731 PRAUC. Training only on absolute labels, i.e. α = 1.0, can only achieve 0.835 AUC, 0.705 Acc., 0.517 F1, and 0.177 PRAUC (c.f. ROP rows in Table 2a). This further attests to the fact that comparisons are more informative than absolute labels and that our methodology is able to exploit this information even in the presence of a small dataset.

Our architecture is robust in the selection of a metric to optimize over the validation set. In Appendix E (c.f. Tables E.4 to E.11), we show that optimizing hyperparameters w.r.t. a metric (e.g., AUC) leads to good performance w.r.t. other metrics (e.g., Acc., and F1). Furthermore, for all metrics and datasets, comparison loss functions (L_c) induced by Bradley-Terry and Thurstone models (given in Eq. (5) and Eq. (6)) outperform cross-entropy and hinge losses. This further motivates the use of these linear comparison models for learning relative orders, compared to standard loss functions such as cross-entropy and hinge.

4.2. Comparison with Competing Methods

In Tables 3a and 3b, we compare our combined neural network (CNN) with Sun et al. (2017)’s method (Norm), logistic regression (Log.), a support vector machine (SVM), and a soft voting ensemble method (Ensemble) linearly combining the predictions of logistic regression and SVM. We evaluate our CNN model both for h_a = h_c, and the special case where h_a ≠ h_c (CNN (h_a = h_c)), combining cross-entropy as L_a and Bradley-Terry loss as L_c. For each a, we optimize the hyperparameters on the validation set following the procedure explained in Experiment Setup. For the best performing α ∈ [0, 1], we evaluate AUC, Acc., and F1 on the test set w.r.t. both D_a and D_c. We demonstrate that learning through the norm comparison loss (Norm) instead of maximum likelihood based on Bradley-Terry or Thurstone models (given in Eq. (5) and (6)) is not suitable for our architecture. Moreover, our method outperforms logistic, SVM, and ensemble methods, particularly in absolute label prediction, by 2% – 13% AUC, 5% – 13% Acc., and 3% – 15% F1 across all datasets. These improvements correspond to 0.772 – 1.0 p-value for AUC, 0.91 – 1.0 p-value for Acc., and 0.8 – 1.0 p-value for F1 under Welch T-test. Our method also outperforms other methods on comparison label prediction, specifically on ROP, by 11% AUC, 12% Acc., and 13% F1. For all of these improvements, p-values are 1.0.

Note that the performance of our architecture is robust to the design choice between single task learning with h_a = h_c vs. generic joint learning with h_a ≠ h_c. More precisely, our model balances the variance introduced by allowing h_a and h_c to differ by improving the bias.