A new classification strategy for human activity recognition using cost sensitive support vector machines for imbalanced data

2.1 Problem formulation and notations

First the raw sensors data are divided in time slices of constant interval Δt. At time t, sensor i is assigned a binary value one (Inline Equation 1) to indicate that the sensor i has been activated at least once in the time interval [t t+Δt], otherwise (Inline Equation 2). Given a set of N sensors, the observation at time t is given by the binary vector (Inline Equation 3). An activity at time slice t is given by y _t with (Inline Equation 4), where K stands for the total number of daily living activities. Examples of such activities include Leaving, Toileting, Showering, Sleeping, Drinking, Preparing Dinner, etc. The recognition task consists of finding a mapping between a sequence of observations (Inline Equation 5) and a sequence of labels (Inline Equation 6) for a total of T time steps. For sake of representation, we shall also use notations x and y to refer to x _1:T and y _1:T .

2.2 HMM

The HMM is a classic way of a probabilistic modeling of a sequential process, consisting of a hidden variable y and an observable variable x at each time step. In our case the observable variable is the vector of sensor readings and the hidden variable is the activity to recognize. The model endorses two commonly dependency assumptions:

• The observable variable at time t, namely, x _t , depends only on the hidden variable y _t , where the hidden variable corresponds in this case to the activity performed, while the observable variable represents the binary sensor reading.

• The hidden variable at time t, namely, y _t , depends only on the previous hidden variable y _t-1 (Markov assumption, Chen et al., 2005).

With these assumptions, the joint probability of the hidden and observable variables p(y_1:T, x_1:T), can be factorized as:

(Equation 7)

The model (1) is fully determined through the knowledge of the initial state distribution p(y ₁), the transition distribution p(y _t |y _t-1 ) representing the probability of passing from one state to the next one; and the observation distribution p(x _t |y _t ), which is computed by assuming that each sensor reading is modeled as an independent Bernouli distribution. Finally, the parameters associated to the above are learned from training data using maximum likelihood approach. Consequently, given a new sequence of observation x_1:T, the best sequence of activity that suits that sequence of observation will be the one that maximizes the joint probability (1). This can also be found more efficiently using Viterbi like algorithm (Chen et al., 2005).

2.3 CRF

In CRF, the nodes corresponding to observable and variables are connected via indirected graph, so that unlike HMM, conditional probability can no longer be used to represent the interaction between nodes, but rather potentials were used. Besides in the same spirit as (Kasteren et al., 2008), one assumes an exponential model for the conditional probability of the entire sequence of labels y_1:T given an input observation sequence x_1:T. CRF is defined by a weighted sum of K feature functions f _i that will return either 0 or 1, indicating whether the underlying feature will be accounted for depending on the values of the input variables. Each feature function is assigned a weight λ _i that quantifies the overall contribution of the feature, which also corresponds to the actual potential value. These weights are the parameters we want to find when learning the model. Unlike HMM, where the parameters are learned by maximizing the joint probability distribution, CRF model parameters are learned using an iterative gradient method by maximizing the conditional probability distribution defined as (Kasteren et al., 2008):

(Equation 8)

One of the main consequences of this choice is that while learning the parameters of a CRF we avoid modeling the distribution of the observations, p(x _1:T ). As a result, we can only use CRF to perform inference (and not to generate data), which is a characteristic of the discriminative models. To find the label y for new observed features, we take the maximum of the conditional probability as in (3). Such inferences can also be effectively performed using altered version of Viterbi's algorithm:

(Equation 9)

3.1 Introduction

Despite its popularity in machine learning, SVM technique has not been extensively used in activity recognition studies as pointed out in Banos et al. (2012), Chathuramali and Rodrigo (2012), Palaniappan et al. (2012). However, bearing in mind the remarkable accuracy rates obtained in other contexts, this would suggest possible success in activity recognition as well. Strictly speaking the few published literature using SVM for activity recognition rather relies mainly on use of standard SVM, which is then extended to multi-class using one-versus-all like approach or alternative method. Typically, SVM maximizes a margin in a hyperplane separating classes. Nevertheless, it is overwhelmed by the majority class instances in the case of imbalanced data sets. CS-SVM (Akbani et al., 2004) has been suggested as a candidate solution for such purpose, where the use of different error costs for positive and negative classes, the underlying hyperplane could be pushed away from positive instances. However, it is still an open debate how to accurately choose the cost functions. This motivates the current research in this paper. For this purpose, CS-SVM is employed. Second, a new method for eliciting the cost factors associated to CS-SVM is put forward. Third, more effort is put on the preprocessing in order to reduce the effect of minority class. For this purpose, SMOTE is employed. This technique creates new instances through “phantomtransduction,” where for each positive instance, its nearest positive neighbors were identified and new positive instances were created and placed randomly in between the instance and its neighbors. Fourth, one-versus-all approach is employed in order to extend the approach to multi-classes related to various activities that need to be recognition. Figure 1 highlights the overall architecture of the solution.

Especially, in the training phase, each feature vector, constituted of binary sensor vectors for the corresponding time slice, is associated to the underlying class of ADL. More specifically, two types of feature representation were employed. The former refers to the aforementioned change-point representation where one assigns value one to sensor whose sensor reading changes during the timeslice. The latter refers to last representation where last sensor that changed state continues to give 1 and changes to 0 when a different sensor changes state. A concatenation vector of both features is then constituted to form the overall feature vector. Next, imbalance class is then corrected using SMOTE strategy. This provides a refined and enhanced data set that will be inputed to CS-SVM, where the cost parameters are appropriately tuned using our suggested criterion. The outcome of trained CS-SVM will then be used to process a new observation during the testing phase where the associated ADL class will be predicted. Each feature vector in the test data is therefore classified into an estimated ADL class.

3.2 SMOTE

This approach effectively forces the decision region of the minority class in the training set. The SMOTE algorithm generates artificial data based on the feature space similarities between existing minority examples. Synthetic examples are introduced along the line segment between each minority class example and one of its k minority class (S _min) nearest neighbors. The k-nearest neighbors (k-NN) are defined as the k elements of subset (Inline Equation 10) with (Inline Equation 11) is one of the k-NN for (Inline Equation 12), whose Euclidian distance between (Inline Equation 13) and (Inline Equation 14) is the minority instance under consideration exhibits the smallest magnitude along the n-dimensions of feature space X. To create a synthetic sample, the k-nearest neighbors are randomly chosen, then multiply the corresponding feature vector difference with a random number (Inline Equation 15), and finally, add it to x _i:

(Equation 16)

3.3 C-SVM

SVM, developed by Vapnik (2000), is based on statistical learning theory and gains popularity because of many attractive characteristics and good performances. We utilize the current standard soft-margin C-SVM (Vapnik, 2000). SVM classifies data by determining a hyperplane into a higher dimensional space (feature space). For a two class problem, we assume that we have a training set (Inline Equation 17) where (Inline Equation 18) are the observations and y _i are class labels either 1 or −1, and given some kernel function K, SVM determines the optimal α _i for each x _i to maximize the margin between the hyperplane and the closest instance to it as seen in Figure 2. The class prediction for a new test instance x is given by (Bishop, 2006):

(Equation 19)

where α _i >0 are Lagrange multipliers and b is the translation factor of the hyperplane from the origin. The training samples where α _i >0 are called support vectors. While α _i and b are determined by minimizing the following primal optimization problem, which maximizes margin 2/K(w,w), see Figure 2, between the two classes and minimizes the amount of total misclassifications (training errors) (Inline Equation 20) simultaneously:

(Equation 21)

where w is a weight vector perpendicular to the hyperplane and φ(.) is a non-linear function which maps the input space into a feature space defined by (Inline Equation 22) that is kernel matrix of the input space. The penalty constant C represents the trade-off between the empirical error ξ and the margin. It corresponds to the cost of SVM. Exponential radial basis function kernel is quite popular: (Inline Equation 23) where σ is the width parameter. In our context, cross-validation technique were used to select the width parameter.

The regularization parameter C is used to control the trade-off between maximization of the margin width and minimizing the number of training error of nonseparable samples in order to avoid the problem of overfitting (Bishop, 2006). A small value for C will increase the number of training errors, while a large C will lead to a behavior similar to that of a hard-margin SVM. In practice both parameters σ and C are varied through a wide range of values and the optimal performance assessed using a cross-validation technique to verify performance using only training set (Taylor and Cristianini, 2004).

Although SVM often produce effective solutions for balanced data sets, they are sensitive to imbalanced training data sets and produces sub-optimal models because the constraint in (4) imposes equal total influence from the positive and negative support vectors. Wu and Chang (2003) pointed out that the imbalance in the training data ratio means that the positive instances may lie further away from the “ideal” boundary than the negative instances. As a result of this phenomenon, SVM learns a boundary that is too close to and skewed toward the positive instances. They hypothesized that as a result of this imbalance, the neighborhood of a test instance close to the boundary is more likely to be dominated by negative support vectors and hence the decision function is more likely to classify a boundary point negative.

3.4 CS-SVM

Because of the fact that in case of imbalanced data sets, the learned boundary is too close to the positive instances, there is a need to bias SVM in a way that will push the boundary away from the positive instances. For this purpose, Veropoulos et al. (1999) suggested using different error costs for the positive (C+) and negative (C−) classes.

In order to deal with imbalanced training data set, the SVM soft margin objective function is modified to assign two different penalty constraints C+ and C− for positive and negative classes, respectively, yielding the quadratic optimization below:

(Equation 24)

Especially, C+ is the higher misclassification cost of the positive class, while C− is the lower misclassification cost of the negative class. Optimization (7) allows us to handle imbalanced data set by the use of different error cost for the positive and negative classes such that the hyperplane could be pushed away from the positive instances. The SVM dual formulation gives the same Lagrangian as in the original soft-margin SVM in (6), but with different constraints on α _i (Lagrangian multipliers) as follows:

(Equation 25)

In the construction of cost sensitive SVM, the cost parameter plays an indispensable role. For the cost information, some authors (Thai-Nghe et al., 2010; Veropoulos et al., 1999; Abidine et al., 2013) have proposed adjusting different penalty parameters for different classes of data which effectively improves the low classification accuracy caused by imbalanced samples. For example, it is highly possible to achieve the high classification accuracy by simply classifying all samples as the class with majority samples (positive observations), therefore the minority class (negative observations) is the error training. Veropoulos et al. (1999) proposed to increase the misclassification cost associated with the minority class (i.e. C−>C+) to eliminate the imbalance effect, although no specific guidelines have been provided by the authors for detailed quantifications of these cost factors. An approach for this purpose is provided in the next subsection.

3.5 Cost sensitive criterion

Our proposed criterion advocates a simple intuitive heuristic to select the penalty constraints C+ and C−. For this purpose, one considers the number of samples m _i in the ith class from the training data. according to the proportion of that class. Following Veropoulos's reasoning where the cost C− associated to the smallest class must be large enough in order to improve the low classification accuracy caused by imbalanced samples, one shall set C+ to one, and determine C− as the proportion of majority samples m ₊ over minority ones m ₋; namely:

(Equation 26)

where [.] is integer part of the quantity under square bracket. Notice that it always holds that (Inline Equation 27) Therefore, (7) is translated into:

(Equation 28)

In this study, the algorithms are implemented in Matlab environment where the software package LIBSVM (Chang and Lin, 2001) was used to implement the multiclass SVM classifier algorithms. It uses the one-vs-one method (Vapnik, 2000).

4.1 Data sets

We used data gathered by Kasteren et al. (2008). The latter deployed a digital sensors in houses having different layouts and different number of sensors (Kasteren et al., 2011). Each sensor is attached to a wireless sensor network node. They attached these sensors to doors, cupboards, a refrigerator, and a toilet flush. Their data were divided in slices of constant length, 60 seconds, without overlapping. The sensors were installed in everyday objects such as drawers, refrigerators, containers to record activation/deactivation events (opening/closing events) as the subject carried out everyday activities. Their data was collected by a base station (BS) and annotated using a personal digital assistant PDA asking what activity she is performing every 15 minute, annotation was later corrected by inspection of the sensor data. Times where no activity is annotated are referred to as Idle (Tables I-III).

4.2 Setup and performance measures

We separate the data into a test and training set using a “leave one day out cross validation” cross-validation approach. This produces unbiased but high-variance error estimates. In this approach, each day of sensor readings is used for testing and the remaining days are used for training; this is repeated l times, with different training sets of size (l−1) and report the average performance measure. By this way, we get inferred labels for the whole data set by concatenating the resulting labels acquired for each test day. A vector of features was generated for each timeslice.

Sensors outputs are binary and represented in a feature space which is used by the model to recognize the activities performed. The vector contained one entry for each sensor, two-state sensors 0 or 1 are used and the features are the states of all sensors. We do not use the raw sensor data representation as observations; instead we use the “Change point” and “Last” representation described in Section 3.2, which have been shown to give much better results in activity recognition (Kasteren et al., 2008).

As the activity instances were imbalanced between classes. We evaluate the performance of our models using the F measure, which is calculated from the precision and recall scores. We are dealing with a multi-class classification problem and therefore define the notions of true positive (TP), false negatives (FN) and false positives (FP) for each class separately, where N is the total number of classes. With highly skewed data distribution, the overall accuracy metric at (11) is not sufficient any more. It does not take into account differences in the frequency of activities. These measures are calculated as follows:

(Equation 29)

(Equation 30)

(Equation 31)

(Equation 32)

in which [a=b] is a binary indicator giving 1 when true and 0 when false. m is the total number of samples.

4.3 Results

We compared the performance of the HMM, CRF, C-SVM, CS-SVM and Our strategy on the imbalanced data set of the house A₍₁₎ in which minority class are all classes that appear at most 1 percent of the time, while others are the majority classes that typically, have a longer duration (e.g. Leaving and Sleeping). Recall that Our strategy integrates the use of SMOTE, CS-SVM using the heuristic pointed out in 3.4 to choose the cost sensitive parameters, while in standard C-SVM and CS-SVM, one only relies on the default parameters as implemented in LIBSVM package for C-SVM without SMOTE and heuristic choice of the cost parameters. Standard CS-SVM employs our adaptive cost parameter but without SMOTE.

In our experiments, for Our strategy, the same C-SVM hyper-parameters (σ _opt , C _opt )=(1, 5) have been optimized for all (Inline Equation 33) decision functions in the range (0.1-2) and (0.1-10000), respectively, to maximize the class accuracy of leave-one-day-out cross-validation techniques. The minority class examples were over-sampled using k=4 nearest neighbors for SMOTE. Then, for CS-SVM, we searched the cost sensitive (Inline Equation 34) adapted respectively for different classes for house A₍₁₎ by using our criterion (Taylor and Cristianini, 2004). Notice that the minority class requires a large value of C compared to the majority class. This fact induces a classifier's bias in order to give more importance to the minority ones.

The summary of the accuracy and the class accuracy obtained with the concatenation matrix of “Changepoint+Last” for HMM, CRF, C-SVM, CS-SVM and CS-SVM+SMOTE are presented in Table IV. This shows that Our strategy performs better in terms of F measure, while CRF and C-SVM methods perform better in terms of accuracy.

We report in Figure 3 the classification results in terms of accuracy measure for each class with HMM, CRF, C-SVM, CS-SVM and Our strategy. One notices that HMM and Our strategy perform better for the minority activities Toileting, Showering, Breakfast, Dinner and Drink comparatively to other methods, while CRF and C-SVM perform marginally better only for the majority activities (other classes). CS-SVM method is shown to outperform CRF and C-SVM in classifying the minority activities. Nevertheless, the result also shows that Our Strategy under perform the Idle class activity. This is partly explained by the less variations occurring in the Idle class, which makes the use of cost sensitive like approach not appropriate.

In order to find out which activities are relatively harder to be recognized, we analyzed the confusion matrices of the better classifiers HMM and Our strategy in Tables V and VI, respectively. In Table V, the activities Leaving, Toileting, Showering and Sleeping give the highest accuracy. Activities Idle and Breakfast sound ill-handled by HMM method. Most confusion takes place in the Idle activity and the three kitchen activities Breakfast, Dinner and Drink.

We can see in Table VI that with Our strategy, the activity Idle perform worst, but the activities Leaving, Toileting, Showering and Sleeping are much better recognized. Strictly speaking, it is acknowledged that the kitchen activities are in general hard to recognize but Our strategy exhibits good performance results as compared to others methods.

Finally, we presented a way of compactly presenting all results in a single, allowing a quick comparison between HMM, CRF, C-SVM, CS-SVM and Our strategy performed using the same representation “Changepoint” with the four real world data sets TK26M₂, TK28 M, TK57 M, Tap80F. We utilize the leave-one-day-out cross-validation technique for the selection of width parameter, where we identified σ _opt =1, σ _opt =1, σ _opt =2 and σ _opt =2 for these data sets, respectively. Our results give us early experimental evidence that our proposed strategy works better for model classification; it consistently outperforms the other methods in terms of the F measure for all data sets.

The above results show that Our strategy performs better than the other methods in terms of Precision and F measure for all Houses. These results show the effect of taking class imbalance into account in the evaluation metric, while HMM and CRF managed to correctly recognize a higher number of timeslices.

4.4 Discussions

Using experiments on three large real world data sets, we showed the class accuracy obtained with house (TK57 M) is lower compared to others houses for all recognition methods for all recognition methods because the house (TK57 M) includes more classes than other houses.

In terms of comparing the performance of the employed classifiers (HMM, CRF, C-SVM, CS-SVM and Our strategy) for the house A₍₁₎, one shall notice the following. First HMM is trained by splitting the training data in which a separate model (Inline Equation 35) is learned for each class, as parameters are learned for each class separately. That partly explains why HMM performs better for the minority activities. The CRF model does not model each action class individually, but rather uses a single model for all classes. As a result classes that are dominantly present in the data have a bigger weight after CRF optimization. This partly results in better performance for the majority activities (“Idle,” “Leaving” and “Sleeping”). On the other hand, regular multiclass C-SVM trains several binary classifiers to differentiate the classes according to the class labels and optimize a single parameter C for all classes. As the weighting is performed in similar way regardless the type of class in C-SVM formulation, this provides an edge to classification of majority classes. While the CS-SVM with tuned heuristic parameter for each class, although it may seem naive at first glance, but it provides higher performance accuracy than C-SVM or CRF methods. Besides, introducing the extra step of the SMOTE) with the cost sensitive support vector machine CS-SVM together with adaptive cost parameter results in enhanced performances that outperform all other methods. This shows the importance of the dealing with imbalanced data set through SMOTE approach in the overall classification strategy, which provides more robustness in classifying the minority class (es). Strictly speaking, from the dependency perspective, one shall mention that the HMM approach explicitly accounts for the dependency between the hidden and observable variables as pointed out in Section 2.1, although such reasoning is missing in case of SVM and its various variants, the use of one-versus-one method to extend the pairwise comparison of SVM to multiple classes partially allows the system to account for such dependency.

The recognition of the three kitchen activities “Breakfast” “Dinner” and “Drink” is lower compared to the others activities for all methods. On the other hand, the recognition of the different tasks as demonstrated by the analysis of the confusion matrix shows the difficulty to recognize the “Idle” class, although from a regulator viewpoint, the recognition of such activity is not so important. The data set showed that this activity is by far the most common, and makes the data set particularly hard because some activities were missed during annotation where training data of the idle class might actually belong to one of the others. It might therefore be useful to assign smaller weigh to such activity. The kitchen activities consisted mainly of food related tasks are acknowledged for their difficulty to be recognized because most of the instances of these activities were performed in the same location (kitchen) using the same set of sensors. For example, “Toileting” and “Showering” are more separable because they are in two different rooms, which make the information from the door sensors enough to separate the two activities. Therefore the location of the sensors is of great importance for the performance of the recognition system.