Mitigating the Impact of Malware Evolution on API Sequence-based Windows Malware Detectors

Graph embedding [27, 28, 29] can represent each API in the knowledge graph as a feature vector. Moreover, The semantically similar APIs are represented closer in the feature space. To achieve this, we employed an existing algorithm called TransE [27] and integrated it into our graph embedding problem. Specifically, suppose there is a relation $R$ that connects the entity $E_{a}$ to the entity $E_{b}$, and they are represented by three vectors: $V_{R}$, $V_{a}$, and $V_{b}$. The core idea of the TransE algorithm is to iteratively adjust these three vectors so that the $V_{a}+V_{R}$ is as close as possible to $V_{b}$. As a result, APIs with similar semantics will have similar vector representations because they will be related to the same other entities. As a result, the entities with similar semantics will have similar vector representations in the vector space. For example, the two API entities RegOpenKeyEx (denoted as $V_{a1}$) and RegOpenKeyTransacted (denoted as $V_{a2}$) have the same prototype RegOpenKey (denoted as $V_{b}$). Thus, there are extend_from relations (denoted as $V_{R}$) connect $V_{a1}$ and $V_{a2}$ to the $V_{b}$. TransE adjusts these vectors so that the $V_{a1}+V_{R}$ and the $V_{a2}+V_{R}$ are as close as possible to $V_{b}$. Therefore, the $V_{a1}$ and $V_{a2}$ are represented more similar.

After graph embedding, each API entity in the API knowledge graph is represented as a fixed-length semantic vector. In other words, for the input of the raw API sequence, the API name of each API can be mapped to the corresponding semantic vector using the knowledge graph. Furthermore, when malware undergoes API replacement during its evolution, even though the API names before and after evolution may differ, if API functions are similar, then their semantic vectors will be very close.

Figure 5 shows an example hooked API whose name is NtCreateFile. For the first argument, its type is integer, and the value is 2. For the second argument, its type is string and the value is “C:$\backslash\backslash$User$\backslash\backslash$Administrator$\backslash\backslash$AppData$\backslash\backslash$...” which is a accessed file path.

To extract system resources, we consider 5 types of string arguments: file paths, dynamic link library file names (DLLs), registry keys, URLs, and IP addresses. These types of resources are accessed frequently and can be extracted directly from the API sequence. For each API in the API sequence, we use regular expression matching to identify its argument values and extract arguments belonging to the 5 types of resources. Specifically, we use “C:$\backslash\backslash$” to identify a file path. The DLLs are arguments ending with “.dll”. The registry keys often start with “HKEY_”. URLs often start with “http”. IPs are those arguments with four numbers (range from 0 to 255) separated by dots. These extracted string arguments are then embedded as feature vectors.

Intuitively, the strings sharing a large number of substrings have very similar meanings. Thus, for each extracted argument, we first parse the whole string into several substrings to capture the hierarchical information. For example, for a path like “C:$\backslash\backslash$f_a$\backslash\backslash$f_b”, three substrings are generated by splitting based on “$\backslash\backslash$”, namely “C:”, “C:$\backslash\backslash$f_a”, “C:$\backslash\backslash$f_a$\backslash\backslash$f_b”. The DLLs and registry keys can also be parsed like the file paths. The Urls and IP address can be parsed by splitting based on “.”. For example, for a url “https://sample.sec.org/”, we only generate substrings from its hostname, and the following substrings will be generated “org”, “sec.org”, and “sample.sec.org”.

We use feature hashing [30] to represent each extracted argument as a fixed-length feature vector. Let $S$ denotes an substring set of the extracted string argument, and $s_{j}\in S$ denotes a substring. Let $N$ denotes the number of bins. The value of the $i$-th bin is calculated by

where $h$ is a hash function that maps the $s_{j}$ to a natural number $n_{1}\in\{1,2,...,N\}$ as the bin index. $\xi$ is another hash function that maps the $s_{j}$ to $n_{2}\in\{\pm 1\}$. After feature hashing, the extracted argument $S$ is represented as a feature vector $[\phi_{1}^{h,\xi}(S),\phi_{2}^{h,\xi}(S),...,\phi_{N}^{h,\xi}(S)]\in\mathbb{R}^{N}$. For example, for the url “https://sample.sec.org/”, if $N$ = 8 and $S$ = {“org”, “sec.org”, “sample.sec.org”}, then $s_{1}$ = “org”, $s_{2}$ = “sec.org”, $s_{3}$ = “sample.sec.org”. After hash mapping, $h\left(s_{1}\right)$ = 1 (i.e., bin index 1), $\xi\left(s_{1}\right)$ = 1, $h\left(s_{2}\right)$ = 2 (i.e., bin index 2), $\xi\left(s_{2}\right)$ = -1, $h\left(s_{3}\right)$ = 4 (i.e., bin index 4), $\xi\left(s_{3}\right)$ = 1. Thus, $\phi_{1}^{h,\xi}(S)$ = 1, $\phi_{2}^{h,\xi}(S)$ = -1, $\phi_{4}^{h,\xi}(S)$ = 1. The feature vector of the extracted argument is $[1,-1,0,1,0,0,0,0]$.

The arguments with a large number of shared substrings will have the similar set $S$ and will be represented very similar. In this way, if the malware accesses similar system resources before and after evolution, then their feature vectors will be very close.

At this point, the API sequence embedding enhancement is complete. When a raw API sequence is input to the model, for each API in the sequence, its API name is mapped to the API knowledge graph and represented as an API name semantic vector. Each argument of the API is checked to identify if it is an accessed resource. If so, it is hashed and represented as an API argument semantic vector. These two vectors are concatenated as the API’s feature vector. Finally, the embedded API sequence (i.e., the API feature vector sequence) is input to the encoder of the detection model.

We design a contrastive learning strategy to enhances the encoder’s ability to capture fine-grained similarities and differences among API sequences and improve performance in detecting evolved malicious samples.

Let $x$ be an embedded API sequence. The ground truth binary label is $y\in\{0,1\}$, where $y=0$ indicates a benign sample, and $y=1$ indicates a malicious sample. Let $y\prime$ be the ground truth multi-class family label. When $y\prime=0$, the label is benign, but otherwise, it is a malware family label. For the detection model $f$, after API sequence embedding, the embedded sample $x$ is first input to an encoder $en$ (e.g., LSTM, Text-CNN, etc.), which outputs the representation of the input sample in the latent feature space $z=en(x)$. Then, a classifier $g$ takes the encoder output and predicts the binary label $f(x)=g(z)=g(en(x))$.

Let $f(x)=g(en(x))$ be the output of the softmax layer for class $y=1$ (i.e., malware) and the benign softmax output is $1-f(x)$. If $f(x)\geq 0.5$, the predicted binary label $\hat{y}$ is $\hat{y}=1$, and otherwise, $\hat{y}=0$.

In general, the training loss of a regular model is defined as computing a classification loss between $f(x)$ and $y$. However, in this paper, we define the training loss is the sum of a contrastive loss and a classification loss, and the detection models are trained end-to-end with this loss. Specifically,

where $\mathcal{L}_{cla}$ is the classification loss and $\mathcal{L}_{con}$ is the contrastive loss for enhancing the encoder (defined below). As a common heuristic approach, we use a hyperparameter $\lambda$ to balance the two terms $\mathcal{L}_{con}$ and $\lambda\mathcal{L}_{cla}$, so that they have a similar mean, thus the overall loss is not overwhelmed by just one term. The classification loss $\mathcal{L}_{cla}$ uses the binary cross entropy loss:

where $i$ ranges over indices of samples in the batch.

The contrastive loss $\mathcal{L}_{con}$ computes a similarity over positive and negative pairs of samples in a batch. It tends to maximize the similarity between positive pairs and minimize the similarity between negative pairs. We design a contrastive learning strategy that encourages the encoder $en$ to satisfy the following two properties:

• •

  Positive pairs: If $x_{1}$, $x_{2}$ are two benign samples, or two malicious samples in the same malware family, then they are positive pairs, and their representations should be similar: i.e., $\left\|en\left(x_{1}\right)-en\left(x_{2}\right)\right\|_{2}$ should be as small as possible.

• •

  Negative pairs: If one of $x_{1}$, $x_{2}$ is malicious and the other is benign, then they are negative pairs, and their representations should be dissimilar: i.e., $\left\|en\left(x_{1}\right)-en\left(x_{2}\right)\right\|_{2}$ should be as large as possible.

Specifically, for a batch of size $2N$, the first $N$ samples in the batch are sampled randomly, denoted as $\{x_{k},y_{k},y\prime_{k}\}_{k=1...N}$. Then, we randomly select $N$ more samples which have the same label distribution as the first $N$ samples, i.e., $\{x_{k+N},y_{k+N},y\prime_{k+N}\}_{k=1...N}$ are chosen so that $y_{k}=y_{k+N}$ and $y\prime_{k}=y\prime_{k+N}$. To capture the positive and negative samples paired with $x_{i}$, the following sets are defined in the batch:

• •

  The positive sample set of $x_{i}$. Both samples are benign or both samples are malicious and in the same malware family:

  $Pos\left(x_{i}\right)\equiv\left\{x_{j}\mid y_{j}=y_{i},y_{i}=1\Longrightarrow y_{j}^{\prime}=y_{i}^{\prime},j\neq i\right\}$

• •

  The negative sample set of $x_{i}$. One sample is benign and the other is malicious:

  $Neg\left(x_{i}\right)\equiv\left\{x_{j}\mid y_{j}\neq y_{i},j\neq i\right\}$

Intuitively, $Pos\left(x_{i}\right)$ contains samples that are considered similar to $x_{i}$, and $Neg\left(x_{i}\right)$ contains dissimilar samples to $x_{i}$.

Let $d_{ij}$ denote the euclidean distance between two arbitrary samples $x_{i}$ and $x_{j}$ in the feature space: $d_{ij}=\left\|en\left(x_{1}\right)-en\left(x_{2}\right)\right\|_{2}$. Let $m$ denote a fixed margin (a hyperparameter). The contrastive loss is defined as:

The contrastive loss has two terms. The first term asks positive pairs from $Pos\left(x_{i}\right)$ to be close together. These pairs are (benign, benign) or (malicious, malicious) pairs with the same malware family. In this way, the encoder will pay more attention to the similarities among samples in the same class. The evolved samples that retain API fragments similar to past samples will be represented as closer to past samples in the latent space. The second term aims to separate benign and malicious samples from each other, hopefully at least $m$ apart from each other. Thus, the encoder will focus on capturing the differences between benign and malicious samples, and prevent the classifier from misclassifying evolved malware as benign ones.

At this point, the encoder enhancement is complete. A contrastive encoder is constructed using our contrastive learning strategy, without altering the structure of the original model. Finally, The enhanced models are trained end-to-end with the loss $\mathcal{L}$.

In this paper, we focus on malware of the Windows portable executable (PE) file which is the most popular malware file format. Our dataset, spanning over five years, contains 76,473 Windows PE files, i.e., 39,349 malicious and 37,124 benign as shown in Table IV. Specifically, The malicious software is obtained from the VirusShare website [31] and using a daily downloading script. The benign software is obtained from popular free software sources, including PortableApps [32], Softonic [33], SourceForge [34], and CNET [35].

To get reliable labels for these samples, we rely on VirusTotal[36] to determine whether a sample is benign or malicious. VirusTotal uses more than 60 anti-virus (AV) engines to vote whether the submitted sample is malicious or benign. In this paper, samples are labeled as malware when at least 10 AV engines report them as malicious, while samples are labeled as benign when no AV reports them as malicious. Note that according to a recent study [37] on measuring the labeling effectiveness of malware samples, this strategy is reasonable and stable. We consider samples up to Dec 2021 because following a previous work [38], the malware labels become stable after about one year, thus choosing Dec 2021 as the finishing time ensures good ground-truth confidence in objects labeled as malware.

Also, we leverage VirusTotal to get the exact appearing time for each sample and make sure that temporal consistency [13] is satisfied at the month level during the testing. Specifically, temporal consistency ensures that training samples should be strictly temporally precedent to testing ones, and all testing samples must come from the same period during each testing to eliminate time bias.

After data collection, the Cuckoo Sandbox [39] is used to run the PE files and gather execution logs. Cuckoo sandbox has been widely used in prior works [7, 8, 9, 10]. It executes each PE file inside virtual machines and uses API hooks to monitor the Windows APIs to form a raw API sequence. In our system, dozens of virtual machines are maintained on the Cuckoo server which is installed with Ubuntu 16.04 LTS. All the virtual machines are installed with a 64-bit Windows 10 system and several necessary drivers to ensure the successful execution of the PE samples in the dataset. The snapshot feature of the virtual machine is leveraged to roll it back after execution to ensure the uniformity of the software running environment. Besides, Cuckoo simulates some user actions (such as clicking a button, typing some texts, etc.) to trigger malicious behavior of malware. In this paper, we set the maximum running time of each sample to 5 minutes. That is to say, the sandbox process completes when the uploaded sample ends itself or runs to 5 minutes. After a PE file is uploaded, Cuckoo server begins to call a free client to execute the file and record the API calls automatically. When the process completes, Cuckoo server will generate a sandbox report about this uploaded file and the raw API sequence can be extracted from this report.

We employ two representative DNNs, i.e., LSTM and Text-CNN, to build the malware detection models. These two models learn the sequence features from API sequences and have been proved to be effective in malware detection. In fact, many exsiting studies have already using these two models or their variants or combinations as the encoder [3, 4, 7, 5, 6]. The details of two DNN models are illustrated as follows:

In this section, we measure the performance of existing malware detection models with and without the help of MME to understand the ability of MME in mitigating model degradation.

The purpose of this experiment is to evaluate how many human efforts MME can save while maintaining a high performance malware detection models.

Specifically, the comparison includes two aspects. On the one hand, we compare the amount of human efforts needed for active learning in maintaining both the regular and the enhanced models. On the other hand, we compare the model performance improvment given a fixed level of human effort.

In this experiment, we want to measure the individual effects of the two parts of the MME framework (i.e., embedding enhancement and encoder enhancement) on enhancing the regular model.

We observed that the malware evolution can disturb the stability of the original feature space, leading to a decline in model performance. In this experiment, we want to measure the stability of the feature space concerning the evolution of malware from the same family to show that the MME-enhanced model can capture the semantic similarity between the original and evolved of malware.

Dynamic malware detection executes the software in a secured virtual environment and monitors its run-time behavior. A running software calls many system APIs, which characterize software behaviors including network access, file creation and modification, etc. These API calls form an API call sequence which has become a widely used data source for malware detection and classification [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 45, 46, 47].

Inspired by deep learning-based squence analysis, many researchers apply some DL models like convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to learn features of the API call sequences. Kolosnjaji et al. [3] use the API sequence as input. Their approach stacks a CNN that uses a 3-sized filter to represent 3 consecutive APIs (like the 3-gram approach). After the CNN, the LSTM is applied to handle the time-series sequence. Agrawal et al. [4] input the API names and the n-gram of the string arguments into several stacked LSTMs. Zhang et al. [10] build a feature engineering about the API names and arguments and then design a deep learning model including gate-CNNs and Bi-LSTM as the malware detector. Catak et al. [7] input API sequences into LSTMs to detect and classify malware. Li et al. [5] combine the Text-CNN with Bi-LSTM to analyze the API sequences and detect malware. Similarly, Chen et al. [6] use the Text-CNN and Bi-LSTM as the baseline models for API sequence-based malware detection.

Obviously, too many works have already using Text-CNN and LSTM models or their variants or combinations as the encoder of the detection models. However, how to perform feature representation so that the model can accurately understand the semantic information in API sequences and thus understand the software behavior remains a challenging issue. Our framework MME enhance the API sequence embedding and can better represent each API as a semantic feature vector, as shown in §III and §IV.

Deep learning techniques were originally designed for stationary environments in which the training and test sets are assumed to be generated from the same statistical distribution. However, this assumption is not valid in the malware domain. Malware samples, including various families, evolve over time due to changes resulting from adding capabilities, fixing bugs, porting to new platforms, etc. Thus, malware detectors are deployed in dynamic environments, where malware variants keep evolving, causing the performance to deteriorate significantly over time. This is known as the problem of model aging or concept drift [13].

There are mainly two methods to address model aging caused by the evolution of malware. The first is to retrain and update detection models with newly labeled samples, or reject drift samples until they can be expertly analyzed. For example, in Android malware detection, DroidEvolver [15] utilizes online learning and pseudo-labels to self-update the detection model. However, the accumulation of pseudo-label errors may lead to model self-poisoning which have catastrophic effects on performance. Some studies focus on detecting drift samples that deviate from existing classes from a large number of test samples and update models using periodical retraining [16, 17]. However, labeling samples and retraining the model still requires a lot of expert knowledge and computing resources. More importantly, it is also difficult to determine when the model should be retrained. Delayed retraining can leave the outdated model vulnerable to evolved malware.

The second method is to deliberately consider the issue of model aging during the process of model design and feature space optimization. Researchers represent features to be more robust against temporal bias and reduce the impact of malware evolution. APIGraph [20] is proposed to enhance state-of-the-art malware detectors with the similarity information among evolved malware in terms of equivalent or similar API usages. It constructs an API knowledge graph based on the API documentation, and use graph embedding and the K-means algorithm to cluster APIs with similar semantics. Similarity, SDAC [21] calculates APIs’ contributions to malware detection and assign APIs to feature vectors. Then, SDAC clusters all APIs based on their semantic distances to create a feature set in the training phase, and extends the feature set to include all new APIs in the detecting phase. However, the detectors above mainly focus on API occurrence or API frequency, which is difficult to be applied to dynamic malware detection based on API sequence analysis. The MME proposed in this paper further extends API sequence embedding and constructs a contrastive encoder to address the model aging issue in API sequence-based malware detection.

The API sequence is a popular type of feature widely adopted by dynamic malware detectors [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], mainly because API sequences are essential in understanding malware behaviors. In our experiments, we validate the effectiveness of MME by enhancing the LSTM and Text-CNN models. There indeed are many other DNN models such as Bi-LSTM, Gate-CNN, or even more variant models commonly used for learning API sequences [4, 5, 6, 10, 47, 48]. Although these models are not individually validated in this paper, they also require the API sequence embedding and encoder training process. We believe that the MME approach can also be applied to these models and achieve similar enhancement effects.

In reality, there are a few instances where the attack methods of certain malware are so advanced that their behavior bears very little similarity to previous malware. For such newly emerged malware, the performance of MME may weaken. In such cases, drift sample detection methods [16, 17] can be used to identify such samples, as they deviate significantly from the original data distribution. Then, such overly advanced malware can be collected and labeled for updating models. We believe that utilizing the MME enhanced models in conjunction with periodic drift sample detection can better address the continuous evolution of malicious software.

To obtain the raw API sequences, sandboxes are widely used to execute malware inside virtual machines and monitor APIs with API hooking techniques. Sandbox evasion refers to techniques employed by malware to avoid detection or analysis within a sandbox environment [49]. For example, when malware detects that it is running within a sandbox environment, it can refrain from executing any malicious operations, or even disguise itself as a legitimate application to exhibit benign behavior. The key to combating such “environment-aware” malware is to optimize the sandbox environment to closely resemble a real system environment. In the sandbox used for experiments, we simulate some user actions (such as clicking a button, typing some texts, etc.) to trigger malware real behaviors. We further utilize the statistical model proposed by Miramirkhani et al. [50] to optimize and fine-tune the sandbox, making it even closer to a real system environment. We believe these operations can help the sandbox capture the real API sequences of malware. In future works, solutions [51, 52, 53, 54] focusing on detecting sandbox evasion can be used to further optimize this issue.