Multi-perspective feedback-attention coupling model for continuous-time dynamic graphs

We describe continuous-time dynamic graphs as a sequence of interaction events occurring in chronological order. Formally, it can be represented as $G\left(T\right) = \left(V\left(T\right),E\left(T\right)\right) = \left[x\left(t_0\right),\;x\left(t_1\right),\;x\left(t_2\right),\ \gt \ldots\right]$. Given time t, it can be expressed as $G\left(t\right) = \left(V\left(t\right),E\left(t\right)\right) = \left[x\left(t_0\right),x\left(t_1\right),\ldots,\\gt x\left(t\right)\right]$, where $V\left(t\right)$ and $E\left(t\right)$ are the sets of nodes and edges at time t respectively. $x\left(t\right) = \left(v_i,v_j,e_{i,j}\left(t\right)\right)$ denotes an event occurring at time t, where v_i and v_j represent nodes, and i and j are node indices. $e_{i,j}(t)$ represents the temporal edge feature.

The goal of continuous-time dynamic graph embedding learning is to acquire node feature representations and utilize them for downstream tasks such as dynamic node classification or future link prediction. Given a dynamic graph G(t) at time t, its embedding learning aims to seek a mapping function $f:V\longrightarrow R^d$, where d is the dimension of node representations. After the mapping function f, G(t) can be represented as $Z\left(t\right) = \left(z_0\left(t\right),\ z_1\left(t\right),\ldots\right)$.

We analyze the evolution of dynamic graphs from two perspectives: original and evolving. The evolving perspective is employed to capture the current state of nodes, while the original perspective is utilized to capture the original state of nodes (these two states are shown in figure 2). Specifically, given a node v_i at time t, its historical interaction at time t₁($t_1\lt t$) is denoted as $(v_i,v_1,e_{i,1}\left(t_1\right))$, where the states of nodes v_i and v₁ at t₁ are represented as $r_i\left(t_1\right)$ and $r_1\left(t_1\right)$, respectively. During the interval from t₁ to t, due to potential changes in nodes v_i or v₁ (e.g. interactions with other nodes), the states of nodes v_i and v₁ evolve into ${\ h}_i\left(t\right)$ and ${\ h}_1\left(t\right)$, respectively. We refer to $r_i\left(t_1\right)$ as the original state of node v_i and $h_i\left(t\right)$ as the current state of node v_i. The original state retains the essence of the interaction events, while the current state records the real-time changes of the nodes.

Zoom In Zoom Out Reset image size

In this section, we design a vector with an initial value of zero for each node, aiming to preserve the current state information of the nodes. The current state of node v_i at time t is represented as $h_i\left(t\right)$, which compresses and encodes all the events that occurred before time t for that node. This encoding method requires only a small number of temporal neighbors to capture the long-term dependencies of the node. During the evolution of the network, whenever an event occurs for node v_i, its current state is updated. This process involves incorporating the event $x\left(t^-\right)$, occurring at node v_i at time $t^-$, into $h_i\left(t^-\right)$ to derive $h_i\left(t\right)$,

$$\begin{align} X_{i,j}\left(t^-\right)&= W_{evol}\left[z_i\left(t^-\right)||e_{i,j}\left(t^-\right)||z_j\left(t^-\right)||\Phi\left(t-t^-\right)\right],\end{align} \tag{ 1 }$$

$$\begin{align} h_i\left(t\right) &= update\left(X_{i,j}\left(t^-\right),\;h_i\left(t^-\right)\right),\end{align} \tag{ 2 }$$

where W_evol is a trainable parameter, $\Phi\left(\cdot\right)$ represents a generic time encoding, and $||$ denotes the concatenation operator. $t^-$ is the most recent interaction time of node v_i. $z_i\left(t^-\right)$ and $z_j\left(t^-\right)$ represent the embedding representations of nodes v_i and v_j at time $t^-$, respectively. $update()$ denotes a learnable update function, which can be GRU or RNN.

Due to the time-varying nature of continuous-time dynamic networks, each node undergoes changes over time (such as interactions with other nodes), and thus no longer maintains its initial interaction state. However, the original state of a node represents the essence of interaction events. Therefore, we propose to preserve the original state of node v_i at the time of interaction with v_j. Specifically, given an interaction $\left(v_i,v_j,e_{i,j}\left(t_j\right)\right)$ of node v_i before time t, the calculation of the node v_i's original state information $r_i\left(t_j\right)\$ is as follows,

$$\begin{equation} r_i\left(t_j\right) = W_{raw}\left[z_i\left(t_j^-\right)||e_{i,j}\left(t_j^-\right)||z_j\left(t_j^-\right)||\Phi\left(t_j-t_j^-\right)\right],\end{equation} \tag{ 3 }$$

where W_raw represents the trainable parameters. For the original state, we allocate a temporary storage unit, which is discarded whenever the original state of a node's neighbor is no longer in use.

In the evolving perspective, we employ a multi-head temporal self-attention module to obtain the fusion of the current state of historical interactions of node v_i at time t, aiming to capture the real-time dynamics of the network. Specifically, the current states of node v_i and its historical interaction nodes at time t are represented as $h_i\left(t\right)$ and $\left[h_1\left(t\right),\ldots,h_j\left(t\right)\right]$, respectively. Then, the fusion of the current states of historical interaction events of node v_i at time t, denoted as $P_{N_i\left(t\right)}^e$, is calculated as follows,

$$\begin{align} P_{N_i\left(t\right)}^e & = MultiHeadAtten\left(Q\left(t\right),K\left(t\right),V\left(t\right)\right) \nonumber\\ & = Softmax{\left(\frac{Q\left(t\right)K\left(t\right)^T}{\sqrt{d} }\right)}V\left(t\right), \end{align} \tag{ 4 }$$

where $N_i\left(t\right)$ denotes the set of historical interaction events of node v_i at time t (i.e. the temporal neighbors). Q(t), K(t) and V(t) are represented as follows,

$$\begin{align} Q\left(t\right) &= h_i\left(t\right)||\Phi\left(0\right)\end{align} \tag{ 5 }$$

$$\begin{align} K\left(t\right) &= \left[h_1\left(t\right)||e_{i,1}\left(t\right)||\Phi\left(t-t_1\right),\right.\nonumber\\ &\quad\left.\ldots,h_j\left(t\right)||e_{i,j}\left(t\right)||\Phi\left(t-t_j\right)\right], \end{align} \tag{ 6 }$$

$$\begin{align} V\left(t\right) &= K\left(t\right).\end{align} \tag{ 7 }$$

The original perspective considers the essence of node interactions. To obtain the fusion of the original states of the node's historical interaction events, we first compute a feedback attention coefficient with growth characteristics. This coefficient can be viewed as the model's growth process from the original state to the current state. After a period of model growth, the model gains deeper insights into previous events. Specifically, given a historical interaction of node v_i at time t_j represented by $\left(v_i,v_j,e_{i,j}\left(t_j\right)\right)$, we compute the feedback attention coefficient from time t_j to the current time t,

$$\begin{equation} I_{i,j}\left(t_j\rightarrow t\right) = \psi\left(g_{i,j}\left(t_j\rightarrow t\right)\right),\end{equation} \tag{ 8 }$$

here, $g_{i,j}(t_j\rightarrow t)$ is utilized to learn the growth characteristics from time t_j to t. A single layer of non-linear mapping is employed, computed as follows,

$$\begin{equation} {g_{i,j}}\left(t_j\to t\right) = relu\left({W_{1}} \left[{h_i}\left(t\right)||{h_j}\left(t\right)||\Phi \left(t - {t_j}\right)\right]\right),\end{equation} \tag{ 9 }$$

where w₁ is a trainable parameter, $||$ is the concatenation operator, and $relu()$ is the activation function. Since the feedback coefficients must be between 0 and 1, the external function ψ must satisfy that all of its values are greater than or equal to zero. Therefore, the sigmoid function is chosen as the external function, i.e,

$$\begin{equation} {I_{i,j}}\left(t_j\to t\right) = {\left(1 + {e^{W_{2} {g_{i,j}}\left({t_j\to t}\right)}}\right)^{- 1}},\end{equation} \tag{ 10 }$$

where w₂ is a trainable parameter. The final feedback coefficient can be obtained as follows,

$$\begin{equation} {a_{i,j}}\left(t_j\to t\right) = \frac{{\exp \left({I_{i,j}}\left({t_j\to t}\right)\right)}}{{\sum\nolimits_{j^{^{^{\prime}}} \in {\textrm{N}_i}\left(t\right),} {\exp \left({I_{i,j^{^{^{\prime}}} }}\left({t_j\to t}\right)\right)} }}.\end{equation} \tag{ 11 }$$

Next, we aggregate the original state information of the historical interactions of the target node v_i using the obtained feedback coefficients,

$$\begin{equation} P_{N_i\left(t\right)}^r = \sum\limits_{j \in {N_i}\left(t\right)} {{a_{i,j}}\left(t_j\to t\right) * {W_\mathrm{trans}} \;} \left[{r_j}\left({t_j}\right)||\Phi \left(t - {t_j}\right)\right],\end{equation} \tag{ 12 }$$

where $W_\mathrm{trans}$ represents the parameters for the linear transformation. So far, we have managed to learn the evolution process of dynamic graphs from two perspectives. Next, we will introduce the mutual coupling of these two perspectives.

To improve the robustness and generalization ability of the model, we perform coupling learning on the information obtained from the two perspectives to obtain the final embedding representation of the nodes. Specifically, we further parameterize the changes in the node itself, the original state, and the current state, and feed them into a two-layer feedforward neural network (FNN) to obtain the final embedding representation $z_i\left(t\right)$ of the node

$$\begin{align} z_i^n\left(t\right) &= {W_{n}} \left[{h_i}\left({t}\right)||\Phi \left(t-t^-\right)\right],\end{align} \tag{ 13 }$$

$$\begin{align} z_i^r\left(t\right) &= {W_r} \left[{h_i}\left({t}\right)||P_{N_{i}\left(t\right)}^e\right],\end{align} \tag{ 14 }$$

$$\begin{align} z_i^e\left(t\right) &= {W_e} \left[{h_i}\left({t}\right)||P_{N_{i}\left(t\right)}^r\right],\end{align} \tag{ 15 }$$

$$\begin{align} z_i\left(t\right) &= \mathrm{FNN}\left(\left[z_i^n\left(t\right)||z_i^r\left(t\right)||z_i^e\left(t\right)\right]\right),\end{align} \tag{ 16 }$$

where W_n, W_r and W_e denote trainable parameters. Next, we will analyze the complexity of the model. The time complexity of MPFA is primarily composed of three parts: the dynamic update module, the multi-head attention from the evolving perspective, and the feedback attention from the original perspective. The dynamic update module uses a recurrent network, resulting in a time complexity of O(n), where n is the number of historical neighbors. The time complexity for each batch in the evolving perspective is dominated by the multi-head attention mechanism, which has a time complexity of $O(kn^2)$, where k is the number of attention heads. Since the number of heads is set to 2 in the experiment, the time complexity of multi-head attention is $O(2n^2)$. The primary time consumption in the original perspective is the calculation of the feedback attention coefficient, with a time complexity of O(n). Therefore, the total time complexity $O(n+2n^2+n)$. Generally, the largest part determines the overall time complexity of the model, making the time complexity of MPFA $O(n^2)$. Although $O(n^2)$ is relatively large, the dynamic update module allows the model to achieve excellent results with a small n, thereby reducing the overall time cost to some extent. The main space complexity of the MPFA algorithm arises from the multi-head attention mechanism from the evolving perspective. Therefore, our analysis focuses on the space complexity of multi-head attention. Assuming that the input shape of each batch is $[B,n,d]$, for attention with k heads, the space required for Q(t), K(t), and V(t) is $O(2Bnd)$. The matrix multiplication $Q(t)K(t)^T$ needs to store intermediate activations, occupying $O(4Bnd)$. The $softmax()$ function requires storing $Q(t)K(t)^T$, which takes up $O(2Bn^{2}k)$. After computing $softmax()$, a dropout operation is performed, and a mask matrix needs to be saved, occupying $O(2Bn^{2}k)$. The space required to store the attention score and the values of V(t) is $O(2Bn^{2}k)$ and $O(2Bnd)$, respectively. Dropout operations for the output map also require saving their inputs, which is $O(2Bnd)$, and the mask matrix, which is O(Bnd). The total space for these operations is $O(3Bnd)$. When the number of heads is 2 and considering only variable changes, the space complexity is $O(Bnd+Bn^2)$.

We employ link prediction loss to train the proposed model. After obtaining the embedding representations $z_i(t)$ and $z_j(t)$ for nodes v_i and v_j at time t, we feed them into an MLP decoder to determine whether nodes v_i and v_j interact at time t, i.e. $p_{i,j}\left(t\right) = \mathrm{MLP}\left(z_i\left(t\right),z_j\left(t\right)\right)$. Link prediction is inherently a binary classification problem, thus we directly utilize binary cross-entropy function as the link prediction loss function, ${l_n} = - ({y_n}*\log (\delta ({s_n}))) + (1 - {y_n})*\log (1 - \delta ({s_n})),n \in 2N$, where y_n denotes the label of the $n{th}$ sample pair, and the s_n represents the probability that the $n{th}$ sample pair is predicted to be positive. The $\delta \left( {{s_n}} \right)$ is the sigmoid function for numerical stability.

UNVote:⁴ UNVote is the roll-call voting network of the United Nations General Assembly. It consists of 201 nodes and 1035742 edges.

LastFM:⁵ This dataset is a bipartite network consisting of interactions between users and songs. The network contains about 1000 users and 1000 of the most listened to songs, and these nodes have generated a total of 1293 103 interactions.

SocialEvo:⁴ This is a university community cell phone proximity monitoring network that tracks student activity over an eight-month period, totaling 2099 519 edges.

MOOC:⁵ MOOC is a network of online course interactions whose nodes are students and course content. Each interaction represents a student's access to the course. The dataset has a total of 7144 nodes and 411 749 edges.

Reddit:⁵ In this dataset, the nodes are 10 000 active users and 1000 active subreddits, and an interaction is a post written by users on subreddits, resulting in 672 447 interactions. Dynamic labels represent whether a user is banned from posting, and each interaction is converted into a feature vector.

USLegis:⁴ USLegis is a Senate co-sponsorship network. It records instances where members of the United States Senate co-sponsored the same bill in a given Congress. The dataset has 225 nodes and 60 396 edges.

Wikipedia:⁵ This dataset contains 157 474 temporal edges with 1000 most edited pages and 8227 active users as nodes. Labels indicate whether users are banned from editing the page, and interactions are transformed into text features. Both Reddit and Wikipedia are bipartite graphs.

ComplexDynamics:⁶ This dataset is a dynamic interaction graph with entities as nodes and different types of events as edges, a non-bipartite graph as the training dataset of the WSDM 2022 Challenge. To generate this dataset, we choose 8295 different nodes, and the number of interactions per node is less than 753, resulting in 150 035 interactions.

To generate the Complex Dynamics dataset, we started with the snapshot of the WSDM 2022 Challenge training data on 2014-10-19 04:00:00. We first cleaned the data by filtering out all missing and unavailable data. Then, in the remaining data, we kept only nodes with less than 753 interactions where the timestamp (the timestamp in Unix epoch) was present. The data processed in this way would cause confusion in the node sequence number and time sequence, so we had to renumber the nodes (with anonymized categorical features) and sort each interaction event (with anonymized categorical features) under the time constraint, resulting in 150 035 interactions generated by 8295 nodes. The resulting dataset has the following characteristics: 1) the number of high-frequency interactions between nodes is relatively small; 2) the rate of continuously repeated interactions between the same nodes or node pairs is low; 3) there are symbiotic events under the same timestamp.

We select nine temporal models (JODIE [24], DyRep [21], TGAT [26], TGN [25], CAWN [29], EdgeBank [30], TCL [27], GraphMixer [39], and DyGFormer [28]) as benchmarks. Additionally, we enhance two superior static graph models, GAT and GraphSAGE, to temporal models (GAT [8]-t and GraphSAGE [9]-t) and use them as new baselines. Refer to the Related Work subsection for baseline details. For the hyperparameters of the baselines, in addition to using their official versions and searching for the optimal settings, we also follow the configurations of algorithms such as DyGFormer [28]. The batch size for all baselines on the eight datasets is set to 200, the node embedding dimension to 172, and the time embedding dimension to 100. Dropout rates are chosen from the range 0.0, 0.1, 0.2, 0.3, and 0.4. The temporal neighborhood setting meets the needs of our scenario. Although GAT and GraphSAGE were originally designed for static graph neural networks, their good performance prompted us to extend them to the temporal graph scenario by adding temporal coding information to the model.

In all datasets and experiments, we use a batch size of 200, a learning rate of 0.0001, a node embedding dimension of 172, and a time embedding dimension of 100, and we set the number of heads in the multi-head attention to 2. During training, we compute the event probabilities using an equal number of random negative and positive samples and use Adam as the optimizer. Reference metrics include average precision (AP), accuracy (ACC), and area under the ROC curve (AUC). For the node classification task on the benchmark datasets (Wikipedia and Reddit), AUC is employed due to label imbalances. Temporal neighbors, selected from the ten nearest neighbors in chronological order, are used in all experiments. The final results are the average of ten model runs. For all tasks, We perform a chronological split, assigning 70% for training, 15% for validation, and 15% for testing, based on node interaction timestamps. All experiments are conducted under the Ubuntu 18.04 system, the Pytorch deep learning framework, and the NVIDIA TITAN XP GPU 12GB.

5.4.1. Dynamic link prediction

5.4.2. Dynamic node classification

5.4.3. Ablation studies

To evaluate the effectiveness of different components of our proposed model, we conduct the ablation studies in this part. We experiment with different combinations of modules from MPFA for dynamic link prediction on the MOOC and USLegis datasets, aiming to observe their specific properties (see figure 4 for details). For W/O RP and W/O EP, we directly remove the Raw and Evolving Perspectives modules. For W/O RED and W/O ED, we set the current state to the intrinsic features of the node when there is no dynamic update in the evolutionary perspective, and to a vector of all zeros when the node does not have intrinsic features. From the results of the overall ablation experiments on both datasets, MPFA with two perspectives achieves the highest performance in both transductive and inductive tasks, as evidenced by superior values in AP, AUC, and ACC. This result substantiates the effectiveness of MPFA. Conversely, the evolving perspective without dynamic updating emerges as the least effective, as indicated by the lowest values across all three evaluation criteria. This underscores the significance of the dynamic updating module designed to capture long-term dependencies. Regarding the evolving and original perspectives, the model exclusively featuring the evolving perspective demonstrates the second-best performance on MOOC, while on USLegis, the model with only the raw perspective achieves the second-best performance. This discrepancy highlights the varying roles of these perspectives in different scenarios. Furthermore, the model incorporating both evolving and original perspectives but lacking dynamic updating performs suboptimally, underscoring the crucial role of long-term dependency in effective model learning.

Zoom In Zoom Out Reset image size

5.4.4. Long dependency propertity

To verify that our model can maintain the long-term dependency property with a smaller number of historical neighbors, we perform experiments on the link prediction performance under different numbers of temporal neighbors for MPFA and five baselines on the MOOC and USLegis datasets. As shown in figure 5, we observe that MPFA maintains a stable state when the number of temporal neighbors varies from 1 to 30, both on the MOOC and USLegis datasets. Compared to other models, especially on the MOOC dataset, all five baselines show the characteristic of performance improvement with increasing number of neighbors. Among them, CAWN shows the most significant performance improvement. Although these baselines occasionally show jumps in the predicted values for the transductive and inductive subtasks of USLegis, they still require an increase in the number of neighbors to achieve performance improvement. In summary, our proposed dynamic update module is effective. Superior predictions with fewer history neighbors suggest that our model requires only a limited number of interactions for effective long-term dependency properties.

Zoom In Zoom Out Reset image size

5.4.5. Hyperparametric analysis

In this section, we examine the performance of MPFA under three main hyperparameters on two datasets, MOOC and USLegis. As shown in figure 6, the performance of MPFA remains relatively stable with variations in batch size, embedding dimension, and number of temporal neighbors. In figure 6(a), the model performance shows a slight decrease as the batch size increases. In figure 6(b), there is a slightly increasing trend in model performance with higher embedding dimensions. Figure 6(c) shows that the model remains stable up to 30, with a slight decrease thereafter. In USLegis, MPFA experiences two significant fluctuation points with different batch sizes: 200 points in the transductive task and 300 points in the inductive task (figure 7(a)). With different embedding dimensions (figure 7(b)), similar to the MOOC dataset, the performance of MPFA tends to increase slightly. In experiments with different numbers of temporal neighbors (figure 7(c)), performance remains stable in the transductive task, while some fluctuations are observed in the inductive task.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

5.4.6. Attention score visualization

In this section, we visualize the attention coefficients for both perspectives on the MOOC and USLegis datasets (see figure 8). This allows us to observe the areas of focus for the two perspectives in transductive and inductive tasks. In general, the evolving perspective has higher attention coefficients, typically reaching values of 0.3 or higher, while the original perspective tends to have smaller coefficients, typically below 0.4. These coefficient values are consistent with the model's ability to gain deeper insights into historical events after learning, emphasizing the extraction of a limited amount of valid information. On both the MOOC and USLegis datasets, the evolving perspective in the transductive task directs more attention to neighbors closer in the time series. Specifically, on the MOOC dataset, the evolving perspective shows exceptional attention to the closest temporal neighbors. Conversely, in the inductive task, the focus is relatively more diffuse. For the original perspective on both datasets, temporal attention is significantly larger and broader on the MOOC dataset in the transductive task. For the USLegis dataset, attention remains essentially the same for both transductive and inductive tasks. From these observations, we can derive the following insights.

Zoom In Zoom Out Reset image size

(1) Importance of the Evolving Perspective: The attention coefficients of the evolving perspective are typically higher, particularly in the transductive subtask. This suggests that when processing time series data, such as continuous-time dynamic graphs, the current state of a node is more valuable for predicting future outcomes. (2) Capturing the Essence of Interaction: From the perspective of the feedback attention coefficient, the original perspective focuses more on the essence of historical interaction events, assigning a smaller coefficient to extract the most useful interaction essence information. (3) Influence of Temporal Neighbors: In transductive tasks, the evolving perspective places greater emphasis on temporal neighbors in the time series. This indicates that for dynamically changing data, the model must rely more on recent historical information to make accurate predictions. (4) Difference in Task Type: Transductive and inductive tasks exhibit distinct attentional distributions. In transductive tasks, the model tends to focus on closer neighbors, whereas in inductive tasks, the attentional distribution is more dispersed. This demonstrates that the MPFA can adaptively allocate attention based on the task type.

5.4.7. Time efficiency

To evaluate the runtime efficiency of MPFA and the current leading baselines, we select five algorithms (TCL, TGAT, GraphMixer, CAWN, DyGFormer) to evaluate the time taken to achieve optimal training results on the MOOC and USLegis datasets, as shown in figure 9. Although TGAT requires the least time to optimize its model parameters on MOOC, it does not perform well in terms of link prediction results. In contrast, although MPFA takes more time than TGAT, it achieves optimal test performance on MOOC. Similarly, TCL has the shortest training time, but does not produce better test results. We attribute this to the fact that both models may have an inadequate fit to the dataset, leading to premature training termination. Overall, MPFA demonstrates a favorable balance between runtime efficiency and optimal performance.

Zoom In Zoom Out Reset image size