Research of news text classification method based on hierarchical semantics and prior correction

2.1.1Data augmentation

First, the dataset is optimized through data augmentation by generating diverse augmented texts with different sentence structures using a multilingual translation approach. This enhances the quality of the simple news titles. Specifically, the text is translated into other languages using a translator. Then, the translated titles are back-translated to generate augmented titles that resemble the original text. These augmented titles serve as additional training data for news titles.

The news body is typically a long and complex text with intricate sentence structures. To avoid sentence structure disorder, a data augmentation method based on keyword avoidance and synonym replacement is employed to transform the news body text. The specific steps involve building a synonym table, generating a keyword dictionary for the news body, and performing random synonym replacement. To construct the synonym table, all the vocabulary in the dataset is extracted. Then, by querying a word vector table, the top k synonyms with cosine similarity greater than the threshold are selected for each word, forming a synonym word list. The process of calculating cosine similarity between words is as follows:

Next, the keyword dictionary for the news body is generated. The TF-IDF correlation coefficients between each word in the dataset and the news body are calculated. From the correlation coefficients, the top m words are selected to form the keyword dictionary for each text. Then, random keyword substitution is performed after tokenizing the input text. About 1t of all words in the news body are randomly chosen to be replaced. If a word is not a keyword in the text and has synonyms available, one of the synonyms is randomly selected to replace that word.

After acquiring the augmented news bodies and titles, they are merged with the original news text training set to create the final dataset for model training.

2.1.2Optimization of model training based on adversarial learning

In the process of training the news text hierarchical semantic classification model, adversarial learning is employed to improve the model’s robustness and ability to generalize. This is achieved by introducing perturbations in the embedding layer, which makes the training process more challenging and encourages the model to adapt to the influence of perturbations.

Firstly, the model is trained normally with the original samples and the loss is computed using the cross-entropy loss function. This process is shown in the following equation:

First, the loss is backpropagated to compute the gradient of the loss for the embedding. The gradient is then scaled by a constraint coefficient to control the perturbation magnitude. The equation reads:

Subsequently, adversarial training learning data is generated by adding the perturbation to the direction of increasing gradient in embedding. It can be expressed as:

The model is then retrained using the data, and the adversarial loss is computed which is defined as follows:

Finally, the original model embedding is restored, and the adversarial loss is backpropagated to update the model parameters.

2.2.1The extraction of news title features

During the extraction of news title features, a pre-trained BERT model is utilized to parse the news titles and obtain the feature matrix title_out. To extract local text features from it, a two-layer convolutional neural network (CNN) is employed to process this feature matrix. Considering multiple dimensions of semantic hierarchy, various convolutional kernels are used to extract features from this matrix. The list of convolutional kernels used is represented as follows:

where pilter_ij donates different convolutional kernels. n represents the number of different sizes of convolutional kernels, and m represents the number of convolutional kernels for each size.

Next, a pooling layer is applied to compress the extracted high-dimensional features, filtering out non-key features in the local features of the title. After the compression through convolution and pooling, each high-dimensional feature is concatenated to obtain the news title feature F_title.

2.2.2The extraction of news body features

The process of extracting features from the news body text mainly involves long-range feature extraction and document attention fusion. During long-range feature extraction, a pre-trained model BERT is initially used to extract the sequence features of the body text, denoted as content_out. Subsequently, content_out is fed into a Bidirectional Long Short-Term Memory(BiLSTM) to assist in extracting semantic features from long-distance sequences in the body text. The last hidden state H_D and the feature matrix E_D are given by the following equation:

2.2.3The fusion of hierarchical semantic features

The feature vector F_title extracted from the news title feature matrix and the feature vector F_content extracted from the news body text feature matrix represent different levels of interpretation of the entire news article, and they have varying influences on the classification of news labels. HSPC uses an adaptive gating network [30] to enable the model to balance the effects of both and select the text semantic level with stronger label relevance. First, the fusion proportion is computed. The process is shown in the following formula:

Next, G is used to perform a weighted sum of the news title feature vector F_title and the news body text feature vector F_content, resulting in the complete feature vector F_{new_gates} for the full news text. The process is illustrated in the equation as follows:

After that, a pre-trained model is used to extract the full-text feature vector F_{new_pre} from the entire news article. Then, a linear concatenation is performed between F_{new_pre} and F_{new_gates} to assist F_{new_gates} in text classification. The formula is demonstrated as follows:

2.2.4the prediction of label probabilities

After obtaining the final news text representation vector F_news, it is converted into probability scores out for the labels. The vector space mapping is defined as follows:

Then, out is normalized to obtain the probability of the text representation vector belonging to each label. The label with the highest probability is selected as the best label for the news text. The process is defined as follows:

where softmax represents the normalization operation and pred represents the label prediction distribution.

2.3.1Confusion prediction set partitioning based on K-entropy

By utilizing the top k highest label prediction probabilities, the confusion score of the predicted label for a news test sample is calculated. Assuming p₁, p₂, p₃, …, p_k are the k highest probability scores, the first step is to normalize these k probability values. The normalization process is defined as follows:

Next, the K-entropy is calculated as the confusion score, expressed as pred _ S:

Algorithm 1 Confusion prediction set partitioning based on K-entropy

Input: probability distribution set of the news test dataset test _ pred; set partition threshold t; top k parameter k.

Output: set of low confusing sample distributions clear; set of highly confusing sample distributions confused.

1: Function set partitontest _ pred, k, t

2: confused = [] , clear = []

3: For pred in test _ pred do

4: pred _ top _ k = sort (pred, k)

5: pred _ top _ k = normalize (pred _ top _ k)

6: pred _ S = S (pred _ top _ k)

7: if pred _ S > t then

8: confused . append (pred _ top _ k)

9: else

10: clear . append (pred _ top _ k)

11: return confused, clear

The confusion partition algorithm is shown in Algorithm 1. The algorithm takes inputs including the probability distribution set of the news test dataset test _ pred, the set partition threshold t, and the top k parameter. It outputs two sets: clear for low confusing sample distributions and confused for highly confusing sample distributions. First, two new probability distribution lists, confused and clear, are initialized to store the partitioned highly confusing and low confusing label predictions, respectively (line 2). Next, iterate over each sample distribution in the test dataset and retrieve the top k highest probability scores from the distribution (lines 3-4). After re-normalizing the scores, calculate the K-entropy as the confusion score for the distribution (lines 5-6). The formula for information entropy S is shown in the following equation:

Then, partition the distribution into the highly confusing sample set if its confusion score exceeds the threshold. Otherwise, assign the distribution to the low confusing sample set (lines 7-10). Finally, output the partitioned highly confusing label prediction set confused and low confusing label prediction set clear.

Next, the time complexity and space complexity of the partitioning algorithm are analyzed. When the training set size is n, the algorithm time complexity of the traversal operation is O (n). If m is the number of labels, the complexity of calculating k-element information entropy is O (m). Moreover, the algorithm for the comparison operation has a time complexity of O (1). Sorting each testing set distribution has a time complexity of O (mlogm). Therefore the total time complexity is O (nmlogm). For space complexity, the additional space is two lists confused and clear. The sum of samples in the list is the number of samples in the training set n. Each sample contains label predictions of length m. Finally, extra space O (nm) is required.

2.3.2Label correction interval selection based on binary entropy

After partitioning the set, the highly confusing label prediction set confused and the low confusing label prediction set clear are obtained. To improve the stability of the correction, it is important to further partition the probability correction interval from the confused set. To do this, the label predictions are sorted in descending order based on their probabilities. For simplicity, the sorted probabilities are denoted as shown in the equation:

To improve the accuracy of predictions, it is often necessary to divide the predicted label distribution into two sub-intervals using a threshold point t. The sub-intervals are the correction interval, denoted as p_select, which lies on the left side of the threshold, and the exclusion interval, denoted as p_noselect, which lies on the right side of the threshold. When partitioning the original label distribution using the threshold point, certain constraints must be satisfied in combinatorial form as follows:

Algorithm 2 Label correction interval selection based on binary entropy

Input: set of high confused probability distributions confused; interval filtering threshold i.

Output: low confused interval filtering query dictionary pred _ dic.

1: function interval_filteringconfused, i

2: pred _ dic = []

3: for pred in confused do

4: pred _ dic [pred] = []

5: sort _ idx = sort (pred)

6: sum = 0

7: for idx in sort _ idx do

8: pred _ dic [pred] . append (idx)

9: sum = sum + pred [idx]

10: if sum < (1 - sum) then

11: continue

12: S = get _ entropy (sum, 1 - sum)

13: if S ≤ i then

14: break

15: return pred _ dic

Algorithm 2 is used for the interval selection. The input of this algorithm is a set of samples that have been partitioned and is highly confusing. The output of this algorithm is a query dictionary that contains correction intervals. The keys of this dictionary are the predicted label distributions of highly confusing samples, and the values are the corresponding correction intervals for probability distributions. Initially, an empty dictionary pred _ dic is created to store the distribution-corresponding correction intervals (line 2). Then, for each highly confusing sample interval, the pred is sorted in descending order (lines 4-5). The pred _ sum variable is initialized, and the correction interval probabilities are accumulated for each boundary point of the probability distribution (lines 6-9). Next, the correction intervals are checked to see if they satisfy the third and fourth inequality conditions to determine if further filtering is needed (lines 10-14). Finally, the query dictionary pred _ dic is returned.

Next, the time complexity and space complexity of the screening algorithm are analyzed. For time complexity, the algorithm goes through each highly confusing interval sample. If there are n samples, the time complexity of this loop would be O (n). Assuming there are m labels, it would require O (m) time complexity to go through all subscripts of the prediction distribution. Lastly, every testing set distribution needs to be sorted, and the time complexity of this step is O (mlogm). Therefore, the time complexity of the interval screening algorithm is O (nmlogm). Space complexity analysis involves calculating the additional space used by algorithm 2. In this case, the algorithm uses a data structure pred _ dic, which contains key and value pairs that are proportional to the number of highly confusing label prediction samples. If the scale of samples is denoted by n and the size of each pair depends on the number of labels m, then the algorithm requires extra space of O (nm).

2.3.3Label prediction correction based on prior distribution

After determining the correction intervals for each confused prediction, the correction intervals for each set are adjusted using the prior distribution. The specific correction steps can be divided into prior distribution statistics and prior correction. During prior distribution statistics, the prior distribution is calculated by counting the occurrences of each label in the training set and normalizing them. The formula is demonstrated as follows:

During the prior correction, firstly, the average distribution of high and low confusing labels is calculated as follows:

Simultaneously, p˜select is used together with the average distribution p¯select to correct the j-th highly confusing sample. This correction process aims to make the label distribution within the correction interval approach the prior distribution. The calculation process is shown as follows, where the multiplication and division operations are applied element-wise to the vectors.

The text step involves re-normalizing pselect(j) within the correction interval to obtain the corrected label prediction probability distribution. The calculation process is shown in the following equation:

After the correction, the sum of probabilities in the original correction interval may change, and it may not necessarily add up to 1 when combined with the probabilities outside the correction interval. Therefore, it is necessary to rescale each recalculated label score to embed it back into the original label prediction distribution, ensuring that the sum of scores for all labels in the original distribution is 1. The rescaling process is shown as follows:

After obtaining the corrected label predictions for the highly confusing samples, the corrected predictions for the highly confusing samples are combined with the label predictions for the low confusing samples to form the final prediction. The label with the highest probability score in each prediction is selected as the final label for the news text.

The calibration algorithm is shown in Algorithm 3. The input to the algorithm includes the query dictionary pred _ dic, the set of low confusing label prediction distributions clear, the set of highly confusing label prediction distributions confused, and the news training set train _ set. The output is the calibrated highly confusing probability distribution of the news text fixed _ confused. First, the algorithm initializes the arrays for the prior distribution and the list for the corrected output probabilities (lines 2-3). Then, the number of labels after frequency counting is normalized, and train _ distri stores the proportion of each label in the training set (lines 4-6). Next, the algorithm iterates through each label prediction distribution in the set of highly confusing samples and performs calibration and re-normalization. The corrected label prediction distribution is then rescaled and embedded back into the original label prediction. The calibration results are stored in the fixed _ confused list (lines 7-11). Finally, the algorithm returns the list of calibrated label predictions, fixed _ confused.

Then the time complexity of the correction algorithm will be analyzed. When the number of training set samples is n and the number of labels is m, the time complexity of calculating the prior probability distribution is O (n). During the correction process, each highly confusing sample needs to be traversed and a prior correction on the label probability within the correction interval needs to be performed. This step has a time complexity of O (nm). Therefore, the time complexity of the correction part is O (nm). The space complexity of the correction algorithm can be analyzed as follows. Algorithm 3 mainly uses two extra spaces, namely train_distribution and fixed_confused. The space complexity for train_distribution is O (n) which means that it is proportional to the number of corrected samples n. The space used by fixed_confused is also proportional to the number of corrected samples, and each label prediction is a vector of length m. Therefore, the space complexity of the correction algorithm is O (nm).

Algorithm 3 Label prediction correction based on prior distribution

Input: query dictionary pred _ dic; set of low confusing label prediction distributions clear; set of highly confusing label prediction distributions confused; training set ts.

Output:calibrated highly confusing probability distribution of the news text fixed_confused.

1: function probability_correction(ts, pred _ dic, clear, confused)

2: tr _ distri = []

3: fixed _ confused = []

4: for label in ts do

5: tr _ distri [lable] = tr _ distri [lable] + 1

6: tr _ distri [lable]/= (sum (lable))

7: for pred in confused do

8: sub _ label = pred _ dic [pred]

9: new _ pred = fix (pred, tr _ distri)

10: new _ pred = handle (new _ pred)

11: fixed _ confused . append (new _ pred)

12: return fixed _ onfused

3.3.1Comparison of classification performance for FCT oriented datasets

The main comparative method in the FCT dataset is RAM [29]. The classification results of various models on the FCT dataset are shown in Fig.  2. According to the figure, it can be observed that HSPC outperforms the RAM model in terms of classification performance on the FCT dataset. Compared to the RAM model, HSPC shows an improvement of 1.03% in accuracy. This is because news titles often provide a brief yet accurate representation of the news content, making them highly relevant to the news labels. While RAM uses information about the radicals of Chinese characters as semantic extensions, it also introduces redundancy and less relevant information in the process. HSPC takes the title information into separate consideration, allowing the title information to complement the news article information. By specifically considering the title and using it as a semantic expansion of the article, HSPC exhibits a more targeted approach compared to the expansion of news semantics using Chinese character-related intentional information.

Fig. 2

Classification ACC performance comparison chart for FCT dataset.

3.3.2Comparison of classification performance for AGNews oriented datasets

The main comparative model in the AGNews dataset is AGN [30] and TF-GCN [12]. Figure 3 provides a comparison of the classification results of various models on the AGNews dataset. According to the figure, it can be observed that HSPC performs better than the BERT+AGN classification model on the AGNews dataset, with an improvement of 1.39% in accuracy. This may be because the primary focus of AGN aims to reduce the influence of less relevant, weakly related, and unrelated words on the labels without extracting more semantic auxiliary feature representations. Meanwhile, the accuracy of HSPC has increased by 2.1% compared to TF-GCN. Although TF-GCN captures three kinds of information including document, topic, and word, as auxiliary information in the heterogeneous information network, it may result in losses when constructing large graphs. In contrast, HSPC considers the hierarchical semantics of news labels separately and uses a gate fusion method to balance the label-level semantics with the article-level semantics. This allows the model to pay more attention to the news semantic hierarchy with stronger label relevance, resulting in better performance.

Fig. 3

Classification ACC performance comparison chart for AGNews dataset.

3.3.3Comparison of classification performance for THUCNews oriented datasets

The comparative method in the AGNews dataset is LCM [31] and WTL-CNN [32] which serves as an enhancement component for CNN and LSTM classification models. The comparison of classification accuracy on this dataset for various models is shown in Fig.  4.

The LCM model is a technique used in machine learning to calculate the similarity between samples and labels during training. It generates a label confusion matrix that reduces the impact of label semantic overlap. Inspired by LCM, HSPC introduces an advanced PD correction algorithm to further adjust the label predictions, which also alleviates the impact of label overlap on classification. One problem with LCM is that LCM treats the news text as a whole and does not consider the importance of news titles when it comes to classifying news labels. In contrast, HSPC divides the semantic hierarchy of news texts, which enhances the influence of news titles on news label classification, and thus improves the model’s classification performance on news texts. In terms of accuracy, HSPC outperforms the BERT+LCM model by 2.55Although WTL-CNN extracts both local and global semantic information, it still lacks data enhancement and highly confusing label correction, and the accuracy of HSPC is improved by 2.8% compared with WTL-CNN.

Next, the reasons why the model performs well on the THUCNews dataset are analyzed. It is found that there is a strong correlation between multiple news titles and news labels in the THUCNews dataset. Some titles directly contain words strongly associated with the labels. By considering the title features separately, the model can fully utilize the most relevant label information in the titles and has a significant improvement in accuracy.

Fig. 4

Classification ACC performance comparison chart for THUCNews dataset.

3.3.4Comparison of classification performance for Ohsumed oriented datasets

The main comparative method in the Ohsumed dataset is BERTGCN [33], HyperGAT [34] and Text-level-GCN [35]. Figure 5 shows the accuracy comparison results of various models.

Fig. 5

Classification ACC performance comparison chart for Ohsumed dataset.

There is still room for improvement in BERTGCN in some aspects. On one hand, the adjacency matrix constructed by BERTGCN for node information exchange remains unchanged during model training, which means the model cannot dynamically focus on strongly related label information in the text. On the other hand, BERTGCN fails to fully consider the differences in the impact of labels and text on label classification, neglecting the influence of titles on news label classification. Similarly, other graph neural networks, whether using Text-level-GNN with document subgraphs or hypergraph neural networks that incorporate topic information, treat news texts as a whole without utilizing the hierarchical semantic features of news texts. Additionally, all other models did not consider highly confusing predictions in label prediction.

Therefore, HSPC, which focuses on hierarchical semantics and performs label prediction correction, exhibits better classification performance on the Ohsumed dataset. It surpasses the BERTGCN model by 1.15% in terms of classification accuracy.

3.4.1Hierarchical ablation comparison

To validate the rationality of hierarchical analysis and the necessity of the gate mechanism, the following ablation experiments are conducted.

Reduce title: This approach adopts the idea of extracting overall features without analyzing the hierarchical semantic characteristics of the news text.

Average: This approach uses the hierarchical analysis method but does not consider the fusion method between the title and the body text.

Hierarchical Semantics news text classification model: This approach adopts the hierarchical analysis method and uses the gate fusion mechanism to combine the title features and the full-text features of the news.

The comparative metrics are accuracy and F1 score. The ablation comparison results in each dataset are shown in Table 4. It can be observed that the Average model achieves better results. On all four datasets, there are improvements in both accuracy and F1 score. The accuracy increases by 1.17%, 1.48%, 1.46%, and 0.84% respectively, while the F1 score increases by 1.30%, 1.56%, 1.37%, and 0.86% respectively. This is because the Average model considers the hierarchical semantics of the labels. Compared to the Reduce title model that adopts overall feature extraction, the Average model shows a significant improvement in accuracy.

Next, the Average model and the HS model are compared. The main difference between them is that HS utilizes the gate fusion mechanism, while Average uses the method of average summation. On the four datasets, compared to the Average, the HS model improves the classification accuracy by 0.15%, 0.24%, 0.07%, and 0.31%, and the F1 score by 0.19%, 0.24%, 0.05%, and 0.37% respectively. It is observed that the HS model performs better on the Ohsumed dataset. This is because the Ohsumed dataset has a significant difference in semantics between the title and the body text, requiring a more balanced consideration of news text samples. The gate fusion mechanism can better balance the semantic relationship between the title and the labels, enabling the framework to focus more on the relevant semantic hierarchy for label classification. Therefore, the HS model with gate fusion performs more prominently on the Ohsumed dataset.

3.4.2Training ablation comparison

To compare the optimization effects of data augmentation and adversarial learning on the Hierarchical Semantics (HS) news text classification model, the following training ablation comparison experiments were designed in this section:

None: No optimization is performed on HS at both the data level and the training level.

+DA: Data augmentation is used to optimize the model training on HS, excluding adversarial learning.

+ADV: Adversarial learning is used to optimize the model training on HS, excluding data augmentation.

ALL: Both data augmentation and adversarial learning are used to optimize the model training on HS.

The comparative metrics are accuracy and F1 score on each dataset. The ablation results on the various datasets are shown in Table 5.

According to the results shown in the 5, the ALL approach achieves the best results. Compared to the None approach which does not utilize data optimization or training optimization, the ALL approach shows improvements in the accuracy by 0.62%, 0.45%, 0.43%, and 0.61% respectively, while the F1 score increases by 0.52%, 0.46%, 0.48%, and 0.57% respectively on the FCT, AGNews, THUCNews, and Ohsumed datasets.

Overall, adversarial learning demonstrates better optimization effects on the model compared to data augmentation across the four datasets. Both data augmentation and adversarial learning can be seen as adding perturbations to the model’s input while controlling the magnitude of the perturbations to ensure that the transformed samples are similar to the original samples in the semantic space. However, there are two key differences between the two methods. Firstly, the directions and magnitudes of the perturbations added by data augmentation and adversarial learning are different. Secondly, data augmentation optimizes the quality of the dataset outside the model by increasing its diversity to enhance the model’s training effectiveness. Adversarial learning optimizes the model internally by adding perturbations in the direction of gradient ascent to enhance the intensity of model training. Therefore, there is no conflict between the two, and the best effect is achieved when using both at the same time. From the analysis of the results, it can be observed that adding only adversarial learning leads to an improvement in accuracy ranging from 0.1% to 0.3% compared to adding only data augmentation. This is because data augmentation has some external technical dependencies. The quality of translation-based data augmentation relies on the performance of translation software, and the effectiveness of synonym replacement data augmentation depends on the quality of the word vector table. On the other hand, adversarial learning, whether it is the direction or magnitude of the added perturbations, is related to the model itself. It is a training strategy specifically designed for the model. Therefore, in most cases, adversarial learning tends to be more effective.

In terms of the comparative analysis of the enhanced results on different datasets, on the FCT and Ohsumed datasets where label imbalance exists, compared to the None approach without any training or data optimization, the +DA approach with data augmentation improves the accuracy by 0.13% and 0.26% respectively, and the F1 score by 0.12% and 0.17% respectively. On the AGNews and THUCNews datasets with more balanced labels, the accuracy improves by 0.09% and 0.08% respectively, and the F1 score improves by 0.08% and 0.07% respectively. This indicates that data augmentation performs relatively better on datasets with label imbalance.

The comparison of training set sizes before and after data augmentation on the FCT, Ohsumed, THUCNews, and AGNews datasets is shown in Table 6. In the case of the AGNews and THUCNews datasets where label imbalance is not an issue, data augmentation is used to increase the size of the training set by doubling the number of training samples, thereby enriching the semantic diversity of the original data. On the Ohsumed and FCT datasets, data augmentation not only enriches the semantic content of the dataset but also mitigates the label imbalance issue in the dataset. Therefore, the optimization effect of data augmentation is better on the FCT and Ohsumed datasets.

3.5.1Overall correction effect analysis

To analyze the calibration results of the PD algorithm, the data before and after calibration on different datasets are recorded as shown in Table 7.

Based on the data in table 7, it can be observed that the calibration algorithm has a certain corrective effect on the data from each dataset. The calibration effect is more significant on the FCT and Ohsumed datasets, with an improvement of 0.05% and 0.15% in predicted accuracy, respectively. The improvement is relatively modest on the AGNews and THUCNews datasets, with only 0.01% and 0.02% improvement, respectively. This indicates that the PD calibration algorithm can optimize the output probability distribution of the model and improve the classification accuracy.

Comparing the calibration performance of the PD algorithm on each dataset, it is evident that the improvement effect of the PD calibration algorithm is more pronounced on the FCT and Ohsumed datasets compared to the AGNews and THUCNews datasets. This suggests that the PD calibration algorithm performs better in correcting models on datasets with a larger number of labels compared to datasets with fewer labels.

Based on the analysis of the above observations, it can be speculated that the accuracy of low confusing samples and the proportion of highly confusing samples are the main factors influencing the calibration effect of the calibration algorithm. On one hand, as the number of labels increases, the classification difficulty of the dataset also increases, and the number of highly confusing label predictions to be handled also increases. Consequently, the calibration algorithm can better demonstrate its efficacy. On the other hand, the higher the accuracy of low confusing samples, the more reliable their probability distribution, making them a better baseline for correcting predictions of highly confusing samples.

To analyze the role of interval selection in the calibration algorithm and validate the correctness of the category filtering idea, the number of instances where the calibrated algorithm decreased the accuracy is recorded after 50 stable model training iterations. The number of calibration errors with and without interval selection is shown in Table 8. It can be observed that in the 50 instances of calibration after training, the calibration algorithm without interval selection has 3 errors in the FCT dataset, 1 error in the AGNews dataset, 0 errors in the THUCNews dataset, and 8 errors in the Ohsumed dataset. After adding interval selection, the number of calibration errors in the four datasets is reduced to 1 error, 0 errors, 0 errors, and 2 errors, respectively. Compared to the calibration algorithm without interval selection, the addition of interval selection resulted in a reduction of 2 errors, 1 error, 0 errors, and 6 errors in the respective datasets. The stability improvement is particularly evident in the Ohsumed dataset, while the calibration stability does not change significantly in the other three datasets. This occurrence may be because the stability of the calibration algorithm’s correction effect is influenced by the magnitudes of P_clear and P_confused in the dataset. The higher the values of P_clear and P_confused in the dataset, the better the stability of the correction.

3.5.2Analysis of low confusing samples

Low confusing samples serve as a reference for correcting highly confusing samples in the calibration algorithm. Analyzing the accuracy and proportion of low confusing samples can provide better insights into the correction effect of the calibration algorithm. The accuracy and proportion of low confusing samples are recorded during the model training process on the four datasets, and their trend changes are plotted in a line graph as shown in Fig.  6.

Fig. 6

Plot of low confusing sample share and ACC with model training.

Comparing the accuracy and proportion of low confusing samples in the AGNews and THUCNews datasets, after the model training stabilizes, the S_clear values in both datasets are similar, around 99.5%. However, the difference lies in the P_clear values, where the THUCNews dataset has a higher P_clear, approximately 3.75% higher than that of the AGNews dataset. Therefore, the main factor affecting the correction effect of the PD algorithm on these two datasets is P_clear. The PD algorithm performs better in correcting the THUCNews dataset compared to the AGNews dataset. Comparing the FCT dataset and the AGNews dataset, the increase in P_clear in the FCT dataset is not significant, around 1.0%. However, compared to the S_clear of approximately 99.5% in the AGNews dataset, the proportion of S_clear in the FCT dataset is around 97.5%, meaning that S_confused in FCT is 2.5%, which is six times that of the AGNews dataset. Therefore, in the FCT and AGNews datasets, the influence of S_clear on the correction algorithm is more significant, and the PD algorithm performs better in the FCT dataset.

This comparative analysis confirms the speculations made in the contrast analysis of the calibration effect. Firstly, when the accuracy of low confusing samples is similar, the larger the proportion of highly confusing samples, the better the correction effect. Secondly, when the proportion of highly confusing samples is approximately the same, the higher the accuracy of low confusing samples, the better the correction effect.

3.5.3Analysis of highly confusing samples

highly confusing samples are the main focus of the PD algorithm’s correction. Analyzing the accuracy and proportion of highly confusing samples can provide a better understanding of the correction effect of the calibration algorithm. The variation trends of the proportion and accuracy of highly confusing samples are shown in Fig.  7.

Fig. 7

Plot of highly confusing sample share and ACC with model training.

Comparing the four subgraphs (a), (b), (c), and (d), except for the Ohsumed dataset, where P_confused is approximately 40%, the P_confused values in the other three datasets are around 50%. However, the S_confused values in the other datasets range from 0.5% to 2.0%, while the S_confused value in the Ohsumed dataset is approximately 22.0%, significantly higher than the other datasets. Therefore, even though the Ohsumed dataset has lower values of P_clear and P_confused compared to the other three datasets, its higher proportion of S_confused allows for more samples to be corrected by the PD algorithm. Overall, the PD algorithm performs the best on the Ohsumed dataset.

The Ohsumed dataset has lower values of P_clear and P_confused compared to the other datasets, indicating that the correction effect in this dataset exhibits instability, confirming the speculation made in the calibration stability analysis. However, due to the presence of an interval selection algorithm, which eliminates most of the unlikely options, the negative impact caused by the low values of P_clear and P_confused in the Ohsumed dataset is alleviated. This improves the stability of the correction and ensures the effectiveness of label prediction.