Fuzzy Multi-Attribute Decision Making for Software Defect Detection Model Evaluation

doi:10.23940/ijpe.20.01.p9.7886

Abstract

Abstract:

With the continuous expansion of the computer system application field, the complexity of software system is also improving. Software defect detection has gradually become an important research direction in the field of software engineering. At present, the mixed statistics and machine learning methods have been proved to be able to implement software defect detection models well. However, the evaluation index of the detection model is diverse and it is difficult to determine which model evaluation indicators are in line with the actual expectations. Aiming at this kind of problem, a software defect detection model evaluation method based on fuzzy multi-objective attribute decision making is proposed. First, extract the characteristics of software modules, use McCabe and Halstead software modules to measure attributes. Then select five common classification algorithms to establish software defect detection models, and obtain seven evaluation index values of each model. Further, based on fuzzy multi-objective attribute decision making method with fuzzy analytic hierarchy process (FAHP) to compare multiple objectives, and obtain the results of index determination. Finally, the fuzzy evaluation algorithm is used to convert the evaluation index of qualitative evaluation into quantitative evaluation to obtain the final decision evaluation value. The experimental results show the effectiveness and practicality of the method.

Key words: multi-objective decision making algorithm, software defect detection, model evaluation

Yunjie Lei, Ying Ma, Shunyi Chen, Yu Sun, and Keshou Wu. Fuzzy Multi-Attribute Decision Making for Software Defect Detection Model Evaluation [J]. Int J Performability Eng, 2020, 16(1): 78-86.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

Figures/Tables 26

Figure 1.

Figure 2.

Table 1

Scale meaning table"

Scale value	Comparative relationship
1	Both are equally important
3	The former is slightly more important than the latter
5	The former is moderately more important than the latter
7	The former is important than the latter
9	The former is significantly important than the latter
2, 4, 6, 8	The intermediate state of the above adjacent scale values
If the ratio of the importance of element a to element b is a_ij, then the ratio of the importance of element b to element a is $a ij = 1 / a ji$

Table 1

Table 2

M i = ∏ i = 1 n a ij, i, j = 1,2, ?, n

(1)

W i' = M i n, i, j = 1,2, ?, n

(2)

W i = W i' ∑ j = 1 n W i', i = 1,2, ?, n

(3)

λ max = ∑ i = 1 n [(AW) i n W i]

(4)

CI = λ max - n n - 1

(5)

Table 3

CR = CI RI

(6)

Figure 3.

N = (N 1, N 2, N 3, ?, N n)

(7)

A = (A 1, A 2, A 3, ?, A n)

(8)

f : N → F A, μ j → f i μ j = r j 1, r j 2, ?, r jm, j = 1,2, ?, n

(9)

R i = R i' ∑ j = 1 n R i', i = 1,2, ?, n

(10)

S = W × R

(11)

Table 4

Table 5

Table 6

Table 7

Table 8

Table 9

Z = S × μ

(12)

Table 1

References 20.

1.	T. Hall, S. Beecham, D. Bowes, D. Gray, et al., “A Systematic Literature Review on Fault Prediction Performance in Software Engineering,” IEEE Transactions on Software Engineering, Vol. 38, No. 6, pp. 1276-1304, 2012
2.	Y. Ma, X. Zhu, S. Zhu, et al., “Combating the Class Imbalance Problem in Sparse Representation Learning,” Journal of Intelligent and Fuzzy Systems, Vol. 35, No. 2, pp. 1865-1874, 2018
3.	Y. Ma, S. Zhu, Y. Chen,J. Li, “Kernel CCA based Transfer Learning for Software Defect Prediction,” IEICE Transactions on Information and Systems, Vol. E100.D, No. 8, pp. 1903-1906, 2017
4.	M. H. Halstead, “Elements of Software Science,”Operating and Programming Systems Series, 1978
5.	T. J. Mccabe, “A Complexity Measure,” Vol. 2, pp. 308-320, 1976
6.	Y. Ma, S. Zhu, K. Qin,G. Luo, “Combining the Requirement Information for Software Defect Estimation in Design Time,” Information Processing Letters, Vol. 114, No. 9, pp. 469-474, 2014
7.	Y. Ma, Ke Qin, and S. Zhu, “Discrimination Analysis for Predicting Defect-Prone Software Modules,” Journal of Applied Mathematics, Vol. 2014, Article ID 675368, 14 pages, 2014
8.	Y. Ma, W. Pan, S. Zhu, et al., “An Improved Semi-Supervised Learning Method for Software Defect Prediction,” Journal of Intelligent and Fuzzy Systems, Vol. 27, No. 5, pp. 2473-2480, 2014
9.	Y. Ma, G. Luo,H. Chen, “Kernel based Asymmetric Learning for Software Defect Prediction,” IEICE Transactions on Information and Systems, Vol. E95-D, No. 1, pp. 267-270, 2012
10.	G. Luo, Y. Ma,K. Qin, “Active Learning for Software Defect Prediction,” IEICE Transactions on Information and Systems, Vol. E95-D, No. 6, pp. 1680-1683, 2012
11.	G. Luo, Y. Ma,K. Qin, “Asymmetric Learning based on Kernel Partial Least Squares for Software Defect Prediction,” IEICE Transactions on Information and Systems, Vol. E95-D, No. 7, pp. 2006-2008, 2012
12.	J. Ren, K. Qin, Y. Ma,G. Luo, “On Software Defect Prediction using Machine Learning,” Journal of Applied Mathematics, Vol. 2014, Article ID 785435, 8 pages, 2014
13.	Y. Ma, G. Luo, X. Zeng,A. Chen, “Transfer Learning for Cross-company Software Defect Prediction,” Information and Software Technology, Vol. 54, No. 3, pp. 248-256, 2012
14.	Y. Zhou, H. Leung,B. Xu, “Examining the Potentially Confounding Effect of Class Size on the Associations between Object-Oriented Metrics and Change-Proneness,” IEEE Transactions on Software Engineering, Vol. 35, No. 5, pp. 607-623, 2009
15.	Y. Zhou, B. Xu, H. Leung, et al., “An in-Depth Study of the Potentially Confounding Effect of Class Size in Fault Prediction,” ACM Transactions on Software Engineering and Methodology, Vol. 23, No. 1, pp. 1-51, 2014
16.	G. Eibl and K. P. Pfeiffer, “How to Make AdaBoost.M1 Work for Weak Base Classifiers by Changing Only One Line of the Code,” inProceedings of European Conference on Machine Learning, 2002
17.	S. Keerthim and S. K. Shevade, “SMO Algorithm for Least-Squares SVM Formulations,” Neural Computation, Vol. 15, No. 2, pp. 487-507, 2014
18.	S. Tesfamariam and R. Sadiq, “Risk-based Environmental Decision-Making using Fuzzy Analytic Hierarchy Process (F-AHP),” Stochastic Environmental Research and Risk Assessment, Vol. 21, No.1, pp. 35-50, 2006
19.	C. Wei, C. F. Chien,M. J. Wang, “An AHP-based Approach to ERP System Selection,” International Journal of Production Economics, Vol. 96, No. 1, pp. 47-62, 2005
20.	M. G.Sobol and G. Klein, “Multi-Objective Decision Making,”Interfaces, 1987

	TP-Rate	FP-Rate	Precision	Accuracy	Recall	F-Measure	AUC
TP-Rate
FP-Rate
Precision
Accuracy
Recall
F-Measure
AUC

Order n	1	2	3	4	5	6	7	8	9
RI	0.00	0.00	0.58	0.90	1.12	1.24	1.32	1.41	1.45
Order n	10	11	12	13	14	15
RI	1.49	1.52	1.54	1.56	1.58	1.59

	TP-Rate	FP-Rate	Precision	Accuracy	Recall	F-Measure	AUC
NaïveBayes	0.897	0.688	0.924	0.189	0.897	0.911	0.722
Logistic	0.971	0.854	0.914	0.176	0.971	0.942	0.787
J48	0.978	0.875	0.912	0.172	0.978	0.944	0.644
AdaBoostM1	1.000	1.000	0.903	0.000	1.000	0.949	0.742
SMO	1.000	1.000	0.903	0.000	1.000	0.949	0.500

	TP-Rate	FP-Rate	Precision	Accuracy	Recall	F-Measure	AUC
NaïveBeyes	0.950	0.796	0.842	0.222	0.950	0.893	0.678
Logistic	0.983	0.898	0.830	0.184	0.983	0.900	0.704
J48	0.928	0.742	0.848	0.233	0.928	0.886	0.670
AdaBoostM1	1.000	0.999	0.817	0.030	1.000	0.899	0.705
SMO	1.000	0.995	0.817	0.065	1.000	0.900	0.503

	TP-Rate	FP-Rate	Precision	Accuracy	Recall	F-Measure	AUC
NaïveBeyes	0.907	0.625	0.888	0.297	0.907	0.897	0.788
Logistic	0.974	0.791	0.870	0.292	0.974	0.919	0.803
J48	0.940	0.662	0.886	0.330	0.940	0.912	0.743
AdaBoostM1	1.000	1.000	0.845	0.000	1.000	0.916	0.779
SMO	0.995	0.963	0.849	0.121	0.995	0.917	0.516