Comparative study of SPI success factors in global and in-house environment for large-scale software companies

Systematic literature review

The SLR is among the most widely adopted methodologies when it comes to the use of evidence-based software engineering. One of the key points of an SLR is that it is a well-planned and systemically executed scheme. In this research, we have followed Kitchenham & Charters (2007) guidelines to execute the SLR process. This SLR process consists of three main steps: (1) defining the protocol, (2) conducting the protocol, (3) reviewing the protocol. In the first step, the protocol contains these elements: (1) research question’s identification, (2) search strategy, (3) study selection, (4) quality assessment, (5) extraction of data and synthesis. The first element i.e., identification of the research question has already been mentioned in the introduction section; search strategy, study selection criteria, and quality assessment are described in section A whereas extraction and synthesis of data are included in results and discussion. This research has been conducted by one student and two academic staff members to avoid biases.

Data collection via questionnaire surveys

An empirical survey is an appropriate research methodology for collecting qualitative and quantitative data from a large group of participants by using techniques such as a questionnaire survey or interview (Niazi, Babar & Verner, 2010). To perform the questionnaire survey, we have used guidelines provided by Kitchenham & Pfleeger (2008). In this research work, online questionnaire surveys have been used because the traditional survey approach has many problems (Misro et al., 2014). During the first SLR, we have identified the 35 SPI success factors for GSD in the case of large-scale software development organizations. Similarly, during the second SLR, we have identified the 33 SPI success factors for the in-house context in the case of large-scale software development organizations. Based on the results of both SLRs, we have developed the first questionnaire survey to ask industry professionals about the additional SPI success factors for both contexts, in the case of large-scale software development organizations, according to their own experience.

The questionnaire survey has three sections: section-I is related to demographics, section-II contains the list of SPI success factors in case of GSD and request to mention additional SPI access factors, and section-III presents the list of SPI success factors in case of in-house development and request to mention additional SPI access factors. Through prior contact via email and telephone, the participants were informed that the data would only be accessible to the research team and only be used for research purposes.

The second questionnaire survey aims to find out the relative importance of common SPI success factors in both contexts. The questionnaire for the second survey has two sections. The first section is to obtain demographics whereas the second section contains 17 common SPI success factors and a Likert scale to measure the relative importance of success factors. The survey employs a 5-point Likert scale that ranges from 1 to 5. The ‘1’ indicates less importance and ‘5’ indicates more importance.

The questionnaire surveys were first tested through a pilot study involving five professionals from different organizations. Based on the results of the pilot study, the final versions of the questionnaire surveys were developed.

Statistical tests

Firstly, we have conducted the parametric correlational analysis (Pearson) (Obilor & Amadi, 2018) which is not appropriate for a small sample size, even though we have used it to achieve results similar to non-parametric tests. The non-parametric correlation (Kendall’s Tau and Spearman) tests are suitable for the analysis of the results of a small sample size (Fagerl, 2012). Kendall’s Tau test is used because it produces more valid and sensible results under small sample conditions. Furthermore, to remove the biases, we have conducted Spearman’s test as well to look into how sensitively the data is behaving under different tests of the same nature. Figure 1 presents the research methodology for this study.

Study selection in GSD context

In the first phase after applying the search strings to different databases, we got 257 results: 53 results from IEEE, 105 from ACM, 40 from Springer, 27 from Science Direct, and 32 from Wiley. In the second step, based on the title and abstract 70 papers were selected. The studies cannot be finalized based on the title and abstract only, so we continued further. In the third step, duplicate papers were removed, and 66 papers were left. Further, based on the full text, we were left with only 28 papers. In the subsequent step, 13 papers were shortlisted primarily based on quality assessment criteria.

In the next step, we carried out the snowballing method (Wohlin, 2014) to check the references of the 13 papers. We found 13 papers and applied the same selection process to these papers, and eventually seven papers were selected in this phase. So, after an extensive filtration process, we got 20 final studies. The process is shown in Fig. 2.

Study selection in in-house context

In the initial part, after applying the search strings to different databases, we got 2,100 results. 415 results from IEEE, 488 from ACM, 569 from Springer, 107 from Science Direct, and 521 from Wiley. In the second step, based on the title and abstract, 111 publications were selected. Within the third step, duplicate papers were removed, and 101 papers were left. Further, based on the full text, we were left with only 38. After this, 17 papers were shortlisted based on quality assessment criteria. In the next step, we applied the snowballing technique (Wohlin, 2014) to check the references of the 17 papers. We found 10 papers, applied the same selection process to these papers, and eventually two papers were selected during this process. Therefore, after an intensive filtration process, we obtained 19 final studies. All this process is shown in Fig. 3. The same quality assessment criteria were used for this SLR as well.

Data extraction and synthesis

Data was carefully extracted from the papers and all authors were involved in this step for the removal of biases. Data synthesis was performed, and we got a list of SPI success factors from 20 studies that were selected for the final stage of SLR performed in the GSD context. Similarly, we got a list of SPI success factors from 19 studies selected during SLR for an in-house context. Initially, we found 35 success factors of SPI related to the GSD environment and 33 SPI success factors related to the in-house development environment. Table 3 represents the SPI success factors in the GSD environment. The success factors have been represented as G₁, G₂, G₃…G₃₅.

This completes the answer to RQ1.

Table 4 represents the success factors of SPI in the in-house context. The success factors have been represented as I₁, I₂, I₃…I₃₃.

This provides the answer to RQ2.

Results of the first questionnaire survey

This section provides the results of the first survey. From the first questionnaire survey, we got four additional success factors of SPI in the GSD environment, and four success factors of SPI in the in-house development environment.

The additional SPI success factors for GSD context are: On-time delivery, Identification of possible roadblocks, Time management, and Understandable documentation. The additional SPI success factors for in-house context are: understandable documentation, on-time delivery, recreational activities, and self learning. This provides the answer to RQ3.

The additional GSD SPI success factors have been represented as G₃₆, G₃₇, G₃₈, and G₃₉ and have been represented in Table 3. The additional in-house SPI success factors have been represented as I₃₄, I₃₅, I₃₆, and I₃₇ and have been represented in Table 4.

Common SPI success factors in GSD and in-house context

By comparing Tables 3 and 4, we have found that there are a total of 17 SPI success factors that are common in both development environments. The common SPI success factors have been represented as C₁, C₂, C₃…C₁₇:

C₁ -SPI leadership.

C₂ -project management.

C₃ -Communication.

C₄ -Teamwork.

C₅ -Setting SPI goals.

C₆ -SPI awareness.

C₇ -Allocation of resources.

C₈ -SPI consultancy.

C₉ -Staff involvement.

C₁₀ -Senior management commitment and support.

C₁₁ -Organizational infrastructure.

C₁₂ -Customer involvement/ Client support.

C₁₃ -Motivation.

C₁₄ -Training.

C₁₅ -Reward schemes.

C₁₆ -On-time delivery.

C₁₇ -Easy to understand documentation.

This provides the answer to RQ4.

The identified common SPI success factors speed up SPI in the case of large-scale companies, where both approaches are followed, and also enable such companies to produce quality software products.

Results of the second questionnaire survey

Another industrial survey was performed to find out the relative importance of the common SPI success factors. This section provides the results of the second industrial survey and completes the answer to research RQ5.

We have conducted a correlation analysis of the survey results to see if there is any relation between the common SPI success factors of GSD and in-house environments. For this purpose, we have used statistical analysis including the Pearson correlation as a parametric test (Obilor & Amadi, 2018) and, Kendall’s Tau and the Spearman correlation as non-parametric tests (Fagerl, 2012) because the sample is small. Two same types of tests help to cross-examine the result. The sample size is significantly small i.e., 29 total correct responses. However, we were able to evaluate which common SPI success factors are more strongly correlated in GSD or in-house working environments respectively. First, we have applied a parametric test which is the Pearson correlation, the results are shown in Table 5.

We have found that SPI leadership, communication, teamwork, SPI awareness, allocation of resources, SPI consultancy, organizational infrastructure, motivation, on-time delivery, and easy to understand documentation are more important in GSD as compared to in-house, whereas, project management, setting SPI goals, staff involvement, senior management commitment and support, customer involvement/client support, training, and reward schemes are more important in the in-house environment as compared to the GSD. To validate the results, we also performed non-parametric tests (Kendall’s Tau and the Spearman correlation), and the results were the same as shown in Tables 6 and 7. Table 6 presents the results of Kendall’s Tau correlation. Table 7 presents the results of the Spearman correlation.

Using Table 5, the SPI success factors that are more important in the GSD environment as compared to the in-house environment can be arranged in descending order, based on respective scores, as shown in Table 8.

So according to the Pearson correlation (Table 5) the top three SPI success factors, for the GSD environment in the case of the large-scale development organization, with respective scores are: allocation of resources (0.457), teamwork (0.355), and communication (0.306). The results can be validated from Tables 6 and 7 as the results are the same in the case of Kendall’s Tau correlation (Table 6) and the Spearman correlation (Table 7).

According to Kendall’s Tau correlation (Table 6), the top three SPI success factors, for the GSD environment, with respective scores are: allocation of resources (0.325), teamwork (0.309), and communication (0.246). According to the Spearman correlation (Table 7), the top three SPI success factors, for the GSD environment, with respective scores are: allocation of resources (0.401), teamwork (0.381), and communication (0.294).

Again, using Table 5, the SPI success factors that are more important in the in-house environment as compared to the GSD environment can be arranged in descending order, based on respective scores, as shown in Table 9.

According to the Pearson correlation (Table 5), the top three SPI success factors, for the in-house environment in the case of large-scale software development organizations, with respective scores are: senior management commitment and support (0.357), training (0.327) and reward scheme (0.292). The results can be validated from Tables 6 and 7 as the results are the same in the case of Kendall’s Tau correlation (Table 6) and the Spearman correlation (Table 7).

According to the Kendall’s Tau correlation (Table 6), the top three SPI success factors, for the in-house environment, with respective scores are: training (0.282), senior management commitment and support (0.235), and reward scheme (0.208). According to the Spearman correlation (Table 7), the top three SPI success factors, for the in-house environment, with respective scores are: training (0.401), senior management commitment and support (0.317), and reward scheme (0.274).

Construct validity

The study is based on 35 SPI success factors in the GSD context and 33 SPI success factors in the in-house environment for large-scale software development companies. In both cases, SPI success factors have been identified through SLR by following the recommended steps. The lists of the SPI success factors have been provided to respondents in both cases. Furthermore, to improve readability and understanding, both questionnaires were tested through pilot studies. Based on the results of the pilot study, the final versions of the questionnaire surveys were developed. Therefore, the participants were familiar with the SPI success factors.

Internal validity

The participants have working experience of 1 to 5 years and they belong to such large software development companies where GSD and in-house approaches are followed. Therefore, the SPI success factors were associated with the working place of the participants. To examine the relative importance of the common SPI success factors in both environments, a 5-point Likert scale has been employed that has already been used in many studies.

External validity

To conduct the study, a sample of 50 professionals has been selected from 50 large-scale software development companies where GSD and in-house approaches are practiced. This is quite possible that all the corresponding practitioners do not entirely agree with the results of this study, but the sample is truly illustrative of the population. As out of 110 companies, 50 were chosen for the study based on size, development approaches being followed, and relevant available respondents. Moreover, to reveal which common SPI success factor carries more value in which development environment and to validate the results, the Pearson correlation (parametric), Kendall’s Tau, and the Spearman correlation (non-parametric) have been calculated.