Research Methodology: 5. Systematic Literature Review (SLR)
Research Methodology: 5. Systematic Literature Review (SLR)
Research Methodology: 5. Systematic Literature Review (SLR)
5. Systematic Literature
Review (SLR)
1
Course Outline
1. Pengantar Penelitian
2. Tahapan Penelitian
3. Literature Review
2
5. Systematic Literature Review
(SLR)
5.1 Pengantar SLR
5.2 Tahapan Planning
5.3 Tahapan Conducting
5.4 Tahapan Reporting
3
5.1 Pengantar SLR
4
Literature Review
• Literature Review is a critical and in depth
evaluation of previous research (Shuttleworth, 2009)
(https://explorable.com/what-is-a-literature-review)
5
Why doing Literature Review?
• To establish connection or relationship
between existing knowledge and the
problem to be solved
• To refine the research problem
• To identify significance of research
• To define research question
6
Laporan Literature Review
• Hasil laporan dari proses Literature Review biasanya
diletakkan di Bab 2 pada skripsi/tesis/disertasi kita
• Cara konvensional menyusun laporan Literature Review
adalah dengan menyusun bab 2 dengan tiga hal berikut:
1. Tinjauan Studi:
• Berisi tentang penelitian-penelitian yang berhubungan erat
(related research) dengan penelitian kita. Gunakan diagram yang
menggambarkan metode yang diusulkan oleh peneliti lain
• Harus bisa menggambarkan posisi penelitian kita, apa bedanya
penelitian kita dengan penelitian sebelumnya
2. Tinjauan Pustaka:
• Berisi landasan teori dari berbagai terminologi yang muncul pada
judul dan abstrak penelitian kita
• Apabila menjelaskan tentang algoritma, harus dijelaskan tentang
tahapan algoritma secara detail
3. Kerangka Pemikiran:
• Berisi kerangka atau framework dari penelitian yang kita lakukan
• Memuat gambaran besar untuk menjelaskan alur dan kerangka
pemikiran penelitian kita secara umum
7
Types of Literature Review
1. Traditional Review
2. Systematic Literature Review or Systematic
Review
3. Systematic Mapping Study (Scoping Study)
4. Tertiary Study
8
Traditional Review
• Provides an overview of the research
findings on particular topics
• Advantages: produce insightful, valid
syntheses of the research literature if
conducted by the expert
• Disadvantages: vulnerable to unintentional
and intentional bias in the selection,
interpretation and organization of content
9
Contoh Traditional Review
• Liao et al., Intrusion Detection System: A
Comprehensive Review, Journal of Network and
Computer Applications, 36(2013)
• Galar et al., A Review on Ensembles for the Class
Imbalance Problem: Bagging-, Boosting-, and
Hybrid-Based Approaches, IEEE Transactions on
Systems, Man, and Cybernetics, Part C (Applications
and Reviews), Vol. 42, No. 4, July 2012
• Cagatay Catal, Software fault prediction: A
literature review and current trends, Expert
Systems with Applications 38 (2011)
10
Systematic Mapping Study
• Suitable for a very broad topic
• Identify clusters of evidence (making
classification)
• Direct the focus of future SLRs
• To identify areas for future primary studies
11
Contoh Systematic Mapping Study
• Neto et al., A systematic mapping study of software
product lines testing, Information and Software
Technology Vol. 53, Issue 5, May 2011
• Elberzhager et al., Reducing test effort: A
systematic mapping study on existing approaches,
Information and Software Technology 54 (2012)
12
Tertiary study
•Is a SLR of SLRs
•To answer a more wider question
•Uses the same method as in SLR
•Potentially less resource intensive
13
Contoh Tertiary study
• Kitchenham et al., Systematic literature reviews in
software engineering – A tertiary study,
Information and Software Technology 52 (2010)
• Cruzes et al., Research synthesis in software
engineering: A tertiary study, Information and
Software Technology 53 (2011)
14
Systematic Literature Review (SLR)
• A process of identifying, assessing, and interpreting
all available research evidence, to provide answers
for a particular research question
• A form of secondary study that uses a well-defined
methodology
• SLRs are well established in other disciplines,
particularly medicine. They integrate an individual
clinical expertise and facilitate access to the
outcomes of the research
15
Systematic Literature Review (SLR)
• The purpose of a systematic literature reviews is
to provide as complete a list as possible of all
the published and unpublished studies relating
to a particular subject area
• While traditional reviews attempt to summarize
results of a number of studies, systematic
literature reviews use explicit and rigorous
criteria to identify, critically evaluate and
synthesize all the literature on a particular topic
17
Tahapan SLR
1. Formulate the review’s research question
PLANNING
2. Develop the review’s protocol
18
5.2 Tahapan Planning
19
The Research Question (RQ)
• Is the most important part in any SLR
• Is not necessarily the same as questions
addressed in your research
• Is used to guide the search process
• Is used to guide the extraction process
• Data analysis (synthesis of evidence) is
expected to answer your SLR’s RQ
20
Formulation of RQ
• Features of good question:
• The RQ is meaningful and important to practitioners and
researchers.
• The RQ will lead to changes in current software
engineering practice or to increase confidence in the value
of current practice
• The RQ will identify discrepancies between commonly held
beliefs and the reality
• RQ can be derived primarily based on researcher’s
interest
• An SLR for PhD thesis should identify existing basis for the
research work and where it fits in the current body of
knowledge
21
Formulation of RQ
The formulation of RQs about effectiveness of a
treatment should focus on 5 elements known as PICOC:
▪ Population (P) - the target group for the investigation (e.g.
people, software etc.)
▪ Intervention (I) - specifies the investigation aspects or issues
of interest to the researchers
▪ Comparison (C)– aspect of the investigation with which the
intervention is being compared to
▪ Outcomes (O)– the effect of the intervention
▪ Context (C)– the setting or environment of the investigation
26
Example of RQs
Radjenovic et al., “Software fault prediction metrics:
A systematic literature review”
27
SLR Protocol
• A plan that specifies the basic review
procedures (method)
• Components of a protocol:
1. Background
2. Research Questions
3. Search terms
4. Selection criteria
5. Quality checklist and procedures
6. Data extraction strategy
7. Data synthesis strategy
28
5.3 Tahapan Conducting
1. Identify the relevant literature
2. Perform selection of primary studies
3. Perform data extraction
4. Assess studies’ quality
5. Conduct synthesis of evidence
29
1. Identifying Relevant Literature
• Involves a comprehensive and exhaustive
searching of studies to be included in the review
• Define a search strategy
• Search strategies are usually iterative and benefit
from:
• Preliminary searches (to identify existing review and
volume of studies)
• Trial searches (combination of terms from RQ)
• Check the search results against list of known studies
• Consult the experts in the field
30
Common Approach to Construct Search String
31
E.g. Search String
Salleh et al. (2011)
32
E.g. Search String
• Kitchenham et al. (2007) used their structured questions to
construct search strings for use with electronic databases:
Population: software OR application OR product OR Web OR
WWW OR Internet OR World-Wide Web OR project OR
development
Intervention: cross company OR cross organisation OR cross
organization OR multiple-organizational OR multiple-
organisational model OR modeling OR modelling effort OR cost
OR resource estimation OR prediction OR assessment
Contrast: within-organisation OR within-organization OR within-
organizational OR within-organisational OR single company OR
single organisation
Outcome: Accuracy OR Mean Magnitude Relative Error
• The search strings were constructed by linking the four OR
lists using the Boolean AND
33
Sources of Evidence
• Digital libraries
• Reference lists from relevant primary studies and review
articles
• Journals (including company journals such as the IBM
Journal of Research and Development), grey literature
(i.e. technical reports, work in progress)
• Conference proceedings
• Research registers
• The Internet (google)
• Direct contact specific researcher(s)
34
E.g. Sources of Evidence
Salleh et al. (2011)
35
E.g. Sources of Evidence
Kitchenham et al. (2007)
• The search strings were used on 6 digital libraries:
• INSPEC , El Compendex, Science Direct, Web of Science, IEEExplore,
ACM Digital library
• Search specific journals and conf. proceedings:
• Empirical Software Engineering (J)
• Information and Software Technology (J)
• Software Process Improvement and Practice (J)
• Management Science (J)
• International Software Metrics Symposium (C)
• International Conference on Software Engineering (C)
• Manual search:
• Evaluation and Assessment in Software Engineering (C)
• Check references of each relevant article
• Contact researchers
36
Managing Bibliography
• Use relevant Bibliographic package to manage large
number of references
• E.g. Mendeley, EndNote, Zotero, JabRef Reference
Manager etc.
37
Documenting the Search
• The process of conducting SLR must be transparent and replicable
• The review should be documented in sufficient detail
• The search should be documented and changes noted
• Unfiltered search results should be saved for possible reanalysis
38
2. Selection of Studies
• Primary studies need to be assessed for their actual
relevance
• Set the criteria for including or excluding studies
(decided earlier during protocol development, can be
refined later)
• Inclusion & exclusion criteria should be based on RQ
• Selection process should be piloted
• Study selection is a multistage process
39
E.g. Selection of Studies
Kitchenham et al. (2007) used the following
inclusion criteria:
• Any study that compared predictions of cross-company
models with within-company models based on analysis
of single company project data.
40
E.g. Selection of Studies
Salleh et al. (2011)
• Inclusion criteria:
• to include any empirical studies of PP that involved higher
education students as the population of interest.
• Exclusion criteria:
• Papers presenting unsubstantiated claims made by the
author(s), for which no evidence was available.
• Papers about Agile/XP describing development practices other
than PP, such as test-first programming, refactoring etc.
• Papers that only described tools (software or hardware) that
could support the PP practice.
• Papers not written in English.
• Papers involving students but outside higher education.
41
3. Assessing Studies’ Quality
• To provide more detailed Inclusion/Exclusion
criteria
• To check whether quality differences provide an
explanation for differences in study results
• As a means of weighting the importance of
individual studies when results are being
synthesized
• To guide the interpretation of findings and
determine the strength of inferences
• To guide recommendations for further research
42
Assessing Studies’ Quality
• Quality relates to the extent to which the study minimizes
bias and maximizes internal and external validity
(Khan et al. 2001)
• Quality Concepts Definition (Kitchenham & Charter, 2007)
Terms Synonyms Definition
Bias Systematic tendency to produce results that depart
error systematically from the ‘true’ results.
Unbiased results are internally valid
Internal Validity The extent to which the design and conduct
Validity of the study are likely to prevent systematic
error. Internal validity is a prerequisite for
external validity
External Generalizabilit The extent to which the effects observed in
Validity y, Applicability the study are applicable outside of the study
43
Assessing Studies’ Quality
44
E.g. Study Quality Assessment - Salleh et al. (2011)
Item Answer
1. Was the article referred? [30] Yes/No
2. Were the aim(s) of the study clearly stated? [16], [67] Yes/No/Partially
3. Were the study participants or observational units adequately described?
For example, students’ programming experience, year of study etc. Yes/No/Partially
[44], [68]
4. Were the data collections carried out very well? For example, discussion
of procedures used for collection, and how the study setting may have Yes/No/Partially
influenced the data collected [44], [48], [67], [68]
5. Were potential confounders adequately controlled for in the analysis? 67] Yes/No/Partially
6. Were the approach to and formulation of the analysis well conveyed? For
example, description of the form of the original data, rationale for Yes/No/Partially
choice of method/tool/package [48], [67], [68]
7. Were the findings credible? For example, the study was methodologically
explained so that we can trust the findings; findings/conclusions are Yes/No/Partially
resonant with other knowledge and experience [48], [44], [68]
45
E.g. Study Quality Assessment
46
4. Data Extraction
• Involve reading the full text article
• Data extracted from primary studies should be recorded using
data extraction form
• The form should be designed and piloted when the protocol is
defined
• Collect all the information that can be used to answer the RQ
and the study’s quality criteria
• Both quality checklist and review data can be included in the
same form
• In case of duplicates publications (reporting the same data),
refer the most complete one
• For validation, a set of papers should be reviewed by 2 or more
researchers. Compare results and resolve any conflicts
47
5. Synthesis of Evidence
• Involves collating and summarizing the results of the
included primary studies
• Key objectives of data synthesis (Cruzes & Dyba, 2011):
• to analyze and evaluate multiple studies
• to select appropriate methods for integrating or
providing new interpretive explanations about them
• Synthesis can be:
• Descriptive (narrative/non-quantitative)
• Quantitative (e.g. meta-analysis)
51
Reporting SLR results in Journals
• Some journals and conferences include a specific topic on
SLR:
• Information & Software Technology has an editor
specializing in systematic reviews
• Journal of Systems and Software
• Expert Systems with Applications
• IEEE Transactions on Software Engineering
• International Symposium on Empirical Software
Engineering & Measurement (ESEM)
• International Conference on Evaluation & Assessment
in Software Engineering (EASE)
• International Workshop on Evidential Assessment of
Software Technologies (EAST)
52
Reporting Structure
• Introduction
• General introduction about the research. State the
purpose of the review. Emphasize the reason(s) why the
RQ is important. State the significance of the review
work and how the project contributes to the body of
knowledge of the field.
• Main Body
• Review method – briefly describe steps taken to conduct
the review
• Results – findings from the review
• Discussion – implication of review for research &
practice
• Conclusions
53
Reference
• Abbott, M., & McKinney, J. (2013). Understanding and Applying
Research Design. John Wiley & Sons, Inc.
• Berndtsson, M., Hansson, J., & Olsson, B. (2008). Thesis Projects:
a Guide for Students in Computer Science and Information
Systems (2nd ed.). London: Springer-Verlag
• Blaxter, L., Hughes, C., & Tight, M. (2006). How to Research (3rd
ed.). Open University Press
• Blessing, L. T. M., & Chakrabarti, A. (2009). DRM, a Design
Research Methodology. Springer-Verlag London
• Cohen, L., Manion, L., & Morrison, K. (2005). Research Methods in
Education (5th ed.). Taylor & Francis Group
• Dawson, C. W. (2009). Projects in Computing and Information
Systems A Student’s Guide (2nd ed.). Pearson Education Limited
• Jonker, J., & Pennink, B. (2010). The Essence of Research
Methodology. Springer-Verlag Berlin Heidelberg
• Lichtfouse, E. (2013). Scientific Writing for Impact Factor Journals.
Nova Science Publishers, Inc.
54
Reference
• Kothari, C. (2004). Research Methodology: Methods and
Techniques. New Age International
• Might, M. (2010). The Illustrated Guide to a Ph.D. Matt.might.net.
Retrieved from http://matt.might.net/articles/phd-school-in-
pictures/
• Marczyk, G., DeMatteo, D., & Fertinger, D. (2005). Essentials of
Research Design and Methodology. John Wiley & Sons, Inc.
• Rea, L. M., & Parker, R. A. (2014). Designing and Conducting
Survey Research: A Comprehensive Guide (4th ed.). John Wiley &
Sons, Inc.
• Runeson, P., Host, M., Rainer, A., & Regnell, B. (2012). Case Study
Research in Software Engineering: Guidelines and Examples. John
Wiley & Sons, Inc.
• Sahu, P. K. (2013). Research Methodology: A Guide for
Researchers In Agricultural Science, Social Science and Other
Related Fields. Springer
• Veit, R., Gould, C., & Gould, K. (2013). Writing, Reading, and
Research (9th ed.). Cengage Learning
55