Effective Literature Review

Informing Science Journal
Volume 9, 2006
A Systems Approach to Conduct an

Effective Literature Review in Support of
Information Systems Research
Yair Levy and Timothy J. Ellis
Graduate School of Computer and Information Sciences,
Nova Southeastern University, Florida, USA
[email protected] [email protected]
Abstract
This paper introduces a framework for conducting and writing an effective literature review. The
target audience for the framework includes information systems (IS) doctoral students, novice IS
researchers, and other IS researchers who are constantly struggling with the development of an
effective literature-based foundation for a proposed research. The proposed framework follows
the systematic data processing approach comprised of three major stages: 1) inputs (literature
gathering and screening), 2) processing (following Blooms Taxonomy), and 3) outputs (writing
the literature review). This paper provides the rationale for developing a solid literature review
including detailed instructions on how to conduct each stage of the process proposed. The paper
concludes by providing arguments for the value of an effective literature review to IS research.
Keywords: Literature review, effective literature review, literature search, literature categorization, literature classification, literature analysis, literature synthesis, doctoral education.
Introduction
A methodological review of past literature is a crucial endeavor for any academic research
(Webster & Watson, 2002, pp. 48-49). The need to uncover what is already known in the body of
knowledge prior to initiating any research study should not be underestimated (Hart, 1998). Some
fields of studies, such as engineering, have chronically suffered from a lack of proper literature
reviews, which has hindered theoretical and conceptual progress (D. Shaw, 1995). Webster and
Watson (2002) also criticized the Information Systems (IS) field for having very few theories and
outlets for quality literature review. Moreover, they noted that the IS field may greatly benefit
from effective methodological literature reviews that are strengthening IS as a field of study
(Webster & Watson, 2002, p. 14). In light of these considerations, the central aim of this study is
to address the issue of developing an effective literature review by proposing a systematic approach that will guide the researcher
Material published as part of this journal, either online or in print,
on such a daunting task.
is copyrighted by the publisher of the Informing Science Journal.
Permission to make digital or paper copy of part or all of these
works for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or
commercial advantage AND that copies 1) bear this notice in full
and 2) give the full citation on the first page. It is permissible to
abstract these works so long as credit is given. To copy in all
other cases or to republish or to post on a server or to redistribute
to lists requires specific permission and payment of a fee. Contact
[email protected] to request redistribution permission.
Editor: Eli Cohen
This paper is divided into four main

sections. The rest of this introductory
section will address what a literature
review is and why a literature review
is crucial for research. The following
three sections will review the three
steps of the proposed systematic approach for literature review. Section
two will address the literature review
A Systems Approach to Conduct an Effective Literature Review
inputs including: ways to find applicable literature, qualifying the literature, ways to read research
literature, and how to know that one is done with the literature search. The third section will review the proposed process for analyzing, synthesizing, and evaluating the literature. The final
section addresses the output step of this systematic process by proposing an approach to writing
the actual literature review following the theory of argument.
What is a Literature Review?

Novice researchers tend to approach the literature review as nothing more than a collection of
summaries of papers or an elaborated annotated bibliography of multiple research manuscripts
(Webster & Watson, 2002). A meaningful literature review is much more. Hart (1998) defined
the literature review as the use of ideas in the literature to justify the particular approach to the
topic, the selection of methods, and demonstration that this research contributes something new
(p. 1). He also noted that for the literature review, quality means appropriate breadth and depth,
rigor and consistency, clarity and brevity, and effective analysis and synthesis (Hart, 1998, p. 1).
J. Shaw (1995) noted that the process of the review should explain how one piece of research
builds on another (p. 326). Webster and Watson (2002) defined an effective literature review as
one that creates a firm foundation for advancing knowledge. It facilitates theory development,
closes areas where a plethora of research exists, and uncovers areas where research is needed (p.
13). From these definitions it is clear that an effective literature review should include the following characteristics: a) methodologically analyze and synthesize quality literature, b) provide a
firm foundation to a research topic, c) provide a firm foundation to the selection of research
methodology, and d) demonstrate that the proposed research contributes something new to the
overall body of knowledge or advances the research fields knowledge-base.
Stages of the literature review process

This paper presents the literature review process in a systematic way following the inputprocessing-output approach. Process is defined in the context of this work as sequential steps
of activities (Sethi & King, 1998). Thus, following the description of what constitutes an effective
literature review combined with the definition of process proposed here, this study defines literature review process as: sequential steps to collect, know, comprehend, apply, analyze, synthesize,
and evaluate quality literature in order to provide a firm foundation to a topic and research
method. Moreover, the output of the literature review process should demonstrate that the proposed research contributes something new to the overall body of knowledge. The term body of
knowledge (BoK) refers to the cumulative research-supported knowledge achieved by building
on each others [research] results (Iivari, Hirschheim, & Klein, 2004, p. 314). Following this
approach, the current paper suggests a three-step literature review process to guide novice researchers on the development of a sound and effective literature review. The three steps of the
proposed literature review process are: 1) Inputs, 2) Processing, and 3) Outputs. Figure 1 provides
an overall view of the process proposed. The following three sections of this paper (2. Literature
Review: Inputs; 3. Literature Review: Processing; 4. Literature Review: Outputs) are organized to
follow the three main steps in the proposed effective literature review process.
What is unique about an ISrelated literature review?

Although the literature review process
proposed in this paper may be generalized to any field of social and behav-
182
Figure 1: The three stages of effective literature

review process.
Levy & Ellis
ioral science, it is especially applicable to the challenges inherent in information systems research. The IS literature universe is comprised of diverse, interdisciplinary work (Webster &
Watson, 2002) which may lead novice researchers to concentrate in limited disciplinary sources
when conducting their literature foundations search, while missing some very fruitful work conducted in another sub-discipline within the IS research literature. For example, if a novice IS researcher uses a particular source (i.e. a given literature database from a particular vendor) the
universe of literature explored is limited. Instead, an effective literature search in the IS-related
literature must exhaust all sources that contain IS research publications (i.e. journals, quality conference proceedings, etc.) that are valid to the proposed study. Thus, the work presented here will
provide discussions on issues related to common pitfalls when attempting to seek quality IS research literature.
There is, finally, a great deal of IS literature of varying quality. Such a large number of electronic
and print sources are available that the novice researcher can be overwhelmed in determining
which sources contain accurate, valid information. It is important to address approaches on where
and how to locate quality IS literature.
Why Conduct a Literature Review?

Before examining how to conduct a literature review, one must first understand the place of the
review in research (Webster & Watson, 2002). Thus, two questions must be answered: What is
research? Why is a literature review needed for any quality research endeavor?
Research is defined as an endeavor that scholars intentionally set out to enhance [their] understanding of a phenomenon and expect to communicate what [they] discover to the large scientific
community (Leedy & Ormrod, 2005, p. 4). Two critical considerations stem from this definition:
a) research must enhance the scientific communitys current understanding of a phenomenon, or
contribute to enhance the BoK, and b) research must communicate what was discovered in the
new study to the scientific community. Knowing the current status of the BoK in the given research field is an essential first step for any research project (Iivari et al., 2004). An effective literature review accomplishes this step by:
1. Helping the researcher understand the existing body of knowledge including where excess research exists (i.e. what is already know?) and where new research is needed (i.e.
what is needed to be known?)
2. Providing a solid theoretical foundation for the proposed study (related to what is already known?)
3. Substantiating the presence of the research problem (related to what is needed to be
known?)
4. Justifying the proposed study as one that contributes something new to the BoK
5. Framing the valid research methodologies, approach, goals, and research questions for
the proposed study
The next three sub-sections address the importance of the literature review for quality IS research.
Build a solid theoretical foundation for your study

Developing a solid foundation for a research study is enabled by a methodological analysis and
synthesis of quality literature (Barnes, 2005; Webster & Watson, 2002). One of the main reasons
for conducting the literature review is to enable researchers to find out what is already known.
However, it is important to remember that not everything reported in the literature is of equal
rigor (Ngai & Wat, 2002). When proposing a new study or a new theory, researchers should en-
183
sure the validity of the study and reliability of the results by making use of quality literature to
serve as the foundation for their research (Barnes, 2005). Quality literature stimulates additional
research studies, thus providing validation of the original theory proposed (Barnes, 2005). Straub
(1989) noted that with validated instruments, researchers can measure the same research constructs in the same way, granting improved measurement of independent and dependent variables,
and in the long run, helping to relive the confounding that plagues many streams of MIS literature (p. 148). Building a solid theoretical foundation based on quality resources enables researchers to better explain as well as understand problems and solutions that address actual issues
with which practitioners are struggling.
Conducting an effective literature review that will yield a solid theoretical foundation should also
provide a firm foundation to the selection of the methodology for the study (Ngai & Wat, 2002).
The selection of the methodology should not be interpreted as placing more rigor on one type of
research such as qualitative, quantitative, exploratory or confirmatory, but rather it should enable
the researcher to understand what methodologies were previously validated (Straub, 1989). Thus,
a solid theoretical foundation should also provide researchers the justifications for a given methodology and enable them to provide justifications for why a given approach is optimal for their
study.
Fitting the literature into your research

An effective and quality literature review is one that is based upon a concept-centric approach
rather than chronological or author-centric approach (Webster & Watson, 2002). Bem (1995)
noted that authors of literature reviews are at risk for producing mind-numbing lists of citations
and findings that resemble a phone book impressive case, lots of numbers, but not much plot
(p. 172). Thus, researchers must continuously ask themselves when reviewing literature and when
writing the literature review: how is the work presented in the article I read related to my study?
Answering this question will allow researchers to tie the literature into their own study. Moreover, during the review of the literature researchers should utilize sources that substantiate the
presence of the problem under investigation (Barnes, 2005). Doing so will enable the researcher
to provide a solid argument for the need for their study as well as spot where literature fits into
their own proposed study. Moreover, use of literature should provide the grounds for legitimization of the research questions proposed in the study as well as validate the approach proposed by
the study.
Place your study in the context of existing work (body of knowledge)

Aside from the need to fit a given article into the proposed study, researchers should also address
the first of their proposed study in the context of the BoK. As noted above, one of the main definitional components of research is the ability to add to the current BOK. As such, quality research
must provide justifications for the potential contributions provided by the proposed study. Such
justifications should demonstrate how the proposed research contributes something new to the
overall BoK or advances the research fields knowledge. A classic example of this approach is the
DeLone and McLeans (1992) paper that noted the importance of defining the IS dependent
variable cannot be overemphasized in recognition of this importance, this paper explores the
research that has been done and attempts to synthesize this research into a more coherent body
of knowledge (p. 61). The evidence in the contribution of the classical DeLone and McLeans
(1992) to the BoK was materialized by the stream of research studies conducted following this
paper and was summarized again by them over a decade later (DeLone & McLean, 2003).
184
Levy & Ellis
Literature Review: Inputs

This section will address key issues related to the process of understanding what quality IS literature is as well as the process of gathering manuscripts for an effective literature review, the two
parts of the inputs stage. In any systematic approach, if the system input is either incorrect, of
low quality, or irrelevant, the resulted output is going to be ineffective regardless of the quality of
the processing stage or, colloquially, garbage-in/garbage-out. Webster and Watson (2002) noted
that a systematic search should ensure that you accumulate a relatively complete census of relevant literature (p. 16). In order to avoid the pitfall of garbage-in/garbage-out and produce an
effective literature review, this section will start by addressing issues related to the quality of literature for the review. The discussion will address some common errors of novice IS researchers
as well as provide resources on where to find quality IS literature. Subsequently, this section will
provide specific approaches on how to perform a quality literature search by elaborating on the
technique introduced by Webster and Watson (2002) with specific examples. The section then
examines how to read research literature for the review, and concludes with a discussion of how
to determine when the literature search is approaching completion.
Where to Look for Quality IS Literature?

Sir Isaac Newton (1676) noted if I have seen a little further it is by standing on the shoulders of
giants. Clearly, researchers who are able to take advantage of knowledge and discoveries of
other researchers will be able to expand the BoK further. However, standing on the shoulders of
midgets will not provide much of a horizon for the novice IS researcher to produce quality work.
The quality of the literature used plays a significant role in advancing the knowledge of the researcher and the overall BoK. The novice IS researcher often finds it difficult to determine the
quality of literature. This section is aimed at providing novice IS researchers some tools to understand the quality of IS literature they are reading.
Validating the quality of IS literature

The significance of academic research work is materialized when it is published (Davison,
Vreede, & Briggs, 2005). As such, the academic research community implemented the peerreview process as a mechanism to control quality and eliminate flaws prior to publication of research manuscripts (Davison et al., 2005). The use of the peer-review process is essential as it
ensures that researchers can use published work with confidence, and use the works of others as
stepping stones and corner stones for advancing new concepts and insights (Davison et al., 2005,
p. 969). It is important to note, however, that all published material is not equal in quality. This
problem is especially troublesome in the IS field where descriptions and discussions of new technology often appears in non-refereed work or in questionable sources. The problem is exacerbated by the presence of corporate sponsorship and its impact on research findings (Hozack,
Ranawat, & Rothman, 2003). Thus, when looking for quality IS research literature, special attention should be made for any work that is not peer-reviewed or is practitioners oriented. Although
not totally unacceptable, use of such sources (i.e. professional magazines, newspapers, etc.)
should be restricted to factual information due to the low theoretical background and application
dependency. Therefore, quality IS research literature from leading, peer-reviewed journals should
serve as the major base of literature review as it provides sufficient theoretical background as
well as leads for additional references on the specific subject matter.
Fortunately, IS scholars are continuously conducting analyses of the quality of IS literature
(Saunders, n.d.) and an extensive list of the highly ranked MIS journals is available from ISWorld.org (Hardgrave & Walstrom, 1997; Katerattanakul, Han, & Hong, 2003; Lowry, Romans,
& Curtis, 2004; Mylonopoulos & Theoharakis, 2001; Peffers & Ya, 2003; Rainer & Miller, 2005;
Walstrom, Hardgrave, & Wilson, 1995; Whitman, Hendrickson, & Townsend, 1999). Figure 2
185
Free Full Text Web Access
EBSCOhost
LEA Journals
Blackwell (Synergy)
JSTOR
ACM (Digital Lib)
Thomson (G. Bus, OneFile)
MIS Quarterly
Information Systems Research
Communications of the ACM
Management Science
Journal of MIS
Artificial Intelligence
Decision Sciences
Harvard Business Review
IEEE Transactions (various)
AI Magazine
European Journal of IS
Decision Support Systems
IEEE Software
Information & Management
ACM Transactions on DB Sys
IEEE Trans on Software Eng
ACM Transactions (various)
J of Computer and System Sci
Sloan Management Review
Communications of the AIS
IEEE Trans on Sys, Man, & Cyb
ACM Computing Surveys
Journal on Computing
Academy of Management Journal
Int'l J of Electronic Commerce
Journal of the AIS
IEEE Transactions on Computers
Information Systems Frontiers
Journal of Management Systems
Organization Science
IEEE Computer
Information Systems Journal
Administrative Science Quarterly
J of Global Info Management
The DB for Advances in IS
Journal of Database Management
Information Systems
Academy of Management Review
Journal of the ACM
Computers & Operations Research
Human-Computer Interaction
California Management Review
Information Technology & People
Journal of Strategic IS
Journal of Global IT Management
ACM Transactions on IS
Informing Science
Journal of Information Management
Operations Research
Journal of Computer IS
Wilson (OmniFile)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
IEEE (Comp Soc & Xplore)
(Based on ISWorld)
Journal Name
INFORMS
No.
Elsevier (ScienceDirect)
Ranked MIS Journals
ProQuest (ABI/INFORM)
Literature Vendor (Database)
Figure 2: ISWorlds top 50 ranked MIS journals and electronic availability
186
Levy & Ellis
provides a list of the top 50 ranked MIS journals based on the list generated by Saunders (n.d.) for
ISWorld. Culnan (1978) noted that availability to research articles is critical to the development
of IS as a field. Thus, Figure 2 also provides information on electronic availability of each of the
journals in the list.
One main intention to add such electronic availability is to help novice IS researchers to understand the diverse resources and outlets where quality IS articles can be found. The columns under
literature Vendor (database) in Figure 2 represent the names of vendors that offer articles from
a given journal in their database. The check mark under a specific vendor/database indicates that
this vendor offers either in full text or abstract and citation of such journal articles, but note that
some limitations on availability (i.e. embargo years) may exist.
Webster and Watson (2002) commented on conference proceedings, indicating that IS researchers
should also examine selected conference proceedings, especially those with a reputation for
quality (p. 16). Although conferences are valuable scientific venues for exchange of ideas and a
major incubator for new research agendas, the overall rigor of conference proceedings is lower
than one found in leading journals (Culnan, 1978). Moreover, there appears to be a growing number of conferences that are purely for-profit and run by questionable organizations (Sorenson &
Fleming, 2004). As such, proceedings from these for-profit conferences may not have been subjected to the same peer-review process as those from conferences run by reputable research/professional associations (i.e. AIS, ACM, INFORM, IEEE, AOM, ISI, DSI, etc.). Thus,
7
8
9
10
11
NR
NR
NR
NR
NR
Proc. CD-ROM for purchase
Full Text Web Access (Free)
Full Text Web Access (Fee)
International Conference on Information Systems (ICIS)

Hawaii International Conference on System Sciences (HICSS)
International Federation for Information Processing (IFIP)
International Conference on Decision Support Systems (DSS)
Decision Sciences Institute (DSI) - National Conference
Society of Information Management (SIM) Conference
International Association for Computer Information Systems (IACIS) Conference
(Proceedings published in Issues in Information Systems)
Institute for Operations Research and the Management Sciences (INFORMS)
Conference
Information Resources Management Association (IRMA) Conference
Academy of Management (AOM) Conference
Decision Sciences Institute (DSI) - Regional Conferences
International Academy of Information Management (IAIM) Conference
American Conference on Information Systems (AMCIS)
Information Systems Education Conference (ISECON)
Institute of Electrical and Electronics Engineers (IEEE) National Conferences
Informing Science + Information Technology Education (InSITE) Conference
ACM (Digital Lib)
1
2
3
4
5
6
No.
INFORMS
Ranked and Non-Ranked IS Conferences

Ranked Order is Based on Hardgrave and Walstorm (1997)
Conference Name
Literature
Vendor
(Database)
No. - Indicates the rank of a conference in general by Hardgrave and Walstorm (1997)'s study
NR - Non-ranked
Figure 3: Ranked and non-ranked IS conferences

with electronic availability of proceedings
187
novice IS researchers should attempt to limit the use of conference proceedings to those they
found referenced in articles from quality IS journals or ones list in Figure 3. Unfortunately, unlike
IS journal articles where studies were done and resources are available to indicate some level of
quality, there appears to have been very little attention given to the development of similar rankings for IS conference proceedings (C. S. Saunders, personal communication, February 9, 2006).
Hardgrave and Walstrom (1997) reported a list of top IS conferences as part of their general quest
for reputable outlets of IS research. Although providing an important insight, their ranking was
based on faculty members perceptions of how valuable each conference in general is to the MIS
field, and no specific indications or measures were done for the quality of these conferences proceedings. Moreover, Hardgrave and Walstroms study was done about a decade ago and over that
time new IS-related conferences sponsored by reputable associations have immerged. Thus, additional work should be done to generate a ranked list of IS conference proceedings similar to the
ranking process done for IS journals noted above. Having said that, proceedings from these reputable IS conferences are certainly valid for IS research (Webster & Watson, 2002).
Figure 3 provides the top eleven ranked IS conferences as reported by Hardgrave and Walstrom
(1997). Additionally, non-ranked IS conferences that appear to be valuable for IS scholars were
added to the list. As noted before, more work is needed to validate this list.
Testing for applicability to your study

While searching for quality literature is essential, it is also important to identify articles that are
applicable to the proposed study. The issue of testing for applicability of research literature to the
proposed study has two critical facets. The first deals with the inclusion or exclusion of articles
from the literature review, and the second deals with ethical and unethical use of references. Both
facets should be addressed during the literature search and gathering process to ensure high quality and effective literature review.
Applicability of literature: Occasionally research studies attempt to combine several theories,
constructs, and/or models (see additional discussions about this under the Apply the Literature
below). As such, a literature review for each stream of theory or construct may be relevant. In
such cases, one may find numerous studies on a specific theory, model, or framework. Thus, deciding which piece of literature should be included as part of the literature review can be a daunting task for novice researchers. One common example is the extensive work done on the Technology Acceptance Model (TAM) by Davis (1989) and colleagues (Davis, Bagozzi, & Warshaw,
1989; Venkatesh & Davis, 1996, 2000). When novice IS researchers produce a search on TAM
related constructs, they encounter numerous studies. At that point, novice IS researchers tend to
ask themselves: should all studies and articles dealing with that theory, model, and/or construct
be included in my review? The approach advocated here is: no, each literature piece should constantly be evaluated on how applicable it is to the proposed study. If an article or study is only
remotely relevant, it may be used as a support for a particular argument, but should not be used as
foundation literature to the proposed study. Thus, only the applicable literature articles that are
relevant to build the theoretical foundations for the validity of the theories, constructs, and measures should be noted. Clearly the key fundamental studies that established the validity and reliability of these theories, constructs, and measures are cornerstones for the discussions and must
be included, like Davis (1989), Davis et al. (1989), and Venkatesh and Davis (1996, 2000) are
key for a TAM review.
Ethical and unethical use of references: Ethical behavior has been a concern for IS scholars
(Couger, 1989). Ethical behavior is defined as being in accordance with rules or standards for
right conduct or practice (Laband & Piette, 2000, p. 24). Unethical behavior constitutes a violation of such conduct or practice. The accurate and valid use of literature should be maintained to
the highest ethical standard (Kock, Davison, Clarke, & Loch, 2000). Unethical use of references
188
Levy & Ellis
includes taking material out of context or even intentional misstatements (Laband & Piette, 2000;
Mason, Bearden, & Richardson, 1990). Such unethical behavior eventually leads to a major devastation to these scholars career, while eroding the credibility of the IS field as a quality research
(Kock et al., 2000). Thus, in order to maintain the integrity of the IS field and protect individual
reputation, IS researchers must ensure that references used in a proper context and with high degree of confidence.
How to Find Quality IS Literature?

A literature search is the process of querying quality scholarly literature databases (i.e.
ABI/INFORM, JSTORE, Elsevier/ScienceDirect, WilsonWeb, etc.) in order to gather applicable research manuscripts related to the phenomenon under investigation. It is at this stage
that many novice researchers struggle with locating appropriate quality sources to query. Webster
and Watson (2002) suggested that the major contributions are likely to be in the leading journals.
It makes sense, therefore, to start with them [researchers] should also examine selected conference proceedings, especially those with a reputation for quality (Webster & Watson, 2002, p.
16). Thus, it is the aim of this section to provide guidelines on how IS researchers should initiate
and conduct an effective literature search. Moreover, due to the interdisciplinary nature of IS literature and the diversification of IS literature outlets, this section will also address issues related
to the various literature database and electronic resources. Additionally, this section will provide
details on several techniques to help novice researchers, in particular IS novice researchers, conduct a more effective literature search. Moreover an attempt will be made to focus such discussion specifically in the context of IS research.
Literature databases & electronic resources

With the increased use of electronic resources for library services, the speed at which researchers
can find relevant literature increased dramatically (D. Shaw, 1995). However, novice researchers
should be fully competent in using such electronic resources in order to produce an effective literature review. As noted previously, one common mistake of novice researchers is attempting to
exhaust a literature search using one or two literature database vendors (i.e. ABI/INFORM,
JSTORE, Elsevier/ScienceDirect, WilsonWeb, etc.), primarily using a keyword search approach. This method leads to two main limitations: a) very narrow literature background, and b)
shallow depth of literature background.
The narrowness of the literature background is in terms of the number of journals or other resources used. By using one or two vendors in the literature search process, the literature revealed
is limited to only the journals and electronic resources provided by such vendors. This narrowness of searching can cause a novice researcher to obtain only partial knowledge about a phenomenon. This problem is particularly acute in the IS domain due to the large dispersion of quality IS literature over hundreds of databases and numerous literature vendors. It is important for
the novice researchers, particularly novice IS researchers, to spend time expanding their literature
seeking skills beyond a given vendor, learning how to conduct a literature search utilizing multiple literature database vendors resources.
Webster and Watson (2002) noted that a systematic search should ensure that you accumulate a
relatively complete census of relevant literature (p. 16). Thus, the next three sections provide
additional details on specific techniques to be used when conducting an effective literature search.
The first technique is meaningful keyword searching, followed by backward searching, and finishing with forward searching. Mastering all three techniques is key for novice IS researchers.
due to the diversification and multidisciplinary nature of IS literature in order to exhaust all literature sources on specific phenomenon, one must be able to utilize all techniques in order to obtain
a wide literature background. Likewise, using all such techniques together should ensure that
189
novice IS researchers do not fall into the trap of limited depth of literature background that results
from using only IT buzzwords.
Keywords search
Keyword search refers to the querying of quality scholarly databases by the use of a specific word
or phrase (i.e. keyword) when attempting to find relevant literature. Keywords can be searched
against several categories such as the documents keywords, title, and abstract. However as noted
above, such an approach may have several limitations, in particular for an IS literature search
where keywords are often related to specific technologies.
Keyword searching presents a classic cold-start problem for the novice researcher; how can one
identify the applicable keywords for an unknown domain. The best source for keywords is, of
course, the literature base for the domain and all articles reviewed should be read with an eye for
potential keywords. Locating the first article can, however, be a considerable challenge. The Association for Computing Machinery (ACM) classification system (http://www.acm.org/class/1998/homepage.html) and MIS Quarterly Roadmap (http://www.misq.org/roadmap/code/level2/h.html) offer the novice researcher a workable starting point.
The keyword search should be just the initial, not the main step for a literature search. A common
mistake by novice researchers, specifically in IS, is to assume that the keyword search yields all
that is available from the literature. Clearly one of the aims of this paper is to address this misconception. The limited depth of the literature background is usually manifested by the use of
only nave keywords for the literature search. Novice IS researchers tend to stick to specific keywords when conducting their literature search. However, due to the nature of the IS field and the
tremendous progress in technology, keywords of IS literature tend to have a limited life span. An
example of this keyword evolution is way the term Manufacturing Resource Planning or Material
Requirements Planning (MRP) systems in the 1970s-1980s became Enterprise Resource Planning
(ERP) systems in the 1990s. The novice IS researchers may report that the theoretical issues surrounding ERP systems have limited evidence, not being aware that one may find an abundance of
work conducted about similar issues with MRP systems.
Another issue with the keyword search process is the use of technology specific terms or buzzwords that appear and disappear from literature. An example can be found in the term phishing
(i.e. phishing e-mail scam) from information security and assurance. The term phishing is a
form of computer crime. If a novice IS researcher attempts to learn more about the phishing phenomenon, searching using this keyword may provide little or no prior work in scholarly literature.
However, attempting to generalize the term and avoiding use of buzzwords would reveal that
much work was done in the area of computer crime that may be very valid and applicable
(Webster & Watson, 2002).
Although most buzzwords appear and disappear from the literature over time, the underlying
theories and theoretical constructs are more stable (Robey, Boudreau, & Rose, 2000). It is, therefore, necessary to go beyond keywords and use the backward and forward approaches, which can
help researchers to follow models, theories, theoretical constructs, and research streams.
Backward search
The idea of backward and forward search originated from Webster and Watsons (2002) article
where they propose conducting backward and forward lookout of IS literature. This section and
the following one will elaborate on such approaches by providing a step by step technique on how
to conduct it in order to achieve higher quality literature search results, i.e. more effective results.
Using an effective keyword search will produce some initial insight into the domain one wishes
to study. If these leads are from quality resources (see Where to look for quality IS literature?
190
Levy & Ellis
above on discussion of literature quality), additional steps should be used to advance the literature
search and the researchers knowledge about the phenomenon under study. One such step deals
with backward literature search. The process of going backward in literature can be divided into
three specific sub-steps: backward references search, backward authors search and previously
used keywords.
Backward references search refers to reviewing the references of the articles yielded from the
keyword search noted above. For example, if one is studying the construct of computer selfefficacy, hopefully in their keywords search the researcher will stumbled upon Compeau and
Higgins (1995)s pioneering article. Conducting a backward references search for computer selfefficacy, will mean to pull out and review all the relevant references of Compeau and Higgins
(1995)s article such as Bandura (1977) or Barling and Beattie (1983) articles. Doing such backward references search provides researchers with the ability to learn more about the origins of the
construct, theory, or model under study. Moreover, a second level backward references search
should be done by pulling the references of the references. Doing so will enable the researcher
to extend his or her knowledge even deeper on the phenomenon under study. Additionally, it may
also provide the researcher an ability to find inconsistencies in literature as well as cases where
references and quotes might have been taken out of context. Following this approach a third,
fourth, or more levels of backward references search can be done to expand even further the
knowledge.
Backward authors search refers to reviewing what the authors have published prior to the article.
Most researchers in the IS field tend to conduct studies within a rather narrow phenomenon domain. Thus, querying an authors prior work may yield fruitful information. For example, if one
is studying the construct of end user computer satisfaction (EUCS) and came across Doll and
Torkzadeh (1991)s work, a backwards authors search would also yield additional studies that the
two authors conducted that have led them to the development of the EUCS construct.
Previously used keywords refers to reviewing the keywords noted in the articles yielded from the
keyword search noted above. For example, in articles related to the study of IS effectiveness there
are several classical manuscripts (Grover, Seung, & Segars, 1996; Pitt, Watson, & Kavan, 1995;
Srinivasan, 1985) that note the term measurement as an important keyword related to such work.
As such, by looking at articles that carry the term measurement as a keyword in previous literature pieces, one may find additional fruitful manuscripts in the literature search on the subject of
IS effectiveness.
Forward search
The third step in the process exploring the knowledge base for references about the phenomenon
deals with forward literature search. Similar to the process of the backward literature search, the
forward search can be divided into two specific sub-steps: forward references search, and forward authors search.
Forward references search refers to reviewing additional articles that have cited the article. For
example, a forward reference search for Compeau and Higginss (1995) article can be accomplished by conducting an electronic library database search for all articles that include either
Compeau or Higgins in their citations. Doing so will enable to researcher to extend their
knowledge even further by locating follow-up studies or newer developments related to the phenomenon under study.
Forward authors search refers to reviewing what the authors have published following the article. For example, a forward authors search for Compeau and Higginss (1995) article can be accomplished by conducting an electronic library database search for all articles that include either
Compeau or Higgins as the author. Doing so will enable the researcher to extend even more
191
their knowledge about the authors by identifying improvements or new findings related to the
phenomenon under study.
How to Tell When You Are Done With the Literature Search?
The literature search is an evolving process in which the researcher is digging into the literature
by uncovering additional literature that may be applicable for the proposed study. Such a digging
process was previously noted in several metaphors. Some of the key metaphors noted by IS
scholars via the ISWorld LISTSERV include: funnel (funneling in), concertina (narrowing and
enlarging your search, like a concertina windbag), and lens (to focus readers) (Metcalfe,
2002). Figure 4 illustrates these three key metaphorical approaches.
By default, the literature search process should continuously be done during the course of the
study. As noted previously, the use of backward and forward literature search techniques should
provide additional valid references as the search progresses. However, it may appear to novice
researchers that this process is a never-ending one. From a practical perspective, one needs to
stop the search and move on to the processing and writing (output) of the literature review. Thus,
the question arises: At what point should the process of gathering additional relevant literature
end?
Leedy and Ormrod (2005) noted that one common rule of thumb is that the search is near completion when one discovers that new articles only introduce familiar arguments, methodologies, findings, authors, and studies. Thus, when reading a new literature piece, if one will get the feeling
that Ive seen this (or something similar to it) before (Leedy & Ormrod, 2005, p. 82), it may
suggest that the literature search is near completion. The end of the search can also be indicated
when no new citations are discovered and articles cited in newly discovered literature have already been reviewed. In sum, as Webster and Watson (2002) observed: You can gauge that your
review is nearing completion when you are not finding new concepts in your article set (p. 16).
Figure 4: Key metaphors for literature search process
Literature Review: Processing

Locating applicable peer-reviewed articles is certainly a necessary condition for a literature review (J. Shaw, 1995); it is not a sufficient condition. The data contained in the sources identified
must be processed into information that can serve as a foundation upon which new research can
be built (Bem, 1995). Accomplishing this processing entails sophisticated cognitive activity. Although the methodology for evaluating the results of that cognitive activity has been explored
rather thoroughly (Boote & Beile, 2005; Hart, 1998), the ways and means for actually accomplishing the necessary processing is less clearly understood (Wu, 2005). How can the new or nov-
192
Levy & Ellis
ice researcher learn to effectively use the articles he or she locates to build the necessary foundation?
There is certainly no shortage of theories regarding human learning (Gagne, Briggs, & Wager,
1992; Jonassen, Tessmer, & Hannum, 1999). Blooms Taxonomy of Educational Objectives
(i.e. Blooms Taxonomy) (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956) has been shown
to both effectively describe the learning process and offer meaningful insight into promoting development within the cognitive domain (Andrews & Wynekoop, 2004; Manton, Turner, & English, 2004; Noble, 2004; Zahn, Rajkumar, & Zahn, 1996). Two concepts are integral to the
Blooms Taxonomy: cognitive capability is a developmental process that can be tracked through
a series of steps, and each step of cognitive development can be identified by a number of specific
types of behaviors. In effect, the Blooms Taxonomy provides a set of sequential steps, each of
which requires gradually more cognitively demanding activities that the researcher should do in
developing the skill to transform the raw data of numerous literature sources into an effective literature review. The following sections provide a review of each of the Blooms Taxonomy steps,
with emphasis on the sequential process that a given step is as the foundation for the followed
step.
Know the Literature

The knowledge level is commonly demonstrated by activities such as listing, defining, describing,
and identifying. At the very least, the researcher must demonstrate that he or she has read the article and extracted meaningful information from it. Figure 5 presents an example of a citation that
points to the literature but does not demonstrate mastery at the knowledge level. Although the
citation certainly tells that Nunamaker, among others, had something to say about individual and
group marks for in-group activities, the reader really dont have any idea what these references
said.
Other research also indicates that individual and group marks should be combined in-group
activities (Buchy & Quinlan, 2000; Lim et al., 2003; Romano & Nunamaker, 1998).
Figure 5: Pointing at the literature
Figure 6 presents this information in a manner that demonstrates knowledge level mastery of the
material. From this example it is clear that the citation provides some germane facts about the
Buchy and Quinlan article: it was a report of some research, that the research was qualitative in
nature, and that one of the conclusions from that qualitative research was that students participating in tutorial groups reported that the group activity made them more attuned to the learning
process.
Buchy and Quinlan (2000) interviewed 36 students participating in tutorial groups. These interviews indicated that the students felt they were becoming more conscious of learning processes of both themselves and their peers.
Figure 6: Knowledge-level mastery
Comprehend the Literature

Comprehension is demonstrated by activities such as summarizing, differentiating, interpreting,
and contrasting. At this level of mastery the researcher demonstrates that not only can he or she
193
repeat what was included in the article but also knows the meaning and significance of the information being reported. Figure 7 illustrates a citation that presents un-interpreted bits of fact.
Han and Kamber (2001) suggest an evolution that moves from data collection and database
creation, towards data management, and ultimately, data analysis and understanding.
Figure 7: Pre-comprehension level mastery
Although the citation in Figure 7 certainly indicates the point raised by Han and Kamber, it really
does not demonstrate mastery over anything beyond a set of buzz-words. The amplified citation
displayed in Figure 8 demonstrates an understanding of the concepts presented by Han and Kamber.
Han and Kamber (2001) suggest an evolution that moves from data collection and database
creation, towards data management, and ultimately, data analysis and understanding. For example, data processing is a base function enabling manipulation and aggregation of data, thus
facilitating searching and retrieval.
Figure 8: Comprehension-level mastery
Cognitive/construct-level
Once the novice researcher starts reading scholarly literature it is often difficult for them to comprehend and thus apply the cognitive level addressed in such manuscripts. This is especially problematic for novice researchers that have extensive experience as practitioners but lack experience
with scholarly research. There are several common terminologies used by scholars that warrant
definition, elaboration, and clarification in order to help novice researchers comprehend scholarly
work. As the following sub-sections will attempt to provide a review of such key terminology
including discussion on what a theory is, in general, and provide a specific list of theories used in
IS research. The following sub-section will discuss theoretical constructs and provide a specific
list of constructs used in IS research. Finally, another sub-section will address theoretical models
or frameworks and provide a specific list of models or frameworks used in IS research.
Theory: Definition and Use in IS Literature

The main essence of theory is to provide an explanation of observed phenomena (Kerlinger &
Lee, 2000, p. 11). Additionally, researchers suggest that theories are tentative explanations of reality and that each theory needs to be validated empirically to determine how well it represents
reality (Kerlinger & Lee, 2000). However, it appears that there are some inconsistencies in the
literature about the definition of theory (Thomas, 1997). Kerlinger and Lee (2000) define theory as a set of interrelated constructs (concepts), definitions, and propositions that present a systematic view of phenomena by specifying relationships among variables, with the purpose of explaining and predicting the phenomena (p. 11). As such they noted that theory has three main
components. The first component of theory includes propositions or hypotheses that are based on
clearly defined and interrelated constructs (or concepts). The second component of theory includes a representation of a systematic view of phenomena by a notation of specific relationships
among a set of constructs (or variables). The third component of theory includes an explanation
of phenomena, mainly in order to help make predictions. Creswell (2003) defines theory as a
scientific prediction or explanation (p. 120). However, literature suggests that theory is used differently in qualitative than in quantitative research (Sekaran, 2003). Quantitative research tends to
194
Levy & Ellis
use theory for deductive purposes or testing generalized perspective, or theory, of a specific phenomenon in a specific context. Qualitative research, on the other hand, tends to use theory for
inductive purposes or developing a theory based on specific observed phenomenon (Sekaran,
2003). Leedy and Ormrod (2005) define theory as verbal statement, visual model, or series of
hypotheses offered to explain the phenomenon in question (p. 155).
Theory
-
Absorptive capacity
Actor-network theory
Adaptive structuration theory
Administrative behavior, theory of
Agency theory
Argumentation theory
Chaos theory
Cognitive dissonance theory
Cognitive fit theory
Competitive strategy (Porter)
Complexity theory
Contingency theory
Critical social theory
Diffusion of innovations theory
Dynamic capabilities
Evolutionary theory
Expectation confirmation theory
Feminism theory
Game theory
General systems theory
Hermeneutics
Illusion of control
Information processing theory
Institutional theory
Theory
-
Knowledge-based theory of the firm

Media richness theory
Organizational information processing theory
Organizational knowledge creation
Punctuated Equilibrium Theory
Real options theory
Resource-based view of the firm
Resource dependency theory
Self-efficacy theory
SERVQUAL
Social exchange theory
Social cognitive theory
Social network theory
Social capital theory
Socio-technical theory
Soft systems theory
Structuration theory
Task-technology fit
Technology acceptance model
Theory of planned behavior
Theory of reasoned action
Transaction cost economics
Transactive memory theory
Unified theory of acceptance and use of technology
Source: Schneberger and Wade (2006): http://www.istheory.yorku.ca/ via ISWorld.net
Figure 9: Main theories used in IS research, adopted from ISWorld

A good metaphor provided to novice IS researchers about theory is the analogy to the house
foundation. In order to have a solid structure, one must have a very solid foundation even if it
cannot be seen. Therefore, it is observed in the literature that theory is often noted as the foundation of the research or theoretical background indicating the fundamental building block for any
research. Figure 9 provides a list of common IS theories. It appears that in the context of IS there
are theories that serve as a foundation for a single construct (i.e. Self-efficacy Theory) and there
are theories that serve as foundation for whole model (i.e. TAM).
Representation of theory appears to be a challenging task. However, Creswell (2003) noted that
researchers tend to represent theory in several forms. The first form of theory representation deals
with a set of propositions or hypotheses and explanations of their interrelationships. A second
form of theory representation, noted by Creswell (2003), deals with a set of causal relationships
between constructs and variables. The third representation of theory is by visual models or conceptual maps. Such an approach is used primarily in a causal modeling to help researchers communicate their verbal theories into visual maps (i.e. theoretical or conceptual maps) that represent
the interrelationships among constructs or variables (Creswell, 2003). These theoretical maps also
often include influential directions noted with + or - among the arrows within the map.
195
Constructs/variables: Definition and Use in IS Literature

Many researchers note the term construct similarly to the term concept. However, there is a
slight distinction between the two. Kerlinger and Lee (2000) noted that a concept expresses an
abstraction formed by generalization from particulars, whereas a construct is a concept with
Construct
-
Acceptance of Quantitative System

Adequacy of Training
Affect of Using Personal Computers
Application Systems Skills
Attending to Interpersonal Relations
Attending to Production And Targeting Work Behavior
Attitude Toward Alternative Media
Attitude Toward Usage
Behavioral Intention for Usage
Behavioral Problems
Budget in Is Devoted to Training
Capabilities of The Planning System
Career Orientation
Career Satisfaction
Chargeback Information Use
Commitment to Ic Concept
Communication in Meetings
Compatibility
Compatibility of An Innovation With Experience
Competitive Strategy
Complexity of Personal Computers
Computer Abuse
Computer Self-Efficacy
Computer User Satisfaction
Conflict As It Affects Systems Development
Conflict Resolution And Outcomes in Systems Development
Coordination of End-User Computing
Cost Variability
Credibility of Computer System
Decentralization
Decision Making Satisfaction
Demonstrability of Results With Innovation
Design Considerations for System
Designer Perceptions of User Shortcomings
Desired Involvement in Computing Activity
Deterrent (To Computer Abuse) Certainty
Deterrent (To Computer Abuse) Severity
Development of New Systems
Discrepancies Between Needs And Features
Discussion Quality in A Meeting
Duration of Cbis
Ease of Use
Ease of Use of Innovation
Ease of Use of Software Package
Economic Value Dimensions for Isd
Efficacy
Efficiency of Meeting Processes
Employees Devoted to Computer-Related Training
End-User Computer Literacy
End-User Computing Satisfaction
Construct
-
End-User Involvement
End-User Satisfaction With Computing Activity
Engagement
Enjoyment in Using A Software Package
Enriching The Job
Environmental Factors of Security
Equity Perception in The Allocation of Is Resources
Executive Involvement in Management of IT
Executive Participation in Management of IT
Expectations About System
Experience With The System
Extent of Boundary-Spanning Activities
Extent of Fulfillment of Key Planning Objectives
Facilitating Conditions -- Resources
Facilitating Conditions -- Technology
Facilitating Conditions of The Use of A Personal Computer
Facilitating End-User Computing
Factors Contributing to Job Satisfaction
Flow
Global User Satisfaction
Health Related Behavior
Image of Innovation Adopter
Impact of Technology
Implementation Failure
Importance of Attributes of Is for Success
Importance of Job Roles
Importance of Skills
Influence in Systems Development
Information Center Support
Intention to Leave Current Organization
Interpersonal Communication About An Innovation
IT Structure
Job Complexity
Job Fit of Personal Computers
Job Satisfaction
Job-Determined Importance (Need) for An Innovation
Level of System Utilization
Long Term Consequences of Using A Personal Computer
Management Quality Interactions
Measuring IS's Effectiveness
Microcomputer Playfulness
Motivational Factors
Nature of Work
Nonsalary Incentives
office Size Interactions
office Type Interactions
Organization Characteristics
Organizational Commitment
Package Characteristics
Participation in Design of Computer-Based Is Reports
Figure 10: Main constructs used in IS research, adopted from ISWorld
196
Levy & Ellis

Construct (Cont.)
-
Participation in Systems Development

Peer Influences
Perceived Behavioral Control Over Usage
Perceived Cohesion in Groups
Perceived Decision Making Purposes
Perceived Disorientation
Perceived Ease of Use
Perceived Future Needs for Information Systems
Perceived Importance of Computer-Related Abilities
Perceived Involvement in Computing Activity
Perceived Job Characteristics
Perceived Management Support for An Innovation
Perceived Medium Richness
Perceived Proficiency in General Is Knowledge
Perceived Proficiency in Is Product Knowledge
Perceived Proficiency in Knowledge About The Organizational Un
Perceived Proficiency in Organizational Knowledge
Perceived Proficiency in Organizational Skills
Perceived Proficiency in Technical Skills
Perceived Usefulness
Perceived Usefulness of A DSS
Perceived Usefulness of General Is Knowledge
Perceived Usefulness of Is Product Knowledge
Perceived Usefulness of Knowledge About The Organizational U
Perceived Usefulness of MIS
Perceived Usefulness of Organizational Knowledge
Perceived Usefulness of Organizational Skills
Perceived Usefulness of Technical Skills
Perceptions of Qualitative System
Perceptions of Quality of Training
Performance Impact of Computer Systems
Personal Innovativeness Towards An Innovation
Physical Accessibility of An Innovation
Planning System Success
Preferred Form of Organizational Structure
Pressure to Use The System
Preventive Measures
Processing Needs
Productivity
Progressive Use of It in The Firm
Quality of Billing Information
Quality of Ic Support Services
Quality of Working Life
Relationships Between Job Characteristics And Data Required
Relative Advantage of An Innovation
Reliance On Evaluation System Measures
Rival Explanations (For Computer Abuse)
Role Ambiguity
Role Clarity
Construct (Cont.)
-
Role Stressors
Skill in Software
Skill On A Task for Which An Innovation Could Be Used
Social Factors Affecting Use of Personal Computers
Socio-Political Value Dimensions for Isd
Software Efficacy Beliefs
Source of Computer-Related Training
Stage of Growth
Stages of Ic Evolution
Status Effects Experienced in Meetings
Strategic Orientation of Business
Strength of Management Message About An Innovation
Stress
Stressors
Subjective Importance of A Task for Which An Innovation Could Be Used
Subjective Norm of Usage
Success of Implementation
Superior Influences
System Usage Related to Software Package
Systems Analysis Availability
Task Characteristics
Task Equivocality
Task Interdependence
Task-Technology Fit (Including Sub-Factors)
Team Work in A Meeting
Technical Value Dimensions for Isd
The Effects of A Technological Environment
Time of Implementation Interactions
Trialability of Other Innovations
Turnover Intentions Among Employees
Type of Application
Use And Satisfaction With The System
Use of Computer-Based Systems
Use of Different Training Methods
Use of Project Teams
Usefulness
Usefulness of Software Package
User Abilities
User Accountability
User Authority
User Data Requirements
User Information Satisfaction
User Involvement
User Satisfaction With Ic
Value Placed On Production And Services
Visibility of Innovation
Voluntariness of Innovation Adoption
Work Attitudes And Intentions
Work Environment Effects On Employees
Role Conflict
Source: Newsted, Huff, and Munro (2006): http://www.ucalgary.ca/~newsted/constructs.htm via ISWorld.net
Figure 10: Main constructs used in IS research, adopted from ISWorld (Cont.)
added meaning deliberately and consciously invented or adopted for a special scientific purpose
(p. 40). It was also noted that many researchers (including some of the definitions provided above
for theory), appear to suggest that constructs are also called variables (Kerlinger & Lee, 2000).
However, also here there is a slight distinction between the two. Kerlinger and Lee noted that a
variable is a property that takes on different values (p. 40) or appears to be observable when
operationally defined (p. 54). Whereas a construct is a type of variable that is unobservable
197
(Kerlinger & Lee, 2000, p. 54). Thus, in order to represent a construct, researchers proposed a
special expression to indicate such unique type of unobserved variable by using the term latent
variable. A latent variable is a scientific representation of a construct that can only be assessed
indirectly (Meyers, Gamst, & Guarino, 2006, p. 28). From both the definition of construct and
latent variables it is apparent that there is no direct way of measuring a construct, however, an
indirect way to measure should exist. Meyers et al. (2006) noted that in research, constructs or
latent variables can be indirectly measured based on a weighted combination or composite of
multiple measured variables (p. 30). (For additional discussion about concepts, constructs and
variables refer to Kerlinger and Lee (2000) chapter 3 Constructs, Variables, and Definitions as
well as Meyers et al. (2006) pages 28-31). Figure 14 provides a list of commonly used constructs
in IS literature.
Models/frameworks: Definition and Use in IS Literature

A model or theoretical framework is a generalized type of theory that indicates relationships between constructs or latent variables. Hart (1998) suggested that models are generalized theories
that enable more things to be explained using a unified approach (p. 83). Therefore, it is common to see in IS literature the use of the term model synonym to the term theoretical framework.
Figure 11: IS universe, models/frameworks, and constructs

Sekaran (2003) noted that a theoretical framework is a conceptual model for how one theorizes
or makes logical sense of the relationships among the several factors that have been identified as
important to the [research] problem (p. 87). Sekaran defined models or theoretical frameworks
as a logical development, described, and explained network of associations among variables of
198
Levy & Ellis
interest to the research study (p. 97). Moreover, he noted that a model or theoretical framework
elaborates the relationships among the variables, explains the theory underlying these relations,
and describes the nature and direction of the relationship (p. 97). Additionally, he noted that
models and theoretical framework are used to conceptualize a phenomenon by suggesting influences or relationships among constructs or variables. Figure 11 provides a visualization of the IS
universe including representation of how theories, constructs, and models or theoretical frameworks are interconnected. The discussion of theoretical framework/models and constructs is key,
because during the literature comprehension process researchers should identify the variables
that might be important as determined by previous research findings [and] in addition to other
logical connections that can be conceptualized, forms the basis for the theoretical model (p. 97).
Apply the Literature

Application is demonstrated by activities such as demonstrating, illustrating, solving, relating,
and classifying. In the context of the literature review, application is most directly revealed by the
two-step process of: a) identifying the major concepts germane to the study and b) placing the
citation in the correct category. Table 1, adapted from Webster and Watson (2002), illustrates the
activities necessary to demonstrate mastery at the application level following the concept-centric
approach discussed previously.
Table 1: Application-level mastery
Article 1
Article 2
Article n
Concept 1
X
Concept 2
Concept n
X
X
X
X
X
Analyze the Literature

Analysis is demonstrated by activities such as separating, connecting, comparing, selecting, and
explaining. In essence, analysis entails identifying why the information being presented is of importance. Figure 12 illustrates a citation that presents the facts from the literature without the necessary analysis.
Data mining is the analyzing and interpretation of large amounts of information. Through analyzing vast amounts of data it is possible to find patterns, relationships and from these discoveries it is possible to make correlations (Chen & Liu, 2005).
Figure 12: Knowledge without analysis
Left unanswered by this citation is an insight into why it would be of any interest or value to find
patterns and relationships in order to make correlations. Figure 13 presents a modification to the
citation that does provide that analysis.
199
Data mining is a process of discovering new knowledge by using statistical analysis to identify
previously unsuspected patterns and clustering in large data sets (Chen & Liu, 2005).
Figure 13: Analysis-level mastery
Synthesize the Literature

Synthesis entails activities such as combining, integrating, modifying, rearranging, designing,
composing, and generalizing. The essence of synthesis is to assemble the literature being reviewed for a given concept into a whole that exceeds the sum of its parts. Figure 14 illustrates a
discussion in which facts are presented as almost a series of isolated bullet points. Figure 15
presents that same information in a well-synthesized discussion in which the research from a
number of sources is very effectively woven together.
The Digital Object Identifier (DOI) is an Internet-based system for global identification and
reuse of digital content (Paskin, 2003). It provides a tracking mechanism to identify digital assets (Dalziel, 2004). The DOI is not widely employed across LOR and databases and is not
universally adapted by content owners (Nair & Jeevan, 2004). The DOI does not provide provision for assets to be tagged with copyright information (Genoni, 2004).
Figure 14: Lack of synthesis
One current DRM initiative, the Digital Object Identifier (DOI), is an Internet-based system for
global identification and reuse of digital content, and provides a tracking mechanism to identify
digital assets (Paskin, 2003; Dalziel, 2004). However, despite being integrated in learning object technologies, this DOI is not widely employed across LOR and databases, nor is it universally adapted by content owners (Nair & Jeevan, 2004). Similarly, while most metadata schema
enables assets to be tagged with copyright information, this method lacks technological enforcement (Genoni, 2004).
Figure 15: Synthesis-level mastery
Evaluate the Literature

Evaluation connotes activities such as assessing, deciding, recommending, selecting, judging,
explaining, discriminating, supporting, and concluding. The essential evaluation in the literature
review is to clearly distinguish among opinions, theories, and empirically established facts. Citations such as the one displayed in Figure 16 do not indicate if the material from the literature has
been evaluated in any way.
Data mining has applicability to education as well as business (Sanjeev, 2002; Ma et al., 2000;
Glance et al., 2005; Abe et al., 2004; Liu et al, 2005).
Figure 16: Non-evaluated citations
200
Levy & Ellis
the applications of data mining fall under the general umbrella of business intelli-gence.
Case studies have reported implementation of data mining applications for: (1) Enrollment
management (to help capture promising students) (Sanjeev, 2002); (2) Alumni management (to
foster donations and pledges) (Ma et al., 2000); (3) Marketing analysis (to better allocate the
marketing funds) (Glance et al., 2005); and (4) Mail campaign analysis (to judge its effectiveness and design new, better targeted mailings) (Abe et al., 2004). Based upon the similarity to
applications within the business community, Liu et al (2005) speculated that data mining could
also be used within the educational community for fraud analysis and detection.
Figure 17: Citations demonstrating evaluation
The material presented in Figure 17, on the other hand, does demonstrate an element of evaluation. The author clearly identifies the type of information being presented case study reports in
the first four citations, opinion in the fifth citation.
Literature Review: Outputs

Hart (1998) noted that the literature review as a piece of academic writing must be clear, have a
logical structure and show that you have acquired a sufficient range of skills and capabilities at
the appropriate level (p. 172). In order to produce such a piece of academic writing, novice researchers should learn about the proper development of argumentation coupled with issues associated with the actual writing of the academic piece, i.e. the literature review. Therefore, the following section provides a review of argumentation theory and provides examples for proper argument writing. Following, a discussion on some myths associated with writing and a discussion
on suggestions for proper writing structure of the literature review are provided. Lastly, a summary of the impact of the body of literature as a whole on the proposed study is provided.
Writing Arguments and Argumentation Theory

Toulmin (1958) proposed a model for argumentation that is being used in the legal environment
for the development of arguments toward case presentations in courts (Walton, 2006). Williams
and Colomb (2003) suggested a similar approach of argumentation process when writing research
manuscripts. The core of the argumentation theory is a problem that motivates the research study.
The problem is addressed by a claim put forth by the study, combined by the support or a reason
Figure 18: Williams and Colomb (2003, p. 42)s argument model
201
to such claim (or backing as noted by Toulmin). A claim is an arguable statement (Hart, 1998,
p. 88). As such, a claim in the context of research study is an arguable statement that proposes a
solution to the problem (Williams & Colomb, 2003, p. 38). Hart (1998) suggests five different
types of claims in research that include: claims of fact, claims of value, claims of concept and
claims of interpretation. He noted that the range in the types of claims from which an argument
can be constructed shows that almost everything is arguable (Hart, 1998, p. 89).
In order to anchor the argument processes around a proposed problem, Williams and Colomb
(2003) suggest using evidence and warrants (or qualifiers and warrants as noted by Toulmin).
Hart (1998) defines evidence as information and/or data used to support the claim (p. 88). Williams and Colomb (2003) noted that reasons are different from evidences in the fact that reasons
are things people construct in their mind in order to provide rational for the argument, while evidences are the facts available somewhere that one can point to in order to anchor the argument.
Additionally, Hart (1998) defined warrant (or permit) as an expectation that provides the link
between the evidence and the claim (p. 88). Williams and Colomb noted that warrants are a
somewhat difficult issue to comprehend. However, they noted that a warrant should consist of
two parts one part names a general circumstancesthe second part states a general conclusion
that [one] can infer from that circumstances (pp. 184-185). They provide the following example
to illustrate the structure of argument: when an institution has its most eminent faculty teach
first-year classes, it can justly claim that it puts its educational mission first [warrant]. We have
tried to make our undergraduate education second to none [claim] by asking our best researchers
to teach first-year students [reason]. For example, Professor Kinahan, a recent Nobel Prize winner
in physics, is now teaching Physics 101 [evidence] (p. 41).
Figure 19: Toulmin (1958)s structure of an argument, adopted from Hart (1998)
Figure 18 provides an overview of the argument model proposed by Williams and Colomb
(2003). A proper argument process should follow the sequence of: [claim] because of [reason]
based on [evidence] whereas a warrant serves to connect a claim and its supporting reason (p.
41). Figure 19 provides another example of the argument model, this one proposed by Toulmin
(1958). Both examples share several similarities, however, the Williams and Colomb (2003)
model uses the claim as the initial step in the argumentation process, whereas the Toulmin (1958)
202
Levy & Ellis
model uses the claim as the ending step in the argumentation process. Both argument methodologies are equally valid, therefore, novice IS researchers can adopt either model in their writing
structure of the literature review section. Additionally, Figure 20 provides a review of the key
points needed to be considered when writing the literature review in order to develop a sound argument following the guidelines propose by Hart (1998).
use a reliable structure that is explicit following proper argumentation.
define the terms you will use carefully with clear examples and
Definition
backed by quality peer-reviewed sources.
provide reason for everything you have included as support.
Reasons
Assumptions substantiate your assumptions; do not leave them as implicit. Use
only reliable assumptions that are free of subjective judgment and
are based on valid reasoning.
avoid fallacies, such as generalization, abstraction and misplaced
Fallacies
concreteness.
use only reliable documented evidence from quality peer-review
Evidence
sources that is legitimate and relevant, not trivial.
Figure 20: Criteria for a sound argument adopted from Hart (1998)
Structure
Writing the Literature Review

Most novice researchers find it difficult to write the literature review itself and blame it mainly on
a list of myths (Troyka, 1993). Such list of myths about the difficulties in writing includes the
claim that writers are born not made, writers have to be in the mood to write, writers do
not have to revise, and that writing can be done at the last minute (pp. 48-49). To address
these myths, Troyka noted that most writers were not born natural writers and that being a good
writer means being a patient writer (p. 49), while Hart (1998) noted that not many people find
writing easy[although] writing has difficulties, it is not something that only other people can
do. Writing is something that most of us can do if we persevere (p. 184). Additionally, Troyka
(1993) notes that due to deadlines, most writers cant afford to wait for the mood in order to
produce their work. Moreover, writers must revise their work constantly in order to ensure the
readers can understand their intended meaning clearly.
Writing the literature review should provide the reader with what the researcher did during the
literature review input (see main section above) and what s/he has learned during the literature
review processing (see main section above). As such the literature review will demonstrate to the
reader the quality of the literature used to build the review. Moreover, Hart (1998) noted that the
literature review will demonstrate to the reader how you have extracted the main points from the
literature by undertaking analysis, and how you have reconstructed the main idea in your own
words by providing a critical synthesis (p. 183). Thus, it is important to note that a quality academic literature review piece requires both time and effort to produce a coherent piece of text
(p. 184). However, most novice researchers find it extremely difficult to put their ideas in writing
as they experience problems with academic writing. Table 2 provides a review of three of such
common problems with possible causes along with proposed solutions.
Hart (1998) suggested developing a plan of action when writing the literature review. The plan
should include pre-writing and literature review structure (i.e. an outline), allocating appropriate
evidences for each section, developing the first draft, allocating appropriate time for revising the
draft, and writing the final draft.
203
Table 2: Writing problems and solutions adopted from Hart (1998)

Problem
Possible cause(s)
Solution(s)
Lack of time.
Especially for part-time students, life makes many demands.
Time management.
Make writing a part of your
personal leisure time.
Socialize family and friends
to recognize that your academic writing work is important.
Unfamiliar with different

styles, especially academic
writing.
Familiar only with style used

in the workplace.
Lack of academic writing
background.
Read different styles.

Work at understanding different conventions for different situations.
Not used to writing at length.
Used to face-to-face communication.

Rarely use writing for argument and persuasion, hence
not familiar with tenses, the
possessive and grammatical
conventions.
Reading and learning!

Writing short piece first.
Subdivide academic writing
(such as dissertation drafts)
into manageable sections.
For empirical studies, Creswell (2003) suggested a model of writing the literature review based
on five main sections: 1) an introduction that tells the organization of the literature review section; 2) the first topic to address the independent variable(s) proposed by the study; 3) the second
topic to address the dependent variable(s) proposed by the study; 4) the third topic to address
studies conducted on the independent variable(s) and the dependent variable(s); 5) a summary
that highlight the key research studies relevant to the proposed study, their general findings that
relate to the proposed study, and support for the need of additional research on the proposed
topic.
A key issue related to the writing of the literature review deals with ethical issues associated with
academic writing. Ethical decision making should be maintained by all students in their academic
endeavors and writing. Hart (1998) suggested that researchers must use their sources properly
when writing the literature review in order to avoid criticism of their work. Such proper use
should ensure that no violations of academic writing standards or code of conduct exist. Figure 21
provides a list of the main academic writing standards or code of conduct violations and their
definitions.
misrepresenting the work of others.
presenting speculations or general claims of others as if they were
facts.
not providing correct citations.
Sloppiness
citing reference of scholars that are not directly addressing the point
Nepotism
that the citation is used for
the act of knowingly using another persons work as passing it off
Plagiarism
as your own.
Figure 21: Hart (1998, p. 181)s list of some academic writing standard violations
Falsification
Fabrication
204
Levy & Ellis
Summary of the Impact of the Body of Literature as a Whole on

Your Study
The literature review serves five major functions in a research endeavor. It is vital in establishing
the current state of the BoK, including identifying where excess research exists (i.e. what is already know?) and where new research is needed (i.e. what is needed to be known?). The review
also provides a solid theoretical foundation for the proposed study (related to what is already
known?). It substantiates the presence of the research problem (related to what is needed to be
known?) and justifies the proposed study as one that contributes something new to the BoK. Finally, the review frames the valid research methodologies, approach, goals, and research questions for the proposed study.
In order to accomplish these functions the review must do more than just point to the literature.
An effective review must describe the major points contained in an article, interpret those points,
classify the articles position in the BoK, explain the importance of the article, compare and contrast the findings and position of the article with other articles from the BoK, and evaluate the
findings of the article in light of the rest of the BoK.
Discussion
Summary of the Value of Effective Literature Review
It was suggested that the real value of published research work is in the dissemination of knowledge for use by others (Barnes, 2005, p. 10). However, the value or importance of an effective
literature review is in ensuring that the researcher demonstrates a full understanding of the BoK
related to the phenomenon under study, while at the same time should be explanatory and creative (Webster & Watson, 2002, p. 11). Moreover, an effective literature review should demonstrate a thorough/systematic examination of the existing BoK by following the three-stage process
framework discussed above, while demonstrating clear distinctions among opinions, theories, and
empirically established facts.
Literature Review: Tips from the Field

Conducting a literature review is not just a cognitive challenge; it is also a management challenge. Developing a foundation in the literature or a research project can easily entail reviewing
well over 100 articles and books. Mining that number of items for gems of information applicable
to the research being conducted, and keeping track of the sources of that information, can be a
very daunting task.
Although there is certainly not a single right way to manage the literature review process every
experienced researcher probably has her or his own preferred techniques the trail-and-error
process of developing a management approach could be quite costly in time and effort for the
novice researcher. As a starting point for the novice researcher, the following sections detail some
field-tested answers to the following management questions: how to get the most out of reading
the literature; how to manage the workflow of the review process; and how to know when the
literature review is complete.
Reading the literature

It is important to remember that the literature is being read purposively actually, for two purposes: to establish a context in the existing BoK for the research being conducted, and to point to
additional literature sources. The following techniques have been found to be very helpful in
promoting the type of active reading necessary for the process:
205
1) Keep an annotated copy of all the articles, book chapters, or conference proceedings you read
in both hardcopy and electronic format. It is unbelievably frustrating to need to refer back to an
article and find you do not have ready access to it.
2) Highlight everything in the article that is applicable to your research, even if you dont think
you will use the material. It is almost impossible to tell exactly what direction research will take,
and it can be very frustrating and time consuming to try to re-locate information you are almost
certain you found in a given paper.
3) Write notes to yourself about the article you are reading regarding issues, thoughts, or general
comments such as nice methodology for , interesting definition of, etc.
4) Write a brief (one to two paragraph(s)) annotated bibliography entry that encapsulates the essential points applicable to your research derived from the article. It is important to note that the
annotated bibliography is specific to your research and is not the same as the abstract for the article, which summarizes the entire study.
5) As one reads literature, be sure to look for and circle any terms or expressions that might serve
as keywords that would facilitate the forward or backward searching described earlier.
6) Remember, it is important to place each article in the context of the body of knowledge by
identifying the applicable model(s), construct(s), theory(ies), and/or literature stream(s) (see section Cognitive/construct-level above). Annotate the applicable model(s), etc. on the front of the
article.
Managing the literature review process

As is evident from the preceding sections, quite a large number of factors must be managed while
conducting a literature review. The magnitude of the organizational challenge can often overwhelm the novice researcher. Several techniques have been field-tested by veteran researchers
and found to be useful. A brief overview of some of these field-tested techniques follows.
Figure 22: Worksheet to help full coverage of literature sources
206
EBSCOhost
LEA Journals
Blackwell (Synergy)
JSTOR
ACM (Digital Lib)
Thomson (G. Bus, OneFile)
Wilson (OmniFile)
INFORMS
Keyword
Searched
Date
ProQuest (ABI/INFORM)
The first technique in managing the literature review process deals with tracking the research
path. In order to adequately explore the literature base, a number of electronic databases (Figure
2) must be searched on a number of different keywords and keyword combinations. A simple
spreadsheet such as the one illustrated in Figure 22 can prove to be quite useful in ensuring full
coverage of the search.
Levy & Ellis
The second technique in managing the literature review process deals with filing the articles.
Whether hardcopies and/or electronic copies of the annotated articles included in the literature
review are retained, an effective filing system is essential. Within the filing system, it is important
to be able to locate articles by author, title, applicable model/construct/theory/literature stream,
and publication date. One approach is to ensure that each article is available both in electronic
format as well as in hardcopy format. The management of the filing of both digital and physical
should be parallel. For example, three inch binders can be used to store the physical articles with
notation on the cover of the binder on the main content (e.g. user satisfaction, computer selfefficacy, etc.), whereas the digital library of such articles should match by having a folder in the
hard-drive (or any other digital storage location) with the exact same notation of main contents.
Inside each digital folder, articles can be saved by the last names of the authors and year of publication (i.e. Levy_Ellis_2006.pdf), whereas in the binders, the articles should be ordered alphabetically based on the authors last names. Figure 23 illustrates the technique for physical and
digital organization of articles during the literature review process.
Digital
Physical
Figure 23: A technique for physical and digital organization of articles

The third technique in managing the literature review process deals with building an annotated
bibliography. As mentioned in the preceding section, it is important to write a brief annotated bibliography entry for each item of literature included in your review. The annotated bibliography is
most useful when each entry contains complete, well-written sentences that can be easily copied
and pasted into the review of the literature and require minimal subsequent editing. The annotated
bibliography should, of course, be linked with the article file to make it easy to find the complete
article if additional information is needed.
207
The fourth technique in managing the literature review process deals with automating the process.
There are several commercial reference management products available. Programs such as EndNote, ProCite, and Reference Manager offer the capability of bringing organization to the
search, storage, and retrieval of references. They also manage the very tedious task of formatting
the citations and references in accordance with a wide range of publishing standards.
When is the review completed?

In a sense, a literature review continues until a study is completed; throughout the course of a
study, one should remain alert to applicable research being reported. The best way to view the
review is as an organic system that is constantly growing and changing as the study develops.
Changes in methodology, addition of new constructs, and conflicting literature can all require
further research. As such, the literature review, as a process, should not be viewed as completed
until all the research associated with the study has been completed.
Conclusions and Recommendations

This paper presented a systematic framework for the construction of an effective literature review
following the traditional data processing model. Following this model, the proposed framework
suggests a three-stage approach to guide novice IS researchers in the development of a sound and
effective literature review: 1) Inputs, 2) Processing, and 3) Outputs. The uniqueness of the proposed framework is in the methodological process that is presented enabling novice IS researchers the ability to decompose the task of the literature review into the three manageable stages.
Additionally, specific examples from the IS field are provided in the discussion of the search
methodology, quality literature sources from the IS field, and in the six-step processing approach
that includes the theoretical IS universe, models/frameworks, and constructs. The theory of argumentation is discussed with examples of two different approaches for development of sound argumentation in the literature review writing. Finally, discussions on the value of the effective literature review, tips from the field, manageable steps on reading the literature, practical tips on
how to actually manage the literature review, along with summary on when the literature review
is practically completed are provided.
There are three groups of audiences that this paper aims to address. The first group includes doctoral students in IS or closely related programs, while the second group includes novice IS researchers. The attempt of this paper is to help novice IS researchers and IS doctoral students to
better understand the value of effective literature review in research. Additionally, the paper aims
to provide novice researchers with ways to understand what is required in order to produce a high
quality literature review. With the three-stage process, novice researchers should be able to see
how each stage is related to the next and understand how the whole process of the literature review is constructed to form the theoretical foundation for their proposed research. The second
audience group includes professors who are teaching doctoral courses in IS or closely related programs. This paper can help professors by providing them a resource to guide their doctoral students on a systematic process for conducting effective literature review. Moreover, the paper can
help such professors by providing them sources for additional readings (see reference list) on the
subject of literature review, quality of sources, processing of the literature and classification, as
well as sources of writing the actual review.
Future work in this endeavor may include expanding the process to provide additional examples
from other fields. Such fields can include informing science, information technology education,
computer technology education, computer science and more. Additionally, an expansion may include full writing samples of proposals, both positive and negative writing examples.
208
Levy & Ellis
Acknowledgments
The authors would like to thank Professor Eli Cohen, the Managing Editor of ISJ and Founder of
Informing Science Institute as well as Professor Scott J. Lloyd, the ISJ Editor-in-Chief for their
encouragements. Additionally, the authors would like to thank the six anonymous referees for
their careful review, constructive comments, and valuable suggestions.
References
Andrews, C. P., & Wynekoop, J. (2004). A framework for comparing IS core curriculum and IS requirements for accounting majors. Journal of Information Systems Education, 15(4), 437-450.
Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review,
84(2), 191-215.
Barling, J., & Beattie, R. (1983). Self-efficacy beliefs and sales performance. Journal of Organizational
Behavior Management, 5(1), 41-51.
Barnes, S. J. (2005). Assessing the value of IS journals. Communications of the ACM, 48(1), 110-112.
Bem, D. J. (1995). Writing a review article for psychological bulletin. Psychological Bulletin, 118(2), 172177.
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals. New York: Longmans: D. McKay Co.
Boote, D. N., & Beile, P. (2005). Scholars before researchers: On the centrality of the dissertation literature
review in research preparation. Educational Researcher, 34(6), 3-15.
Compeau, D. R., & Higgins, C. A. (1995). Computer self-efficacy: Development of a measure and initial
test. MIS Quarterly, 19(2), 189.
Couger, J. D. (1989). Preparing IS students to deal with ethical issues. MIS Quarterly, 13(2), 211-220.
Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd
ed.). Thousand Oaks, CA: Sage Publications.
Culnan, M. J. (1978). An analysis of the information usage patterns of academics and practitioners in the
computer field: A citation analysis of a national conference proceedings. Information Processing &
Management, 14(6), 395.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance. MIS Quarterly,
13(3), 319-341.
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), 982-1004.
Davison, R. M., Vreede, G.-J. de, & Briggs, R. O. (2005). On peer review standards for the information
systems literature. Communications of the Association for Information Systems, 16(4), 967-980.
DeLone, W. H., & McLean, E. R. (1992). Information system success: The quest for the dependent variable. Information Systems Research, 3(1), 60-95.
DeLone, W. H., & McLean, E. R. (2003). The DeLone and McLean model of information systems success:
A ten-year update. Journal of Management Information Systems, 19(4), 9-30.
Doll, W. J., & Torkzadeh, G. (1991). The measurement of end-user computing satisfaction: Theoretical and
methodological issues. MIS Quarterly, 15(1), 5.
Gagne, R. M., Briggs, L. J., & Wager, W. W. (1992). Principles of instructional design (4th ed.). Fort
Worth, TX: Harcourt Brace Jovanovich College Publishers.
Grover, V., Seung, J. R., & Segars, A. H. (1996). Information systems effectiveness: The construct space
and patters of application. Information & Management, 31(4), 177-191.
209

Hardgrave, B. C., & Walstrom, K. A. (1997). Forums for MIS scholars. Communications of the ACM,
40(11), 119-124.
Hart, C. (1998). Doing a literature review: Releasing the social science research imagination. London,
UK: Sage Publications.
Hozack, W. J., Ranawat, C., & Rothman, R. H. (2003). Corporate sponsorship and research: Impact and
outcome. The Journal of Arthroplasty, 18(8), 953-953.
Iivari, J., Hirschheim, R., & Klein, H. K. (2004). Towards a distinctive body of knowledge for information
systems experts: Coding ISD process knowledge in two IS journals. Information Systems Journal,
14(4), 313-342.
Jonassen, D. H., Tessmer, M., & Hannum, W. H. (1999). Task analysis methods for instructional design.
Mahwah, NJ: Lawrence Erlbaum Associates.
Katerattanakul, P., Han, B., & Hong, S. (2003). Objective quality ranking of computing journals. Communications of the ACM, 46(10), 111-114.
Kerlinger, F. N., & Lee, H. B. (2000). Foundations of behavioral research (4th ed.). Holt, NY: Harcourt
College Publishers.
Kock, N., Davison, R., Clarke, R., & Loch, K. (2000). IS research ethics (panel session): Defining ethical,
barely ethical, and unethical behavior. Proceedings of the Twenty First International Conference on
Information Systems, Brisbane, Queensland, Australia
Laband, D. N., & Piette, M. J. (2000). Perceived conduct and professional ethics among college economics
faculty. American Economist, 44(1), 24-34.
Leedy, P. D., & Ormrod, J. E. (2005). Practical research: Planning and design (8th ed.). Upper Saddle
River, NJ: Prentice Hall.
Lowry, P., Romans, D., & Curtis, A. (2004). Global journal prestige and supporting disciplines: A scientometric study of information systems journals. Journal of the Association for Information Systems,
5(2), 29-75.
Manton, E., Turner, C. T., & English, D. (2004). Testing the level of student knowledge. Education,
124(4), 682-687.
Mason, J. B., Bearden, W. O., & Richardson, L. D. (1990). Perceived conduct and professional ethics
among marketing faculty. Academy of Marketing Science. Journal, 18(3), 185-197.
Metcalfe, M. (2002, June 21). Metaphors for literature reviews (responses summary for ISWorld listserv email request).
Meyers, L. S., Gamst, G., & Guarino, A. J. (2006). Applied multivariate research: Design and interpretation. Thousand Oaks, CA: Sage Publication.
Mylonopoulos, N. A., & Theoharakis, V. (2001). Global perceptions of IS journals. Communications of the
ACM, 44(9), 29-33.
Newton, I. (1676, February 5). Letter to Robert Hooke. Available at
http://www.quotationspage.com/quotes/Isaac_Newton/
Ngai, E. W. T., & Wat, F. K. T. (2002). A literature review and classification of electronic commerce research. Information & Management, 39(5), 415-429.
Noble, T. (2004). Integrating the revised Bloom's taxonomy with multiple intelligences: A planning tool for
curriculum differentiation. Teachers College Record, 106(1), 193-211.
Peffers, K., & Ya, T. (2003). Identifying and evaluating the universe of outlets for information systems
research: Ranking the journals. JITTA : Journal of Information Technology Theory and Application,
5(1), 63-84.
210
Levy & Ellis

Pitt, L. F., Watson, R. T., & Kavan, C. B. (1995). Service quality: A measure of information systems effectiveness. MIS Quarterly, 19(2), 173-187.
Rainer, R. K., & Miller, M. D. (2005). Examining differences across journal rankings. Communications of
the ACM, 48(2), 91-94.
Robey, D., Boudreau, M.-C., & Rose, G. M. (2000). Information technology and organizational learning: A
review and assessment of research. Accounting, Management and Information Technologies, 10(2),
125-155.
Saunders, C. (n.d.). IS world's MIS journal rankings. Retrieved April 5, 2006, from
http://www.isworld.org/csaunders/rankings.htm
Sekaran, U. (2003). Research methods for business. A skill building approach (4th ed.). New York, NY:
John Wiley & Sons.
Sethi, V., & King, W. R. (1998). Organizational transformation thought business process reengineering.
Upper Saddle River, NJ: Prentice Hall.
Shaw, D. (1995). Bibliographic database searching by graduate students in language and literature: Search
strategies, system interfaces, and relevance judgments. Library & Information Science Research,
17(4), 327-345.
Shaw, J. (1995). A schema approach to the formal literature review in engineering theses. System, 23(3),
325-335.
Sorenson, O., & Fleming, L. (2004). Science and the diffusion of knowledge. Research Policy, 33(10),
1615-1634.
Srinivasan, A. (1985). Alternative measures of system effectiveness: Associations and implications. MIS
Quarterly, 9(3), 243-253.
Straub, D. W. (1989). Validating instruments in MIS research. MIS Quarterly, 13(2), 147-170.
Thomas, G. (1997). What's the use of theory? Harvard Educational Review, 67(1), 75-104.
Toulmin, S. E. (1958). The uses of argument. Cambridge, UK: University Press.
Troyka, L. Q. (1993). Simon and Shuster handbook for writers (3rd ed.). NJ: Prentice-Hall.
Venkatesh, V., & Davis, F. D. (1996). A model of the antecedents of perceived ease of use: Development
and test. Decision Sciences, 27(3), 451-482.
Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model: Four
longitudinal field studies. Management Science, 46(2), 186-205.
Walstrom, K. A., Hardgrave, B. C., & Wilson, R. L. (1995). Forums for management information systems
scholars. Communications of the ACM, 38(3), 93-102.
Walton, D. (2006). Examination dialogue: An argumentation framework for critically questioning an expert
opinion. Journal of Pragmatics, 38(5), 745-777.
Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, 26(2), 13-23.
Whitman, M. E., Hendrickson, A. R., & Townsend, A. M. (1999). Research commentary. Academic rewards for teaching, research, and service: Data and discourse. Information Systems Research, 10(2),
99-109.
Williams, J. M., & Colomb, G. G. (2003). The craft of argument (2nd ed.). New York, NY: Longman.
Wu, M.-M. (2005). Understanding patrons' micro-level information seeking (MLIS) in information retrieval situations. Information Processing & Management, 41(4), 929-947.
Zahn, S. B., Rajkumar, T. M., & Zahn, C. J. (1996). Incorporation of student portfolios in the TQM classroom. International Journal of Instructional Media, 23(4), 327-340.
211
Biographies
Dr. Yair Levy is an assistant professor of Management Information
Systems at the Graduate School of Computer and Information Sciences
at Nova Southeastern University. During the mid to late 1990s, he assisted NASA to develop e-learning systems. He earned his Bachelors
degree in Aerospace Engineering from the Technion (Israel Institute of
Technology). He received his MBA with MIS concentration and Ph.D.
in Management Information Systems from Florida International University. His current research interests include cognitive value of IS, of
online learning systems, effectiveness of IS, and cognitive aspects of
IS. Dr. Levy is the author of the book Assessing the Value of eLearning systems. His research publications appear in the IS journals,
conference proceedings, an invited book chapters, and encyclopedias. Additionally, he chaired
and co-chaired multiple sessions/tracks in recognized conferences. Currently, Dr. Levy is serving
as the editor-in-chief for the International Journal of Doctoral Studies (IJDS) and serving as an
associate editor for the International Journal of Web-based Learning and Teaching Technologies
(IJWLTT). Moreover, he is serving as a member of editorial review or advisory board of several
refereed journals. Additionally, Dr. Levy has been serving as a referee research reviewer for numerous national and international scientific journals, conference proceedings, as well as MIS and
Information Security textbooks. He is also a frequent speaker at national and international meetings on MIS and online learning topics. To find out more about Dr. Levy, please visit his site:
http://scis.nova.edu/~levyy/
Dr. Timothy Ellis obtained a B.S. degree in History from Bradley
University, an M.A. in Rehabilitation Counseling from Southern
Illinois University, a C.A.G.S. in Rehabilitation Administration from
Northeastern University, and a Ph.D. in Computing Technology in
Education from Nova Southeastern University. He joined NSU as Assistant Professor in 1999 and currently teaches computer technology
courses at both the Masters and Ph.D. level in the School of Computer
and Information Sciences. Prior to joining NSU, he was on the faculty
at Fisher College in the Computer Technology department and, prior to
that, was a Systems Engineer for Tandy Business Products. His research interests include: multimedia, distance education, and adult
learning. He has published in several technical and educational journals including Catalyst, Journal of Instructional Delivery Systems, and Journal of Instructional Multimedia and Hypermedia.
His email address is [email protected]. His main website is located at
http://www.scis.nova.edu/~ellist/
212

Effective Literature Review

Uploaded by

Copyright:

Available Formats

Effective Literature Review

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Effective Literature Review

Uploaded by

Copyright:

Available Formats

Informing Science Journal

A Systems Approach to Conduct an

Editor: Eli Cohen

This paper is divided into four main

A Systems Approach to Conduct an Effective Literature Review

What is a Literature Review?

Stages of the literature review process

What is unique about an ISrelated literature review?

Figure 1: The three stages of effective literature

Levy & Ellis

Why Conduct a Literature Review?

Build a solid theoretical foundation for your study

A Systems Approach to Conduct an Effective Literature Review

Fitting the literature into your research

Place your study in the context of existing work (body of knowledge)

Levy & Ellis

Literature Review: Inputs

Where to Look for Quality IS Literature?

Validating the quality of IS literature

A Systems Approach to Conduct an Effective Literature Review

Free Full Text Web Access

ACM (Digital Lib)

Thomson (G. Bus, OneFile)

IEEE (Comp Soc & Xplore)

Ranked MIS Journals

Literature Vendor (Database)

Figure 2: ISWorlds top 50 ranked MIS journals and electronic availability

Levy & Ellis

Proc. CD-ROM for purchase

Full Text Web Access (Free)

Full Text Web Access (Fee)

International Conference on Information Systems (ICIS)

ACM (Digital Lib)

IEEE (Comp Soc & Xplore)

Ranked and Non-Ranked IS Conferences

Figure 3: Ranked and non-ranked IS conferences

A Systems Approach to Conduct an Effective Literature Review

Testing for applicability to your study

Levy & Ellis

How to Find Quality IS Literature?

Literature databases & electronic resources

A Systems Approach to Conduct an Effective Literature Review

Levy & Ellis

A Systems Approach to Conduct an Effective Literature Review

Figure 4: Key metaphors for literature search process

Literature Review: Processing

Levy & Ellis

Know the Literature

Comprehend the Literature

A Systems Approach to Conduct an Effective Literature Review

Theory: Definition and Use in IS Literature

Levy & Ellis

Knowledge-based theory of the firm

Source: Schneberger and Wade (2006): http://www.istheory.yorku.ca/ via ISWorld.net

Figure 9: Main theories used in IS research, adopted from ISWorld

A Systems Approach to Conduct an Effective Literature Review

Constructs/variables: Definition and Use in IS Literature

Acceptance of Quantitative System

Figure 10: Main constructs used in IS research, adopted from ISWorld

Levy & Ellis