Performance Impacts of Information Technology: Is Actual Usage the Missing Link?

Author(s): Sarv Devaraj and Rajiv Kohli Reviewed work(s): Source: Management Science, Vol. 49, No. 3 (Mar., 2003), pp. 273-289 Published by: INFORMS Stable URL: http://www.jstor.org/stable/4133926 . Accessed: 18/02/2012 14:32
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].

INFORMS is collaborating with JSTOR to digitize, preserve and extend access to Management Science.

http://www.jstor.org

Performance Impacts

of

Information

Is Technology:Actual Missing Link?

Usage

the

Sarv Devaraj * Rajiv Kohli

Departmentof Management, Mendoza College of Business, University of Notre Dame, Notre Dame, Indiana 46556 [email protected]* [email protected]

(IT) and its effect on orgaperformance practitioners. In many cases, due to the nature of the research design employed, this stream of research has been unable to identify the impact of individual technologies on organizational performance. This study posits that the driver of IT impact is not the investment in the technology, but the actual usage of the technology. This proposition is tested in a longitudinal setting of a healthcare system comprising eight hospitals. Monthly data for a three-year period on various financial and nonfinancial measures of hospital performance and technology usage were analyzed. The data analysis provides evidence for the technology usage-performance link after controlling for various external factors. Technology usage was positively and significantly associated with measures of hospital revenue and quality, and this effect occurred after time lags. The analysis was triangulated using three measures of technology usage. The general support for the principal proposition of this paper that "actual usage" may be a key variable in explaining the impact of technology on performance suggests that omission of this variable may be a missing link in IT payoff analyses. IT (InformationTechnology; Payoff;IT Usage; Healthcare; LongitudinalStudy; Profitability;Quality)

investment The relationship between continues toin information technology nizational interest academics and

1. Introduction
Organizations view investments in information technology (IT) as a way to combat competition by improving productivity, profitability, and quality of operations. The Department of Commerce estimates that about 46% of all equipment spending in the United States is in IT equipment and software (U.S. Department of Commerce 1998) and in spite of economic slowdown, spending by the IT sector is expected to increase (U.S. Department of Commerce 2001). With increased investments in technology comes the responsibility to provide economic justification. Today, more than ever, IT executives encounter the justification issue due to senior man0025-1909/03/4903/0273$5.00 1526-5501 electronicISSN

agement's insistence that the investment be properly utilized. In recent years, a surge in the number of studies that examine the IT payoff is a testimony to this challenge (Mukhopadhyay et al. 1995, Brynjolfsson 1996, Dewan and Min 1997, Francalanci and Galal 1998, Bharadwaj et al. 1999, Devaraj and Kohli 2000, Sohal Moss and Ng 2001). In doing so, the IT payoff literature has examined the relationship between investments in IT capital and labor and their effect on organizational performance. However, largely due to the nature of the research designs employed, this stream of research has not definitively attributed the effects of usage and impact of individual technologies on organizational performance (Loveman 1994,
MANAGEMENT SCIENCE @

2003 INFORMS

Vol. 49, No. 3, March2003 pp. 273-289

DEVARAJ AND KOHLI PerformanceImpacts of InformationTechnology:Is Actual Usage the Missing Link?

Lee and Barua 1999, Papp 1999, Rao et al. 1995). Kelley (1994) states that one reason the technologyproductivity connection seems to be elusive is that the aggregated unit of analysis at the organizational level makes it difficult to isolate the impact of any individual technology. Furthermore, IT payoff studies have differed in the selected variables and the level at which those variables were collected, i.e., strategic business unit level, firm level, or industry level. Research designs have also varied from a snapshot of the performance variables to longitudinal data (Devaraj and Kohli 2000). Perhaps one of the most serious issues has been that few studies have captured the actual usage of the IT. In addition, merely examining the dollars invested in IT may not be an accurate reflection of the effectiveness of IT because the extent of its usage may vary across industries, firms, or processes. Thus, there is a void in the IT payoff literature in evaluating the impact of individual technology usage on organizational performance. An associated stream of research examines the issue of technology usage (Straub et al. 1995, Szajna 1996, Taylor and Todd 1995) in the context of technology acceptance (Davis 1989, Igbaria et al. 1997, Lucas and Spitler 1999). The comparability of self-reported usage and objective or actual usage remains a controversial point in information systems research (Straub et al. 1995, Venkatesh and Davis 2000). Self-reported usage might induce biases due to having the same respondents answer similar questions on their perceptions of the IT and its effectiveness (known as commonmethod variance). Further, there is evidence suggesting that actual usage and perceived usage may not be congruent (Straub et al. 1995). Because most studies to date have been in laboratory settings with student subjects, there is merit to examining an independently monitored and objective measure of technology usage. This study extends this line of research by examining the technology usage issue in the field setting of a hospital network. Because the IT payoff literature has largely overlooked IT usage, and the usage literature has not examined the effect of actual usage on organizational performance, the goal of this study is to bridge the IT payoff and usage literature by examining technology usage and its impact on firm performance. Fur-

ther, this paper considers the application of a strategic technology in healthcare organizations-namely, decision support systems (DSS) and their impact on organizational performance. Monthly data on technology usage and firm performance for eight hospitals over a three-year time horizon were collected. The relationship between various financial and nonfinancial measures of performance and technology usage was estimated to assess the payoff resulting from technology use, after controlling for relevant external variables.

2.

Literature Review and Proposition

2.1. IT Payoff The last decade has witnessed an unparalleled growth in investment in IT applications. The mainstream academic literature has documented numerous studies that examine the relationship between investments in technology and payoffs realized in terms of enhanced organizational performance (for a detailed discussion, please refer to a literature survey by Brynjolfsson and Yang 1996 and a meta-analysis by Kohli and Devaraj 2003). It is evident that there are significant differences among studies in terms of the level of analyses, methodologies employed, variables, and contexts examined. Many economy-level studies (Baily 1986, Roach 1987, Morrison and Berndt 1991) observed a negative relationship between technology-related variables and performance. At the industry level the results are mixed, with some studies documenting a positive impact of technological investments (Kelley 1994, Siegel and Griliches 1992) while other studies by Berndt and Morrison (1995) and Koski (1999) detect no significant advantage to IT investments. At the more-detailed firm level, Diewert and Smith (1994), Hitt and Brynjolfsson (1995), and Dewan and Min (1997) present results indicating a positive relationship between technology and performance. Other firm-level studies by Menon et al. (2000) and Devaraj and Kohli (2000) found evidence of the positive effect

274

MANAGEMENT SCIENCE/Vol. 49, No. 3, March 2003

DEVARAJ AND KOHLI PerformanceImpacts of InformationTechnology:Is Actual Usage the Missing Link?

1 Table

A Summary Firm-Level Payoff of IT Studies

Study Diewert Smith and (1994) and Barua, Kriebel, Mukhopadhyay (1995) Hitt Brynjolfsson and (1995) Prasad Harker and (1997) Dewan Min and (1997)

Variables used costs, rate, Inventory holding growth purchases, levels sales, inventory utilization, turnover, Capacity inventory quality, relative and product introduction price, new

Duration over Quarterly 6 quarters over Annually 3 years

Key findings IT tolarge led productivity gains IT positively tosome was related intermediate measures profitability, of that but theeffect generally was too small measurably final to affect output IT to leads increased and productivity but consumer surplus, nothigher profitability inIT not Additional investment may capital have benefits real IT is for capital a netsubstitute ordinary and i.e., investment capital labor; IT leads higher to returns IT investment to higher leads productivity and quality firms productive spent moreon Highly client-server less on in-house and application development

IT Value over added, stock, noncomputer ROA, Annually 5 capital, labor ROE, return, expense, shareholder IT years sales stock/employee, investment, capital market debt, stock share, R&D firm growth, IT non-IT non-IS Annually 3 over IS capital, capital, labor expense, labor expense years IT non-IT labor value over capital, capital, expense, Annually 5 of added, number employees sales, years 39accounting over periods 3 years 1 year

Total on-time labor machine Mukhopadhyay, Rajiv, output, output, hours, and Srinivasan level absenteeism hours, ofautomation, rate, (1997) of degree supervision of and Numberyears intheposition, CIO of Prattipati Mensah proportion software resources onclient server spent (1997) ofsoftware applications, percentage budget on development spent new IT FrancalanciGalal and and investments; managerial, clerical, income professional composition; per (1998) total employee; operating expense IT medical medical Menon, and Lee, capital, capital, capital Eldenburg (2000) and number BPR of initiatives, Devaraj Kohli (2000) Revenue, quality IT investment indicators,capital, support labor,

in expenses associated IncreasesIT with are benefits accompanied when productivity in bychanges worker composition Annual 19years IT for contributes to positivelytheproduction in ofservices thehealthcare industry over investment contributes to higher revenue, Monthly 3 years IT butthe effectis morepronounced when with initiatives combined BPR 10-year period

of IT capital and labor on outcome measures among hospitals. Notably though, Barua et al. (1995) and Strassman (1990) did not observe this same relationship between technology and performance. What we can say is that, overall, the more detailed the level of analysis, the better the chance to detect the impact, if any, of a given technology. It also seems to be the case that moderating variables, such as business process reengineering (BPR), have an impact on the linkage. Table 1 presents a summary of firm-level studies and key findings. Some of the following methodological characteristics shared by IT payoff studies may also explain the variance in observed results:

1. Anecdotal and small sample data, 2. Cross-sectional data analysis that limits the ability to examine lag effects as well as causal connections between IT adoption and organizational performance, 3. Limited set of control variables that account for extraneous factors such as market conditions, etc., and 4. Aggregated units of analyses where it is difficult to accurately observe the impact of organizational initiatives. As we shall see later, the present study aims to overcome such methodological weaknesses by examining monthly longitudinal usage data for a threeyear period while controlling for factors that might affect organizational performance.

MANAGEMENT SCIENCE/Vol. 49, No. 3, March2003

275

DEVARAJ AND KOHLI PerformanceImpacts of InformationTechnology:Is Actual Usage the Missing Link?

2.2. Technology Usage Past studies have generally utilized self-reported usage. Self-reported usage can provide an important indicator in assessing how well IT is used within an organization. It can also identify IT that is helpful to decision makers and can assist in decisions to expand or curtail future IT investments. However, self-reported usage measures have several limitations. Self-reported measures of usage have been found to be strongly related to self-reported independent variables, while actual usage demonstrated significantly weaker links with self-reported usage (Straub et al. 1995). Similarly, Szajna (1996) concluded that self-reported usage might not be an appropriate surrogate for actual usage. Possible explanations for this discrepancy are subjects' difficulty in recalling their past usage, exaggeration of the extent of usage to fit in with their superiors' expectations, attention lapses, and bounded rationality. Wagner and Gooding (1987) concluded in their meta-analysis that users are poor estimators of aspects of their own behavior. Therefore, we believe that technology impacts can be assessed by examining actual IT usage rather than self-reported IT usage. 2.3. Usage-Performance Link The significance of the link between usage of an IT, particularly DSS, and organizational performance has long been discussed in the literature (Keen and Scott-Morton 1978, Delone and McLean 1992). Doll and Torkzadeh (1998) note that "System-use is a pivotal construct in the system-to-value chain that links upstream research on the causes of system success with downstream research on the organizational impacts of information technology." Similarly, the relationship between IT utilization/use and organizational performance has been proposed as pivotal by Delone and McLean (1992) and Beath et al. (1995). For IT impacts to occur, it is imperative that usage is tied to organizational performance metrics. Although previous research has highlighted the importance of the relationship of DSS usage and organizational performance, this critical link has generally been examined in simulated laboratory settings (Sharda et al. 1988, Forgionne and Kohli 1996). 276

Technology usage results in organizational impacts only when the suitability of the application is matched with technology. Goodhue and Thompson (1995) contend that task-technology fit has to be established before IT utilization can lead to individual performance impacts. Task-technology fit is achieved when the technology is compatible with the targeted application and there are qualified users who use it. In our research, the strategic nature of the DSS (technology) to the hospitals and the business value of the contract modeling application (task) mediated by IT usage are steps that enable the DSS investment toward organizational financial and quality outcomes. Thus, as illustrated later in Figure 2, the task-technology fit suggested by Goodhue and Thompson holds true in our research setting. In addition, required specialized skills affirm that the DSS users and technology form a synergy to monitor hospital reimbursement, a strategic area that is increasingly critical to the organization's financial stability. This line of argumentation is consistent with Weill's (1992) call to examine the steps between IT investment and the resulting business value. 2.4. Voluntariness of Use Performance improvements from the use of IT might also be affected by whether the use was voluntary or mandatory. There has been recent attention in the IS literature to the voluntariness of system use. Voluntariness refers to the "extent to which potential adopters perceive the adoption decision to be nonmandatory" (Agarwal and Prasad 1997, Hartwick and Barki 1994). Hartwick and Barki (1994) found that subjective norms had a significant effect on intention to use in mandatory settings, but not in voluntary settings. They also found that even in mandatory settings, system use varies because some users are not willing to comply with organizational mandates. We examine systems where the use of the system was voluntary and therefore can be expected to affect productivity. Thus, consistent with this line of thought, the highlevel proposition is: The actual usage of strategic information technology by an organizationwill be associatedwith
PROPOSITION.

MANAGEMENT SCIENCE/Vol. 49, No. 3, March2003

DEVARAJ AND KOHLI

Is Technology: ActualUsagetheMissingLink? of Impacts Information Performance

1 Figure

of Model Relationships

Variables Independent of Technology Usage

* * Reports
Processing (CPU) time

of S Number records
accessed

(DISKIO)Performance

DependentVariables of Hospital Performance

* * (MORT) Mortality Revenueper admission (NPRADM)

Control
*

Variablesamion
Medicare

* *

Patient

income

M Revenue perday (NPRDAY)

employees

Number

of

* *

(FTE)

Ageof hospital Outpatients

after controllingfor the effect of exterhigher performance, nal variables. Figure 1 presents this proposition in a research model within the context of a hospital network. The actual usage of the strategic IT is represented by the total number of reports executed by a user, the CPU time, and the number of records accessed for each report. Consistent with past studies, we include several variables as control variables (described below). "Hospital Performance" in the model is captured by the hospital's mortality level and two financial variablesrevenue per admission and revenue per day.

3.

The use of a particular technology might vary across organizations and also between different time periods for the same organization. Additionally, the impact of technology usage for a given time period might be realized over several time periods. Therefore, a crosssectional set of hospitals combined with time-series data is ideal to examine the effect of the IT usage on measures of profitability and quality, while controlling for other factors.

Research Design

3.1. Sample The research design incorporates data over several time periods as well as across hospitals, also referred to as "panel data" in the econometrics literature. Data were collected for 36 monthly periods from eight hospitals of a health system network that had implemented a DSS. The DSS helps decision makers analyze contracts by comparing costs of expected services and expected payments from insurers and identifying areas of cost cutting and operational improvements necessary for financial viability of the hospital. In addition, decision makers utilize the "what-if" analysis to generate alternatives of variable pricing options and their impact on the hospital revenue. Insights gained from such analyses assist marketing managers in renegotiating existing contracts and negotiating favorable terms in new contracts. We ensured that our usage data consisted of DSS reports that have strategic value by searching each report that related to the application and ascertaining all DSS users were using these reports. Next, we verified that all reports were included by examining every report executed by the relevant users and crosschecked them with those identified above. Furthermore, for conclusion validity purposes, we excluded

MANAGEMENT SCIENCE/Vol. 49, No. 3, March 2003

277

DEVARAJ AND KOHLI PerformanceImpacts of InformationTechnology:Is Actual Usage the Missing Link?

reports that ran for more that eight hours, an outlier that almost always indicates a problem with the logic of the report. This sample of hospitals within a health system network was chosen so that the DSS usage was for a similar decision-making situation and as such did not introduce a confounding variable. With similar business, strategic, and competitive business conditions in the healthcare industry, the use of the contract modeling within the hospitals provides an opportune setting for examining the usage of DSS and its relationship with organizational performance. As is the case in our research setting, DSS in most industries are used to monitor financial performance, albeit through a variety of industry-specific variables. 3.2. Research Setting The hospitals included in this study are members of a private health system. The health system is a national organization with member hospitals in various markets across the United States. Each hospital is an independent legal entity with its own board and financial statements. The member organizations of the health system network are suburban, mid-sized hospitals with over 4,000 combined beds, 20,000 employees, and annual operating revenue of approximately $1.5 billion. As noted previously, the technology focus of the present study is a DSS. A DSS is a computer system designed to help improve the effectiveness and productivity of managers (Keen 1981). This is a problem area in hospitals. Hospital executives report they do not receive the factual information they need to make decisions (85% responding in the affirmative), with a high percentage stating that they expect their computer systems to generate such information (Butters and Eom 1992). DSS in healthcare organizations are therefore critical for the delivery of effective and efficient services, strategy formation, and outcome measurement. DSS in the Reported Hospitals Among the hospitals included in our study, DSS especially plays a critical role in the identification of clinical and financial improvement opportunities and is viewed by hospitals as a strategic resource. This DSS application, developed and used for over a decade, 3.3.

also serves as a repository of the financial, clinical, and quality outcomes. It is one physical system with logically partitioned programs and databases for each hospital. This enables efficient program maintenance and database administration activities, yet ensures the privacy and security of each hospital's data. Figure 2 presents a vignette to illustrate the DSS usage in hospitals and how it affects their performance. The strategic value of DSS is also demonstrated by its use by several constituencies across all of the hospitals. Clinicians and risk-management professionals analyze competing patient treatment plans through outcomes such as length of stay, readmission rates, and mortality, and identify those with better-than-expected outcomes. One analysis using DSS data developed a screening program to predict which patients will suffer adverse outcomes. It indicated that applying the screening program over three years will reduce 1,241 hospital days, save 11 lives, and reduce costs by $10.5 million (Zarling et al. 1999). Similarly, activity-based costing data within DSS is analyzed by finance managers to review activities that consume considerable resources. Such analysis has led to process redesign by applying IT to reduce process steps as well as costs. 3.4. Validation of Instrumentation 3.4.1. Independent Variables: Usage. Actual measures of usage in the study were based on usage records generated from a log created by a utility program to track user resource consumption (Table 2). The utility is designed for charging back the departments for IT usage. The usage or independent variables captured under the utility program are: Reports, Disk I/O, and CPU Time. The log provides the information by user name and date and time of report execution.' Each measure was aggregated for a given month for all users of each hospital. "What if" analysis in strategic DSS involves historical data and complex calculations (Keen 1981) to
1 A variable-total report running time-was captured but excluded because it is confoundedby factorssuch as the number of DSS users, otherprocessingjobs, system maintenanceactivities,etc. CPU usage and Disk I/O are not susceptibleto such confounding effects.

278

MANAGEMENTSCIENCE/Vol. 49, No. 3, March 2003

DEVARAJ AND KOHLI PerformanceImpacts of InformationTechnology:Is Actual Usage the Missing Link?

2 Figure

the Performance Vignette Illustrating Use of DSSforImproving Hospital

August 1. Jerry, the Heart Center manager, calls the DSS Analyst Clare to discuss how she can find out if the cardiac patients in the Progressive Care Unit (PCU) are getting the same level of care as the patients admitted through the Emergency Room. August 2. Clare ascertains that Jerry needs to examine the resource consumption profiles of the two patient types and the quality outcomes produced by the DSS report "DOSSIM." To enhance the integrity of the data she examines the principal diagnosis vs. admitting diagnosis to exclude patients who may have been admitted with a chest pain but were later found not to have a cardiac incident. In addition, she adds patient's severity index to the report so that resource consumption could be examined for comparable patients. The report is completed by the end of the day. August 3. Jerry and Clare discuss the results. Clare indicates that the patients in ER are sicker (indicated by higher severity index) and therefore their resource consumption is higher. However, she indicates that they are also being given a thrombotic drug that appears to improve the clinical outcomes. August 6. Jerry attends the regularly scheduled meeting of the physician committee and shares Clare's findings. Jerry also requests research papers on the use of thrombotic drugs from the library. August 10. The physicians' peer group, including the ER and cardiac physicians, meet. After discussing the evidence of the outcomes from the "DOSSIM" report combined with the published research, they decide to use a new treatment protocol for PCU patients.

support decision makers' creativity in problem solving (Elam 1990), and is thus related to the number of reports generated. Available in the time-sharing utility, CPU time also measures usage activity. CPU time is the amount of time consumed by the Central Processing Unit (CPU) for the processing and execution of a report. The variation in CPU time depends upon the number of variables utilized for the analysis, the duration for which analysis is conducted, and the number of records involved. These conditions account for the analytical intensity and result in decisions that are expected to impact organizational performance. CPU time, a

proxy for the intensity of usage, has been utilized as a measure of system performance in several past studies in the information systems (IS) measurement literature (Anderson and Chen 1992, Cao and Kanafani 1997, McKeen and Smith 1993). Disk Input/Output (I/O) is a measure of the number of records read and written in processing a report (LeFebvre 1999). The I/O measure is amount of detail, as opposed to complexity, of a report and can be a proxy for the length of the report. 3.4.2. Dependent Variables. In accordance with the prior literature in healthcare management, we

2 Table

Variables Description and Independent

variable Independent

Description

in References theliterature and 2000 Devaraj Kohli Barker LeFebvre 1999, 1999 and Anderson Chen Cao 1992, and Kanafani 1997, and McKeen Smith 1993

of a Reports report (REPORTSit_2) Numbertimes particular wasexecuted two periods forhospital i lagged time Disk Measurethenumber data of of records accessed Input-Output (I/0) i lagged time two periods orwritten hospital by (DISKlOit-2) Central Unit Computer utilized a thus Processing processing capacity time measure thecomplexity report of for ofthe (CPU) two periods i lagged time hospital (CPUit_2)

MANAGEMENTSCIENCE/Vol. 49, No. 3, March 2003

279

DEVARAJ AND KOHLI PerformanceImpacts of InformationTechnology:Is Actual Usage the Missing Link?

collected

data on various measures of hospital

performance.

3.4.2.1. Measuresof hospitalperformance. Traditionally, profitability measures, calculated by subtracting costs from the revenue, are utilized in measuring payoff. Profitability measures are well understood and accepted because they provide a net result of a business operation in creating business value. However, choosing profitability as a performance measure for hospitals has several weaknesses. First, due to lack of accurate cost-accounting systems, most hospitals use a fixed ratio of cost to charges (RCC), instead of true costs, to determine the cost of services. Furthermore, RCC is a direct transformation of revenue. Second, even when costs are calculated, they are allocated to chargeable items and are likely to change from one accounting period to another based upon the expenses incurred by departments for that particular period. Third, contractual agreements (i.e., discounts offered by hospitals to insurers) and write-offs for charity care affect the accurate costs of reimbursed services. Each of these deductions can obfuscate the profitability measures. To account for the above weaknesses, revenue is utilized as a dependent variable of hospital performance by previous studies in hospital profitability (Gapenski et al. 1993, Langland-Orban et al. 1996, Shi 1996, Teplensky et al. 1995). Following this literature, revenues were chosen as measures of finanTable 3 Variables Description and Dependent

cial performance. To minimize the bias resulting from the varying length of stay of patients, two measures of revenue were employed: net patient revenue per day (NPRDAY) and net patient revenue per admission (NPRADM). By agreement with the hospitals, these data had to be disguised by multiplying with a constant to protect the confidentiality of the data. It was verified that such a transformation did not affect the relationships between variables or the inferences drawn from the analyses. Also, as described earlier, technology use is related to quality outcomes for the hospitals. In this study, we utilize patient mortality rate as the measure of hospital quality. Table 3 provides a brief description of the dependent variables for profitability and quality. 3.4.3. Control Variables. Conceivably, the performance of healthcare organizations can be affected by a number of contextual variables other than technology usage. Therefore, an extensive literature survey of determinants of healthcare productivity and profitability was conducted. The list and labeling of control variables employed in this study reflect the extant literature in healthcare management (Gapenski et al. 1993, Langland-Orban et al. 1996, Shi 1996, Teplensky et al. 1995). Table 4 provides the control variables and their description. More detail on the control variables is included in the Appendix.

variable Dependent Net revenue patient per day(NPRDAYit)

Description of i Ratio thetotal revenue realized hospital by time to number of during t periods thetotal in under days theperiod consideration

in References theliterature Vogel, Gapenski, and 1993, Langland-Orban Langland-Orban, Gapenski, and Shi 1996, 1996, Vogel etal. Teplensky 1995

of Ratio thetotal revenue realized hospital i by to time number of during t periods thetotal admissions theperiod under during patient consideration Numbermortalities 30days of within rates for Selker Thomas, 1993, Mortality i during t periods an time of and 1993 hospital Holloway, Guire (MORTALITY,) divided operative procedure bythetotal number operative of conducted procedures inthetime under period consideration Net revenue patient admission (NPRADMit)

280

MANAGEMENT SCIENCE/Vol. 49, No. 3, March 2003

AND KOHLI DEVARAJ Is Performance Impacts Information of Technology: ActualUsagetheMissingLink?

4 Table

and Variables Description Control

Measure AGEit CASEMIXit

FTEit

Description i during periods time t of Age hospital i during periods mix time Case forhospital t of at Numberfull-time employeeshospital, time t during periods at of admissionshospitals Percent Medicare i time t during periods admissionshospital at i Percent Medicaid of time t during periods of for Per income theregion/market capita i during periods time t hospital revenue total revenue for Ratio outpatient of to i during period time t hospital

References literature inthe Gapenski, and Vagel, Langland-Orban etal.1996, 1993, Langland-Orban Shi1996, etal. Teplensky 1995

MEDICAREi, MEDICAIDi, INCOMEi,

OUTPATNTit

3.4.4. Lag Effects. While most IT payoff studies have tested for the impact of IT investments in a particular time period with performance in the same time period, it is conceivable that impacts may actually be observed at later stages. Longitudinal studies offer the advantage of modeling and testing such timelagged relationships. Peffers and Dos Santos (1996) concluded that impact of IT on performance in banks was after certain time lags and that early adopters gained significant benefits compared to late adopters. They also contend that cross-sectional studies, conducted soon after an application is installed, may fail 5 Table Proposed Hypotheses to find benefits even if large benefits are obtained. Hypothesis Description We employed three criteria to establish the credibilwill Total number DSS of accessed bepositively H1 reports ity and optimality of the lag effects considered: with associated NPRDAY a. field interviews with managers to assess their will H2 accessed bepositively Total number DSS of reports with associated NPRADM expert opinion on appropriate lags, will H3 Total number DSS of accessed benegatively b. literature review of studies in healthcare that reports with associated MORTALITY document time-lagged effects on hospital and perforH4 with CPU will positively time be associated NPRDAY mance, and with H5 CPU will positively time be associated NPRADM with H6 CPU will negatively time be associated MORTALITY c. use of statistical criteria (Akaike's Information Criteria and Schwarz's Criteria) (Akaike 1974, H7 Total number records input of for or accessed output to will with written disk bepositively associated Schwarz 1978) to compare models with varying lag NPRDAY effects to select the best-fitting model. H8 Total number records input of for or accessed output Given that the DSS infrastructure to support clinwill with written disk bepositively to associated ical and financial initiatives is already in place and NPRDAM for H9 Total number records input of accessed output or months of careful planning has already taken place, to will with written disk benegatively associated the DSS usage is highly focused and geared to idenMORTALITY tifying changes in a short period of time. This is in In ?2, we presented the proposition that actual usage of technology by an organization will be associated with higher financial and nonfinancial performance after controlling for the effect of various external variables. Therefore, our nine hypotheses, specified in Table 5, propose that each independent variable (Reports, CPU Time, and Disk I/O) will be associated with each measure of hospital performance (NPRADM, NPRDAY, and MORTALITY).
MANAGEMENT SCIENCE/Vol. No. 3, March2003 49,

281

DEVARAJ AND KOHLI PerformanceImpactsof InformationTechnology:Is Actual Usage the Missing Link?

contrast to start-up organizations where the infrastructure deployment can take several months or even years. Managers responsible for implementing the improvement initiatives also supported this. One manager stated that
...the impact resulting from the change in practice patterns will appear shortly, approximately two months in most cases, because revenue is booked right after the patient is treated. Therefore the lags are primarily due to the physicians incorporating the new process into practice and the time involved in compiling medical and billing records.

4.1. Specification Tests Causality Tests. Longitudinal analysis presents

some challenges in the estimation procedure as compared to cross-sectional data because there is an implicit assumption of causality in the analyses. Therefore, a useful specification test is to check for reverse causality. In other words, can the same or similar results be obtained by reversing the dependent and independent variables. A standard statistical procedure to address this issue is the Granger Causality Test. We were unable to reject the null hypothesis that usage "granger causes" performance. Therefore, we believe that it is not performance that drives technology usage, but vice versa. Omitted Variable Tests. The omitted variable test is a test of whether a set of variables that is added to an existing equation makes a significant contribution to explaining the variation in the dependent variable. The null hypothesis is that the additional variables are jointly not significant. Such tests were conducted for all three performance variables. F-statistics and likelihood ratios based on the difference between the residual sum of squares of the two models did not lead to a rejection of the null hypothesis. 4.2. Diagnostic Checks on Residuals Several diagnostic checks were done on the residuals to insure that the assumptions of time-series analyses are not violated. Specifically, tests for serial correlation, normality, and heteroscedasticity were performed. An observation of correlograms as well as Q-statistics confirmed that the final models did not exhibit serial correlation. Given that the data constitute a panel dataset, we examined the Durbin-Watson (DW) statistic for each hospital separately. Such an examination revealed that the DW statistic was not significantly different from 2 for all the hospitals, indicating that serial correlation was not a problem. Normality was tested using histograms as well as the Kolmogorov-Smirnov test. The p-values obtained did not indicate a violation of the normality assumption. White's test (White 1980) is a test of the null hypothesis of no heteroscedascity against heteroscedascity of some unknown general form. White's test statistic, as well as the reported F-statistic, did not indicate heteroscedascity to be a problem.

Further detailed interviews with managers suggested that due to the billing cycles involved it takes about 45-60 days for the financial accounting systems to reflect any benefit from system usage. Published studies in clinical settings suggest lag effects of less than six months. Shorter lead times in clinical settings can be ascribed to the implementation of "evidence-based practice" that promotes making immediate process changes based on published clinical evidence (Silagy and Lancaster 1995). For example, process changes in the emergency department resulted in a reduction of laboratory utilization with a lag of six months (Dickinson 1987) and respiratory infections were reduced significantly after a lag of four months of a redesign initiative (Joiner et al. 1996). Our analyses of lag effects using all the criteria listed above suggested a two-period lag of the usage variables. That is, we test the impact of technology usage in time period t- 2 on hospital performance in time period t.

4.

Models based upon the system of equations represented in Figure 1 specify the relationships between the use of technology and financial and nonfinancial measures of performance, controlling for various external factors. We employ time-series analyses to estimate the different models. Specifically, we estimated a fixed effects model for each dependent variable. This model is appropriate because it accounts for differences across hospitals by incorporating a separate intercept term for each hospital (Greene 1993).

Data Analysis

282

MANAGEMENTSCIENCE/Vol. 49, No. 3, March 2003

DEVARAJ AND KOHLI PerformanceImpacts of InformationTechnology:Is Actual Usage the Missing Link?

Table 6
Variable

Statistics Correlations and Descriptive

NPRADMNPRDAYMORTALITY MEDICAID REPORTS CPU DISKIO AGE CASEMIX FTE MEDICARE INCOME

NPRADM NPRDAY 0.82* MORTALITY -0.00 REPORTS 0.33*

-0.17 0.34*

-0.33*

CPU

0.14

-0.01
-0.09 0.28* 0.16* -0.35* 0.55* 0.18* -0.25* -0.28*

-0.11
-0.09 -0.17 0.46* 0.29* 0.02 0.09 -0.05 0.09

0.51*
0.46* 0.32* 0.31* 0.03 0.46* 0.16 -0.27* 0.06 0.39* 0.37* 0.28* 0.21* 0.23* 0.03 0.29* 0.12 0.18* 0.07 0.19* 0.39* 0.17 0.12 0.40* -0.29* -0.37* -0.54* 0.41* 0.22* 0.49*

0.05 DISKIO AGE 0.12 CASEMIX 0.36* FTE -0.23* MEDICARE 0.71* MEDICAID 0.26* INCOME -0.33 OUTPATNT-0.04 *

0.53* 0.65* 0.56* -0.45* 0.15*

-0.02 0.45* -0.07 0.70*

0.49* -0.57* -0.11

-0.40* 0.08

-0.14

at Significant the0.05 level.

4.3.

Results

Descriptive statistics and pairwise correlations between variables employed in this study are shown in Table 6. The correlations indicate that while the two revenue-based dependent variables are related significantly, mortality as a quality outcome might be independent of these financial measures. Also, among the control variables, Casemix and FTE are correlated significantly with most of the variables highlighting the need to test for multicollinearity in the estimation models. The estimation results for number of reports accessed and executed (a measure of technology usage) as the dependent variable is shown in Table 7. Technology usage, as measured by the number of Table 7

reports accessed, was positively and significantly associated at the 0.05 significance level with NPRDAY, NPRADM, and MORTALITY.Thus, there is strong statistical support for the principal link between technology usage and performance. Variables that significantly affect hospital performance, in addition to technology use, are MEDICARE, FTE, and CASEMIX. These independent variables are included in the study primarily as control variables and are not the focus of this study. Therefore, for the sake of brevity, detailed implications of these findings are not discussed. The second set of results corresponding to CPU time as a measure of technology usage is presented in Table 8. The use of technology significantly affects NPRDAY and NPRADM but not MORTALITYat the Estimation Results: Technology MeasuredCPU Usage by Time
variables Dependent variables Independent MORTALITY NPRDAY 0.379* -3.10 -718.51 -250.56 0.356* 149.78 -0.003 0.126 0.828 NPRADM 0.178* 51.59 2,680.2 130.32 1.856* 1,632.1 -0.002 0.129 0.861

Estimation Results: Technology Measured Number Usage by of Reports Table 8

variables Dependent MORTALITY NPRDAY NPRADM 3.122* -0.010* 14.437* 0.025 3.313 87.13 0.060 -962.35 1,260.8 0.305 -340.19 -1,088.4 -0.001 0.295 1.79* 0.114 355.18 2,367.5 0.001 -0.001 -0.020 0.001 0.126 0.040 0.268 0.804 0.883

variables Independent

of Usage(number reports executed) Age Medicare Medicaid FTE Casemix Outpatient Income Adjusted R-square * at Significant the0.05 level.

time -0.366 Usage(CPU utilized) -0.013 Age Medicare 0.175 Medicaid 0.392 FTE -0.001 Casemix 0.081 -0.001 Outpatient Income 0.010 0.212 Adjusted R-square * at Significant the0.05 level.

MANAGEMENT SCIENCE/Vol. 49, No. 3, March2003

283

DEVARAJ AND KOHLI PerformanceImpactsof InformationTechnology:Is Actual Usage the Missing Link?

Table 9

Estimation Results: UsageMeasured Disk Technology by Input/Output variables Dependent

Table 10

Results Hypotheses of Description Estimation results (at0.0level)

Hypothesis

variables Independent

MORTALITY NPRDAY NPRADM H1 H2 H3 H4 H5 H6

of I/0 -0.014* 69.02* 270.79* Usage (numberdisk performed) -0.010 -33.45 -87.01 Age Medicare 0.032 -835.85 1,999.3 Medicaid 0.454 -246.89 58.80 FTE -0.001 0.362* 1.87* Casemix 0.101 275.55 2,197.8 0.002 -0.005 -0.005 Outpatient Income 0.001 0.034 1.091 0.819 0.303 0.840 Adjusted R-square *Significant 0.05level. atthe

0.05 level. Thus, there is support for the technology usage-financial performance link from these results as well. The control variable FTE has a significant impact on performance. Finally, Table 9 presents results with Disk Input/ Output as a measure of technology usage. As can be observed, usage has a strong significant relationship with all three measures of hospital performance at the 0.05 level. The effect of control variables is very similar to results obtained with CPU as the usage measure. In general, results obtained with all three technology usage variables provide support for the proposition that the actual use of technology can have a significant positive impact on financial outcomes as well as quality outcomes of hospital performance. A summary of results obtained for the proposed hypotheses are presented in Table 10. The statistical results presented in Tables 7-10 are indicative of the statistical significance of the relationship between usage and measures of hospital performance.2 However, the economic or practical significance of the results is not borne out in the tables. Because the confidential nature of the data required us to disguise the data, the value of the coefficients should not be
While a refers to the probabilityof a Type I error,P refersto the probabilityof a TypeII error.A TypeII erroroccursif we acceptthe null hypothesis when it is false. The power of a test, (1 - P), is the probabilitythat the hypothesis test will rejectthe null hypothesis when it is false. We insured that the statisticalpowers of the tests were within acceptablebounds.
2

H7

H8

H9

Total of number DSS reports Supported will accessed bepositively with associated NPRDAY Total number DSS of reports Supported will accessed bepositively with associated NPRADM Total number DSS of reports Supported accessed benegatively will with associated MORTALITY CPU will positively time be associated Supported with NPRDAY CPU will positively time be associated Supported with NPRADM CPU will negatively time be Not supported with associated MORTALITY Total number records input of for Supported accessed output or written disk to will positively be with associated NPRDAY Total number records input of for Supported accessed output or written disk to will positively with be associated NPRADM Total of number records input for Supported or written disk to accessed output will negatively with be associated MORTALITY

interpreted directly as the increase in the dependent variable for a unit increase in the independent variable holding all else constant. However, a separate set of analyses of the undisguised data indicated actual usage had a significant economic impact on the two measures of revenue. These results were presented to hospital administrators and have received further validation. We also tested other models with mortality as the dependent variable using the undisguised data (not reported here) and found usage positively correlated with mortality. Since mortality measures are on a more tightly spread scale, even a 0.05% annual decrease in mortality represents several human lives and is a favorable outcome for the hospitals. The average increase in R-square after the addition of the usage variable in the models for NPRDAY and

284

MANAGEMENTSCIENCE/Vol.

49, No. 3, March 2003

AND KOHLI DEVARAJ

PerformanceImpacts of InformationTechnology:Is Actual Usage the Missing Link?

NPRADM was 12.2% and 14.7%, respectively, while the increase in R-square for Mortality was 9.8%. Our analyses indicated strong support for the usage-performance link in the hospitals included in the study. An interesting question to investigate might be if this represents an improvement over conventional measures such as IT investment. To address this, we carried out similar analyses but included a measure of IT labor costs, for which data were available, in place of the usage variables. In the interest of brevity, we do not present results for these models. We observed that the IT investment variable was not statistically significant in the estimated models. Thus, we believe that capturing usage might provide insights that constitute an improvement over traditional measures of IT investment.

5. Discussion and Conclusions

The results of this study provide general support for the proposition that the greater the actual usage of technology, the better the financial and quality performance of hospitals. These results were robust to the measure of technology usage employed. 5.1. Implications Several managerial implications follow from the results of this study. First, there is evidence that investments in technologies have positive payoffs when actual usage of the technology is considered. While past studies may not have observed a positive impact in a given cross-sectional time period, the longitudinal nature of data analyzed in this study enables us to detect significant effects on various measures of hospital performance. Our research design, which utilizes objective usage measures captured by the DSS, also frees managers from the responsibility of periodically collecting data from users. Asking users to frequently report usage data can be a time imposition, as well as less reliable over time because many users may not be able to recall the changes in their usage patterns over time. IT managers have struggled for ways to justify investments in technologies. To that effect, this paper presents an analysis of the performance impacts of technologies

in a field setting. The methodology accounts for other contextual variables that may also impact organizational performance. This study presents evidence for the monetary impact and temporal impact due to the use of technology. Because the data analyzed were disguised by multiplying with a constant, actual estimates of monetary impacts are not possible. However, in separate analyses of the unmodified data presented to management of the hospitals, it was straightforward to compute the impact of usage on financial performance using coefficients from the estimation models. Additionally, the results of this study suggest a temporal aspect to technology payoff; i.e., payoffs may not be realized instantaneously, but only after certain periods of time. There is a need in the IT literature to examine longitudinal firm-level data to observe such lagged effects. Also, it is possible that technology usage has a contingent impact on organizational performance. Most studies have attempted to assess the relationship between technology investments and organizational performance directly. However, the process-based approach proposed by Soh and Markus (1995) and Weill (1992) suggests that technology's impact on organizational performance may be mediated through another variable, such as usage reported in our study. Traditionally, investment in technology has been viewed as a black box and its impact on performance measured with little context. Actual usage of IT places the investment in perspective and examines the "missing link," which is often estimated through the expected use. As is the case with all organizations, hospitals benefit from quantification of the usage by justifying the IT investment, which often competes with investment in clinical technologies. A quantitative estimate of the impact of IT investment through usage can also facilitate an accurate cost-benefit analysis for future investment in IT. The IT payoff literature has documented mixed findings on the impact of IT investment on performance. However, recent studies seem to suggest that, discounting for contingency effects, firm-level studies may offer opportunities to accurately observe IT impacts. This study extends the above reasoning to suggest that, regardless of the unit of analysis, the

MANAGEMENT SCIENCE/Vol. No. 3, March 2003 49,

285

DEVARAJ AND KOHLI PerformanceImpactsof InformationTechnology:Is Actual Usage the Missing Link?

ability to capture and model the actual usage of technology may be key in assessing the relationship between implementing information technologies and benefits or payoffs resulting from them. 5.2. Limitations and Future Research This study employs data from hospitals belonging to a health system network; therefore, the principal limitation of this study is in the generalizability of its findings to other healthcare organizations or other industries. This, however, is a limitation of field studies in general. On the other hand, field studies such as the one reported here have the advantage of providing a richer operationalization of reality and the ability to track detailed data over time. As such, the findings reported here cannot be generalized to the larger population of hospitals or extended to other types of firms in other industries. However, while the results are not generalizable, the approach presented in this study can be generalized to other contexts. Our use of revenue as a measure of organizational performance has limitations because it does not take into account the costs of the business operation. Therefore, it is better viewed as a measure of organizational performance than a measure of organizational profitability. Organizations that have reliable cost information and limited charitable and bad-debt occurrence may use profitability measures as a dependent variable. To the best of our knowledge, this is one of the first studies to examine actual usage of a specific technology in a detailed and longitudinal manner. Future work in this vein should be performed and such analyses will provide the ability to detect payoffs from the actual use of technology over an extended period of time and across multiple measures of performance. Large sample cross-sectional studies in conjunction with longitudinal studies are called for to examine the IT payoff issue in detail and be able to make generalizations across industries, firms, etc. The random selection of companies through a survey-based approach would provide the ability to make generalizations across the population of industries or firms under consideration. A comprehensive framework that remains to be tested is one that incorporates perceptual variables that reflect users' attitudes towards technologies,

actual usage of the technology, and its impact on individual and organizational performance. Similarly, usage of technology for particular processes can give additional insights into organizational impacts. Acknowledgments
The authors thank Hank Groot for his continued support of this research project and Frank Piontek for his insights and suggestions on clinical research. They acknowledge the assistance of Don Irmiger, Linda Martin, Doug Anthony, Clare Sobieralski, and Diana Utterback in obtaining the data. This paper has also benefited from comments from Detmar Straub, Moez Limayem, Khalil Matta, and Kajal Mukhopadhyay.

Appendix:

Control Variables

Service Index (CASEMIX). The service index, or the casemix index, is a measure of the range of services offered by the hospital. The higher this measure, the more complex and resource intensive the services rendered by the hospital. This index factors in the severity and resource demands of incoming patients. Age (AGE). The healthcare management literature suggests that newer hospitals may be better positioned to acquire and use recent technologies than older and established hospitals. Therefore, the age of the hospital was included as a control variable in the estimation models. Labor Intensity (FTE). The number of full-time employees per patient day provides a measure of the labor intensity. The relationship between personnel efficiency and profit margins has been of interest to healthcare management researchers and thus should be controlled for. The next two variables assess the extent to which services offered by the hospital were related to Medicare and Medicaid patients. The reasons for controlling for these effects is because: (1) Medicare and Medicaid payments are typically less than payments from other payers for similar services, and (2) Medicaid and Medicare patients are more costly to treat than other patients (Gapenski et al. 1993). Medicare (MEDICARE). Percent of admissions that are Medicare patients compared to total admissions. Medicaid (MEDICAID). Percent of admissions that are Medicaid patients compared to total admissions. Outpatient Mix (OUTPATNT). There is a general belief in healthcare management that outpatient services are more profitable than inpatient services. To control for the existence of any such effect on profitability, the ratio of outpatient revenue to total revenue was included as a control variable.3 3A measure for competition in the market was also computed. This measure, termed Herfindahl's Index in the healthcare literature (Ashton and Press 1997, Nauenberg et al. 2001) is computed as a function of the beds in the market and beds in a specific hospital. However, due to minimal variation across the various time periods, the inclusion of this variable presented estimation problems. Hence, this variable was dropped from the analyses.

286

MANAGEMENTSCIENCE/Vol. 49, No. 3, March 2003

DEVARAJ AND KOHLI PerformanceImpacts of InformationTechnology:Is Actual Usage the Missing Link?

Per Capita Income (INCOME). It is conceivable that with higher patient incomes, hospitals may be able to charge more for their services as well as lose less due to bad debts. For this reason, the per capita income of region/market for each of the hospitals was included in the analyses.

References
Agarwal, R., J. Prasad. 1997. The role of innovation characteristics and perceived voluntariness in the acceptance of information technologies. Decision Sci. 28 557-582. Akaike, H. 1974. A new look at statistical model identification. IEEE Trans.Automatic Control 19 716-723. Anderson, E. E., Y. M. Chen. 1992. A simultaneous equation model of cost performance relationships in large-scale computing. IIE Trans. 24(5) 155-165. Ashton, T., D. Press. 1997. Market concentration in secondary health services under a purchaser-provider split: The New Zealand experience. Health Econom. 6(1) 43-56. Baily, M. N. 1986. What has happened to productivity growth? Science 234 443-451. Barker, R. 1999. I/O, I/O, it's faster to work we go. UNIX Review's PerformanceComput. 17(6) 13-15. Barua, A., C. Kriebel, T. Mukhopadhyay. 1995. Information technologies and business value: An analytic and empirical investigation. Inform. Systems Res. 6(1) 3-24. Berndt, E. R., C. J. Morrison. 1995. High-tech capital formation and economic performance in U.S. manufacturing industries: An exploratory analysis. J. Econometrics65 9-43. Bharadwaj, A. S., S. G. Bharadwaj, B. R. Konsynski. 1999. Information technology effects on firm performance as measured by Tobin's q. Management Sci. 45(7) 1008-1024. Brynjolfsson, E. 1996. The contribution of information technology to consumer welfare. Inform. Systems Res. 7(3) 281-300. S. Yang. 1996. Information technology and productivity: A -, review of literature. Adv. Comput. 43 179-214. Butters, S., S. Eom. 1992. Decision support systems in the healthcare industry. J. Systems Management 43(6) 28-31. Cao, J. M., A. Kanafani. 1997. Real-time decision support for integration of airline flight cancellations and delays. Transportation Planning Tech.20(3) 201-217. Davis, F. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quart. 13(3) 319-340. Devaraj, S., R. Kohli. 2000. Information technology payoff in the healthcare industry: A longitudinal study. J. Management Inform. Systems 16(4) 41-67. Dewan, S., C. Min. 1997. The substitution of information technology for other factors of production: A firm level analysis. Management Sci. 43(12) 1660-1675. Dickinson, J. C. 1987. From process to policy: A generic prescription for test over-utilization in the emergency department. Family Practice Res. J. 7(1) 12-21.

Diewert, E. W., A. M. Smith. 1994. Productivity measurement for a distribution firm. National Bureau of Economic Research working paper no. 4812, Washington, D.C. Doll, W. J., G. Torkzadeh, 1998, Developing a multidimensional measure of system-use in an organizational context. Inform. Management 33(4) 171-185. Elam, J. J., M. Mead. 1990. Can software influence creativity. Inform. Systems Res. 1(1) 1-22. Forgionne, G., R. Kohli. 1996. HMSS: A management support system for concurrent hospital decision-making. Decision Support Systems 16(3) 209-229. Francalanci, C., H. Galal. 1998. Information technology and worker composition: Determinants of productivity in the life insurance industry. MIS Quart. 22(2) 227-241. Gapenski, L., B. Vogel, B. Langland-Orban. 1993. The determinants of hospital profitability. Health Hospital Admin. 38(1) 63-80. Goodhue, D. L., R. L. Thompson. 1995. Task technology fit and individual performance. MIS Quart. 19(2) 213-236. Greene, W. H. 1993. EconometricAnalysis. Macmillan, New York. Hartwick, J., H. Barki. 1994. Explaining the role of user participation in information system use. Management Sci. 40 440-465. Hitt, L., E. Brynjolfsson. 1995. Productivity, business profitability, and consumer surplus: Three different measures of information technology value. MIS Quart. 20(2) 121-142. Igbaria, M., N. Zinatelli, P. Cragg, A. Cavaye. 1997. Personal computing acceptance factors in small firms: A structural equation model. MIS Quart. 21(3) 279-305. Joiner, G. A., D. Salisbury, G. E. Bollin. 1996. Utilizing quality assurance as a tool for reducing the risk of nosocomial ventilator-associated pneumonia. Amer. J. Medical Quality 11(2) 100-103. Keen, P. G. W. 1981. Value analysis: Justifying decision support systems. MIS Quart. 5(1) 1-15. M. S. Scott Morton. 1978. Decision Support Systems: An Orga-, nizational Perspective. Addison-Wesley, Reading, MA. Kelley, M. 1994. Productivity and information technology: The elusive connection. Management Sci. 40(11) 1406-1425. Kohli, R., S. Devaraj. 2003. Measuring information technology payoff: A meta-analysis of structural variables in firm-level empirical research. Inform. Systems Res. Forthcoming. Koski, H. 1999. The implications of network use, production network externalities and public networking programmes for firm's productivity. Res. Policy 28(4) 423-439. Langland-Orban, B., L. C. Gapenski, W. B. Vogel. 1996. Differences in characteristics of hospitals with sustained high and sustained low profitability. Hospital Health Services Admin. 41(3) 385-399. LeBlanc, L. A., K. A. Kozar. 1990. An empirical-investigation of the relationship between DSS usage and system performance: A case-study of a navigation support system. MIS Quart. 14(3) 263-277.

MANAGEMENTSCIENCE/Vol. 49, No. 3, March 2003

287

DEVARAJ AND KOHLI PerformanceImpactsof InformationTechnology:Is Actual Usage the Missing Link?

Lee, B., A. Barua. 1999. An integrated assessment of productivity and efficiency impacts of information technology investments: Old data, new analysis and evidence. J. ProductivityAnal. 12(1) 21-43. LeFebvre, W. 1999. Measuring and changing system performance. UNIX Review's PerformanceComput. 17(3) 60-63. Loveman, G. W. 1994. An Assessment of the ProductivityImpact of the InformationTechnologies.Oxford University Press, New York. Lucas, H. C., Jr., V. K. Spitler. 1999. Technology use and performance: A field study of broker workstations. Decision Sci. 30(2) 291-311. McKeen, J. D., H. A. Smith. 1993. The relationship between information technology use and organizational performance. R. Banker, R. Kauffman, M. A. Mahmood, eds. Strategic Information Technology Management. Idea Group, Harrisburg, PA, 405-445. Menon, N. M., B. Lee, L. Eldenburg. 2000. Productivity of information systems in the healthcare industry. Inform. Systems Res. 11(1) 83-92. Morrison, C., E. Berndt. 1991. Assessing the productivity of information technology equipment in U.S. manufacturing industries. National Bureau of Economic Research working paper no. 3582, Washington, D.C. Mukhopadhyay, T., S. Kekre, S. Kalathur. 1995. Business value of information technology: A study of electronic data interchange. MIS Quart. 19(2) 137-156. , S. Rajiv,K. Srinivasan. 1997. Information technology impact on process output and quality. ManagementSci. 43(12) 1645-1659. Nauenberg, E., Y. Andrijuk, M. Eisinger. 2001. Reconsideration of discharge data to measure competition in the hospital industry. Health Econom. 10(3) 271-276. Papp, R. 1999. Business-IT alignment: Productivity paradox payoff? Indust. Management Data Systems 99(7-8) 367-373. Peffers, K., B. L. Dos Santos. 1996. Performance effects of innovative IT applications over time. IEEE Trans Engrg. Management 43(4) 381-392. Prasad, B., P. Harker. 1997. Examining the contribution of information technology toward productivity and profitability in U.S. retail banking. Working paper no. 97-09, Financial Institutions Center, The Wharton School, University of Pennsylvania, Philadelphia, PA. Prattipati, S., M. Mensah. 1997. Information systems variables and management productivity. Inform. Management 33(1) 33-43. Rao, H. R, C. C. Pegels, A. F. Salam, K. T. Hwang, V. Seth. 1995. The impact of EDI implementation commitment and implementation success on competitive advantage and firm performance. Inform. Systems J. 5(3) 185-202. Roach, S. 1987. America's technology dilemma: A profile of the information economy. Special Economic Study, Morgan Stanley, New York. Schwarz, G. 1978. Estimating the dimension of a model. Ann. Statist. 5 461-464.

Selker, H. P. 1993. Systems for comparing actual and predicted mortality rates: Characteristics to promote cooperation in improving healthcare. Ann. Internal Medicine 118 820-822. Sharda, R. M, S. H. Barr, J. C McDonnell. 1988. Decision support system effectiveness: A review and empirical test. Management Sci. 34(1) 139-159. Shi, L. 1996. Patient and hospital characteristics associated with average length of stay. Health CareManagemantRev. 21(2) 46-61. Siegel, D., Z. Griliches. 1992. Purchased services, outsourcing, computers, and productivity in manufacturing. Z. Griliches, ed. Output Measurement in the Service Sectors. University of Chicago Press, Chicago, IL, 429-458. Silagy, C., T. Lancaster. 1995. The cochrane collaboration in primary care: An international resource for evidence-based practice of family medicine. Family Medicine 27(5) 302-305. Soh C., M. Markus. 1995. How IT creates business value: A process theory synthesis. Proc. Sixteenth Internat. Conf. Information Systems. Amsterdam, The Netherlands, 29-41. Sohal, A. S., S. Moss, L. Ng. 2000. Using information technology productively: Practices and factors that enhance the success of IT. Internat. J. Tech.Management 20(3-4) 340-353. Strassman, P. A. 1990. The Business Value of Computers.Information Economics Press, New Cannan, CT. Straub, D., M. Limayem, E. Karahanna-Evaristo. 1995. Measuring system usage: Implications for IS theory testing. Management Sci. 41(8) 1328-1342. Szajna, B. 1996. Empirical evaluation of the revised technology acceptance model. Management Sci. 42(1) 85-92. Taylor, S., P. Todd. 1995. Understanding information technology usage: A test of competing models. Inform. Systems Res. 6(2) 144-176. Teplensky, J. D., M. V. Pauly, J. R. Kimberly, A. L. Hillman, J. S. Schwartz. 1995. Hospital adoption of medical technology: An empirical test of alternative models. Health Services Res. 30(3) 437-465. Thomas, J. W., J. J. Holloway, K. E. Guire. 1993. Validating riskadjusted mortality as an indicator of quality of care. Inquiry 30 6-22. U.S. Department of Commerce. 1998. The emerging digital economy. Report ESA-OPD 98-4, http://www.ecommerce.gov. Washington, D.C. 2001. U.S. Corp R&D advance estimate. http://www.ta.doc. -. gov/ reports/techpolicy/us_corp_est_Oll11015.pdf. Washington, D.C. Venkatesh, V., F. D. Davis. 2000. A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management Sci. 46(2) 186-204. Wagner, J. A., III, R. Z. Gooding. 1987. Effects of societal trends on participation research. Admin. Sci. Quart. 32 241-262. Weill, P. 1992. The relationship between investment in information technology and firm performance: A study of the valve manufacturing sector. Inform. Systems Res. 3(4) 307-333.

288

MANAGEMENT SCIENCE/Vol. 49, No. 3, March 2003

DEVARAJ AND KOHLI PerformanceImpacts of InformationTechnology:Is Actual Usage the Missing Link?

White, H. 1980. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica 48(4) 817-838.

Zarling, E., F. Piontek, R. Kohli. 1999. The utility of hospital administrative data for generating a screening program to predict adverse outcomes. Amer. J. Medical Quality 14(6) 242-247.

receivedOctober9, 2000. This paper was with the authors 13 monthsfor 2 revisions. Accepted by Chris F. Kemerer;

MANAGEMENTSCIENCE/Vol. 49, No. 3, March 2003

289