A Randomized Controlled Trial of A Computer-Based Physician Workstation in An Outpatient Setting: Implementation Barriers To Outcome Evaluation
A Randomized Controlled Trial of A Computer-Based Physician Workstation in An Outpatient Setting: Implementation Barriers To Outcome Evaluation
A Randomized Controlled Trial of A Computer-Based Physician Workstation in An Outpatient Setting: Implementation Barriers To Outcome Evaluation
Research Paper
A Randomized Controlled
Trial of a Computer-based
Physician Workstation in an
Outpatient Setting:
Implementation Barriers to
Outcome Evaluation
Affiliations of the authors: Veterans Affairs Palo Alto Health Care Fellowship Program. Supported by a grant from the Am-
Care System, Palo Alto, CA (BLR, ANS, TWM, DKO); Division bulatory Care Education Initiative, Department of Veterans Af-
of General Internal Medicine, Stanford University, Stanford, CA fairs.
(BLR, TWM, DKO); Department of Health Research and Policy, Correspondence and reprints: Douglas K. Owens, MD, MSc,
Stanford University, Stanford, CA (BWB, DKO) Hewlett-Pack-
Section of General Internal Medicine (lllA), Veterans Affairs
ard Laboratories, Palo Alto, CA (PDS, DG, MCH, HJS, CY).
Medical Center, 3801 Miranda Avenue, Palo Alto, CA 94304.
e-mail: [email protected]
Dr. Owens is supported by a Career Development Award from
the VA Health Services Research and Development Service. Drs. Received for publication: 12/6/95; accepted for publication:
Rotman and McDonald were supported by the VA Ambulatory 5/14/96.
Journal of the American Medical Informatics Association Volume 3 Number 5 Sep / Oct 1996 341
Collecting, processing, and recording information zation of users to experimental and control groups,
during an outpatient visit is already difficult. It is the traditional approach for clinical evaluations, is the
growing more complicated because more health care most suitable method for ensuring comparability.
services are delivered in the outpatient arena, an ag- However, experimental designs must allow user ac-
ing population has more chronic diseases, and ex- cess to the existing system unless the new system con-
panding medical technologies have created more di- tains all the clinically relevant information available
agnostic and therapeutic options to know and use. In through the existing one. A second threat to the va-
addition, with the advent of restrictions on govern- lidity of an evaluation of clinical information systems
mental funding for health care and the growing prom- is the secular trend: changes in the clinical environ-
inence of managed care, health care providers are fac- ment that occur during the study period and that may
ing increasing pressure to maximize their efficiency. affect the study outcomes. For example, the results of
New clinical information systems can assist providers a study of the influence of a computer-based clinical
with this challenge by enhancing data management information system on pharmacy costs may be con-
and providing decision support. founded if changes in the hospital formulary during
Prior evaluations of clinical information systems and the study period reduce (or increase) drug costs. Ap-
expert systems that provide patient-specific advice propriately designed control groups enable investi-
have found both successes and failures.lm6 A system- gators to control and adjust for such confounding
atic review found that three of four studies of com- trends in the analysis of results.
puter-assisted drug dosing, one of five studies of com- In the Physician Workstation (PWS) project, we eval-
puter-aided diagnosis, seven of nine studies of uated an innovative, computer-based PWS designed
computer-aided active medical care, and four of six to facilitate essential information-processing activities
studies of computer-assisted preventive care showed in ambulatory care. An internally developed research
improvement in clinicians’ performance.’ Only three prototype,10-16 the PWS was designed based on eth-
of ten studies that examined patient outcomes found nographic studies during a process of formative eval-
improvements, however.5 Information systems also uations.13”7 We evaluated the PWS in a two-period,
reduce costs in certain settings. For example, Tierney parallel design that assigned subjects randomly, con-
and colleagues conducted a randomized controlled trolled for secular trends, and maximized statistical
trial of computer-based feedback during order entry power. The objective of our study was to evaluate the
in a hospital that showed a reduction in drug effects of PWS on user satisfaction, on health-related
charges.” outcomes, and on costs. Because the intervention sys-
Rigorous evaluation of clinical information systems is tem was used infrequently, we were not able to eval-
important for two reasons7 Simply providing more uate whether such an intervention, if used, would af-
information, or providing it in a more usable manner, fect health or economic outcomes. This paper reports
may not be sufficient to improve clinical outcomes or the study design, describes the implementation prob-
to reduce costs. Provider behavior is notoriously dif- lems that contributed to low usage, and recommends
ficult to influence.” Also, innovative systems may ways to avoid these problems in evaluative studies.
have unexpected limitations, even if these systems
have certain clear advantages over current ones.7 Lab- Methods
oratory evaluations of clinical information systems are
insufficient for determining clinical utility.’ Thus, it is
Clinical Environment
essential to demonstrate that an information technol-
ogy actually change health and economic outcomes. We performed our study in the General Medical
As with studies! of other medical interventions, eval- Clinic (GMC) at the Veterans Affairs (VA) Palo Alto
uations of information systems may be subject to bi- Health Care System from July 1994 to June 1995. The
ases that undermine the validity of their results.’ Two GMC provides continuity care for a patient popula-
threats to validity warrant particular attention. In a tion of predominantly elderly men who have multiple
study comparing two groups of users, the users in chronic illnesses. During the study, 2,071 patients
each group should be comparable in terms of skill, were seen in the GMC. Our study subjects were 37
training, interest; and clinical needs. Otherwise, selec- Stanford University Internal Medicine residents. Each
tion bias may occur because the users of the new sys- resident cared for GMC patients one half-day per
tem are those with particular a interest or aptitude. week. Each examination room contained a computer
This type of selection bias is particularly relevant to terminal connected to the hospital’s medical infor-
evaluations of clinical information systems, and it mation system. For the intervention period, we added
may lead to speciously optimistic results. Randomi- PWS terminals and printers to the examination rooms
342 ROTMAN ET AL., Physician Workstation Evaluation
training. For each physician, we calculated the differ- We developed a definition of drug-drug interactions
ence in the outcome variables between the two time that emphasized clinically relevant interactions. Using
periods. the Drug Therapy Monitoring System (DTMS), a com-
mercial database developed by Medispan, we selected
We chose to assess the effect of the PWS system on only level-l interactions, the most dangerous and
drug costs, drug-drug interactions, and user satisfac- best-documented category of the five possible levels.
tion. We analyzed GMC prescription data from the Additionally, we defined a clinically relevant interac-
1992-93 academic year to select outcome variables tion as one with,no laboratory evidence that the pro-
that would be feasible for analysis and sensitive to our vider monitored the patient for adverse events related
intervention.‘“,” Because our intervention targeted to the interaction.” Many interactions, even those at
physicians’ prescribing behavior, we expressed our level 1, are inherent in the practice of medicine and
drug cost and drug interaction variables in terms of may cause no morbidity if the provider carefully mon-
quantities prescribed by each physician. itors for potential deleterious effects by performing
laboratory studies. For example, if a physician gen-
To derive drug costs, we performed a two-step pro-
erated a level-l interaction by concurrently prescrib-
cess, generating a mean daily cost per patient and
ing warfarin and allopurinol, the increased risk of
then aggregating each physicians’ panel of patients to
bleeding would require laboratory monitoring. For
generate a mean daily cost per physician per patient.
this interaction, we examined the patient’s laboratory
This procedure allowed us to correct for the fact that
data for evidence of a prothrombin-time study within
different physicians cared for different numbers of pa-
14 days of issuing the interacting prescriptions. If
tients and for patients who were not enrolled in the
these data were absent, we assumed that the provider
clinic for the entire study. We used drug purchase
prices at our institution. We calculated each patient’s was unaware of the potential for a serious drug-drug
mean daily drug cost by dividing the patient’s total interaction. We calculated the percentage of drug in-
prescription cost by the number of days that the pa- teractions as the number of clinically relevant inter-
tient was followed in GMC, as defined by an algo- actions generated by each physician divided by the
rithm based on appointment and prescription data. total number of prescriptions written by each physi-
We obtained patient-visit information, prescription cian.
data, and drug costs from the DHCP.
To determine users’ reactions to the PWS and DHCP
We determined the influence of recommendations for computer systems, we designed a questionnaire that
cost-effective drug substitutions by measuring com- evaluated user satisfaction with each computer sys-
pliance with the recommendations. We defined com- tem, usage patterns, and computer-based and other
pliance as the proportion of prescriptions in which a sources of information retrieval. At the beginning of
provider accepted the recommended drug substitu- the study, both groups answered the questionnaire
tion. We calculated this proportion as the number of concerning the DHCP computer system. At the end
recommended drug prescriptions accepted by the pro- of the study, we administered the same questionnaire,
vider divided by the total number of prescriptions for but the DHCP group replied about the DHCP, and the
both recommended and substitutable drugs. PWS group replied about the PWS. We used a series
344 ROTMAN ET AL., Physician Workstation Evaluation
Table 2 l
User-Satisfaction Rating* ‘.
Preintervention Intervention
Phase Phase Difference
(mean t SD) (mean 2 SD) (mean % SEM) p Value
PWS group (N = 16) 3.44 k 0.533 2.98 z 0.547 ( -0.343 -t 0.112 p = 0.008t
DHCP group (N = 18) 3.23 k 0.400 3.72 z 0.333 0.488.~ 0.084 p < 0.0001t
Group difference comparison 0.831 1.0.138 p < 0.0001~
(PWS vs. DHCP)
*Values are based on responses to a five-point Likert scale, with 1 = strongly disagree and 5 =. strongly agree.
tPaired t test (two-tailed) for difference between pre- and post-satisfaction ratings for PWS and DHCI?
*Two-sample t test (two-tailed) for difference between PWS group and DHCP group (PWS group difference minus DHCP group
difference).
346 ROTMAN ET AL., Physician Workstation Evaluation
viders used the PWS system for writing only a small better suited to the menu-based data transfers of the
fraction (2.8%) of their prescriptions, so it is not sur- DHCP (in which small amounts of patient data were
prising that the number of drug interactions and the transmitted at a time) than to the PWS model (in
prescription costs were not affected. The implemen- which a patient’s entire record was transmitted ini-
tation of the intervention at our institution precluded tially). Users’ perceptions of the PWS’s lack of speed
requiring the intervention subjects to use only the may have been related to this initial delay, when the
PWS. Thus, the low level of usage may reflect the dif- entire record was transmitted. In addition, the syn-
ficulty users encountered in using the PWS, or it may chronization between the DHCP’s Massachusetts
reflect our clinicians’ satisfaction with the comprehen- General Hospital Utility Multi-Programming System
sive computer-based and paper-based systems that (MUMPS)-based database and the PWs’s object-ori-
were already available in our outpatient setting. The ented database was difficult to maintain. The nightly
DHCP gained many new features during the one-year uploads of new patient data were disrupted by ma-
study; these changes probably explain the increased chine or network downtime, and frequent mapping
level of satisfaction in our control group. The low changes between databases required a manual resto-
PWS usage limits our ability to draw conclusions ration of the data bridge, which also led to interrup-
about the usefulness of its novel features, such as a tions in PWS availability.
graphical interface and decision support.
Third, evaluators should investigate what works and
How can researchers design evaluative studies that does not work with the current clinical workflow. A
maximize the chance of observing changes in the rel- major component in our intervention involved mod-
evant outcomes yet also provide useful information if ifying the prescription-writing workflow in our clinic.
the intervention fails? The first task requires identi- We assumed that the potential reduction in drug in-
fying and reducing potential impediments in evalua- teractions and drug costs would be sufficient moti-
tions and selecting an appropriate design. The second vation for users to adapt to a new system. However,
task requires incorporating a “just-in-case” mentality we underestimated the value of our current paper-
in experimental design to ensure that information ca- based system. We did not assess users’ perceptions of
pable of explaining a negative study is collected. Our problems in their work environment and, thus, may
experience illustrates crucial areas that require consid- not have been providing the most beneficial technol-
eration. ogy for the users.
First, investigators and developers should specify the Fourth, investigators should provide sufficient train-
performance characteristics and support for both the ing time for users, Our users were in the clinic one
experimental and control systems required for the afternoon per week and had about a fourth of the
evaluation. These specifications should include pa- clinics canceled because of vacations and rotations to
rameters that assess completeness of data, speed of other hospitals. We provided training over a two-
operation, reliability, and accessibility of the system. month period before our intervention. This may not
This step will help reduce problems related to the new have been enough exposure to the PWS, especially
information system itself. The PWS project was an in- because users were already familiar with the DHCP
dependent study that was not part of the Palo Alto and continued to use it.
VA’s overall information-management strategy. The
DHCP was upgraded frequently during the study pe- Finally, investigators should choose an experimental
riod, and these upgrades often changed the DHCP’s design that maximizes statistical power to detect
underlying schema, which led to further development differences in the relevant outcomes. We selected a
requirements and to significant downtime for the two-period parallel design that improved our experi-
PWS. mental power compared with a traditional random-
ized controlled trial.” Depending on the variances of
Second, investigators and developers should assess the outcome variable, the two-period parallel design
the reliability and speed of the computing infrastruc- may reduce the number of study subjects required.”
ture. The PWS system used terminals connected to a Randomization should account for all providers, in-
server over a local-area network (LAN). The server cluding attendings, if possible. Other experimental
was connected to the DHCP. Thus, an interruption in designs, such as the crossover design:’ may further
DHCP function, a problem with the LAN, or a prob- enhance statistical power but may introduce sources
lem with the PWS itself could potentially lead to the of bias in interpreting results.
PWS being unavailable for the user. In a client-server
architecture, network stability is of paramount impor- In addition to these steps, we recommend that inves-
tance. The network infrastructure in our clinic was tigators design evaluative studies to be informative,
Journal of the American Medical Informatics Association Volume 3 Number 5 Sep / Oct 1996 347
even if a study does not find positive effects on health vice for assistance with prescription analysis; Merlynn Bergen
or economic outcomes. A negative study is useful if for assistance with questionnaire design and analysis; Andria
Cardinalli for help with preparation of the manuscript and data
the investigators can determine why their system analysis; Lyn Dupre for comments on the manuscript; and the
failed to affect the outcomes of interest and how to reviewers for their helpful suggestions.
correct the problem. We suggest the following guide-
lines, with examples from our study First, the design
of the study should enable the investigators to distin- References n
guish between an intervention that was not used and
one that was used but was ineffective. For example, 1. Tape TG, Campbell JR. Computerized medical records and
we documented that the number of prescriptions writ- preventive health care: success depends on many factors.
ten with the PWS was too low to have a substantial Am J Med. 1993;94:619-25.
2. McDonald CJ, Hui SL, Tierney WM. Effects of computer
influence on costs. Had the system been used fre- reminders for influenza vaccination on morbidity during
quently and costs were still unaffected, an optimal de- influenza epidemics. MD Comput. 1992;9:304-12.
sign would have enabled the investigator to deter- 3. Tierney WM, Miller ME, Overhage JM, McDonald CJ. Phy-
mine which aspects of the intervention required sician inpatient order writing on microcomputer worksta-
improvement. For example, were recommended drug tions. JAMA. 1993;269:379-83.
4 Tierney WM, Miller ME, McDonald CJ. The effect on test
substitutions considered inappropriate? Did the pro- ordering of informing physicians of the charges for outpa-
viders notice the recommendations? tient diagnostic tests. N Engl J Med. 1990;322:1499-1504.
5. Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects
The study should link user feedback to objective of computer-based clinical decision support systems on
measurements of system performance. We suggest in- clinical performance and patient outcome: a critical ap-
corporating objective measurements into the study praisal of research. AM Intern Med. 1994;120:135-42.
that help corroborate and explain user feedback. For 6. Safran C, Rind DM, Davis RM, et al. An electronic medical
example, our subjects commented on the system’s record that helps care for patients with HIV infection. In:
Safran C (ed). Proceedings of the Seventeenth Annual Sym-
“slow speed” and “how it needs some reports it posium on Computer Applications in Medical Care, Wash-
didn’t have.” Interpretation of our results would have ington, DC. New York, NY: McGraw-Hill, 1994;224-8.
been aided by a systematic quantification of perfor- 7. Tierney WM, Overhage JM, McDonald CJ. A plea for con-
mance parameters, which could have complemented trolled trials in medical informatics. J Am Med Inform As-
user perceptions. These parameters include compre- soc. 1994;1:353-5.
8. Massaro TA. Introducing physician order entry at a major
hensive measurements of system availability; of the academic medial center: I. Impact on organizational culture
time from initiation of a command until completion; and behavior. Acad Med. 1993;68:20-5.
and of user interaction with the system, including a 9. Wyatt J, Spiegelhalter D. Field trials of medical decision-
record of requests that could not be satisfied. In ad- aids: potential problems and solutions. In: Clayton PD (ed).
dition, our study would have benefited from a more Proceedings of the Fifteenth Annual Symposium on Com-
puter Applications in Medical Care, Washington, DC. New
systematic approach for assessing subjective re- York, NY: McGraw-Hill, 1992;3-7.
sponses, including open-ended questions in our sur- 10. Annevelink J, Young CY, Tang PC. Heterogeneous database
vey instrument. Finally, the study should enable in- integration in a physician workstation. In: Clayton PD (ed).
vestigators to prioritize system improvements. Proceedings of the Fifteenth Annual Symposium on Com-
puter Applications in Medical Care, Washington, DC. New
As several observers have suggested,5,7 clinical infor- York, NY: McGraw-Hill, 1992;368-72.
mation systems should be evaluated with the same 11. Pham TQ, Young CY, Tang PC, Suermondt HJ, Annevelink
rigor applied to other medical interventions. The ran- J. Implementing a Physician’s Workstation using client/
server technology and the distributed computing environ-
domized controlled trial is the gold standard of eval- ment. In: Ozbolt JG (ed). Proceedings of the Eighteenth An-
uation. We agree that controlled trials will provide the nual Symposium on Computer Applications in Medical
most rigorous evaluation of clinical information sys- Care, Washington, DC. Philadelphia, PA: Hanley & Belfus,
tems, and that more are needed.7 However, evaluation 1995;626-30.
of a clinical information system in an outpatient set- 12. Suermondt HJ, Tang PC, Strong PC, Young CY, Annevelink
J. Automated identification of relevant patient information
ting is a time-consuming, labor-intensive project that in a physician’s workstation. In: Safran C (ed). Proceedings
requires careful attention to workflow and personnel of the Seventeenth Annual Symposium on Computer Ap-
as well as to problems of system performance, secu- plications in Medical Care, Washington, DC. New York, NY:
rity, and functionality. McGraw-Hill, 1994;229-32.
13. Tang PC, Annevelink J, Fafchamps D, Stanton WM, Young
CY. Physician’s Workstations: integrated information man-
The authors thank Jeff Shyshka, Dean Robinson, Minerva Sims, agement for clinicians: In: Clayton PD (ed). Proceedings of
and the other staff at the VA Information Resources Manage- the Fifteenth Annual Symposium on Computer Applica-
ment Service for help with computer support. We thank Robert tions in Medical Care, Washington, DC. New York, NY: Mc-
Coleman and the other staff at the Palo Alto VA pharmacy ser- Graw-Hill, 1992;569-73.
348 ROTMAN ET AL., Physician Workstation Evaluation
14. Tang PC, Annevelink J, Suermondt HJ, Young CY. Semantic tions for design. In: Bullinger H. Human Aspects in Com-
integration of information in a physician’s workstation. Int puting: Design and Use of Interactive Systems and Work
J Biomed Comput. 1994;35:47-60. with Terminals. New York: Elsevier Science Publishers, B.V.;
15. Young CY, Tang PC, Annevelink J. An open systems archi- 1991;709-15.
18. Rotman BL, Sullivan AN, McDonald T, et al. A randomized
tecture for development of a physician’s workstation. In:
evaluation of a computer-based physician’s workstation:
Clayton P (ed). Proceedings of the Fifteenth Annual Sym-
design considerations and baseline results. In: Gardner RM
posium on Computer Applications in Medical Care, Wash-
(ed). Proceedings of the Nineteenth Annual Symposium on
ington, DC. New York, NY: McGraw-Hill, 1992;491-5. Computer Applications in Medical Care, New Orleans, LA.
16. Stanton WM, Tang PC. Knowledge-based support for a New York, NY: McGraw-Hill, 1996;693-7.
Physician’s Workstation. In: Clayton I’ (ed). Proceedings of 19. Rotman BL, McDonald TW, Suermondt HJ, et al. Computer-
the Fifteenth Annual Symposium on Computer Applica- based detection of drug interactions in an ambulatory care
tions in Medical Care, Washington, DC. New York, NY: Mc- practice. Med Decis Making. 1994;14:440.
Graw-Hill, 1992;649-53. 20. Brown B. The crossover experiment for clinical trials. Bio-
17. Fafchamps D. Ethnographic workflow analysis: specifica- metrics. 1980;36:69-79.