A visual representation of microbiological culture data improves comprehension: a randomized controlled trial

Background

Half of all patients receive antibiotics during their hospital stay.¹ The timely use of appropriate antibiotics may improve mortality.^2,3 However, the selection of a suitable antimicrobial agent can be challenging and may be further complicated by antibiotic resistance. There are more than 2 million people with resistant infections⁴ resulting in more than 150 000 estimated deaths⁵ each year in the US, while inadequate coverage may be achieved in 22.4% to 29.8% of patients.^6–9

Antibiotic treatment becomes more nuanced in those with severe infections,¹⁰ and, even in the outpatient setting, antibiotic selection requires increasing levels of sophistication as the modern microbial milieu demonstrates complex patterns of resistance.^11,12

Importance

Antimicrobial stewardship programs facilitate the judicious use of these medications and have been a mainstay in minimizing misuse as relatively few novel therapeutic agents are in development,¹³ and tools such as molecular diagnostics are not yet practical for widespread clinical use.¹⁴ One avenue towards facilitating antibiotic prescribing may be to improve clinicians’ understanding of a patient’s microbiological culture history.

Past antibiotic sensitivities should inform prescribing; however, reviewing all prior resistance data and then synthesizing these into clinical decision-making can be cumbersome. Additionally, electronic health records (EHRs) have been noted to be lacking in functionality to assist in antimicrobial stewardship goals.¹⁵ Given this difficulty in gathering information and time constraints in clinical care, prescribing practices may fall short of optimal behavior, particularly when patients’ clinical situations are complex. While EHRs can increase availability to large amounts of data, the data can be overwhelming¹⁶ and lead to idiosyncratic workarounds with resultant patient safety concerns.^17,18

Visual representations of data may assist clinicians in synthesizing and understanding voluminous medical data.^19,20 Data related to infectious diseases,²¹ specifically culture sensitivity data,^22–24 may be particularly amenable to this approach. In contrast to discrete laboratory data such as a potassium level, microbiological data is inherently multidimensional: a sample from a specific location may reveal multiple species of bacteria with varying degrees of resistance to an array of antibiotics. Furthermore, these results will vary over time and must be interpreted accordingly. Many EHR representations do not fully convey the depth of microbiological data, as they typically comprise of time-sorted lists of individual culture samples requiring “drilling down” to find the details of the sensitivities, and offer little assistance in summarizing or comparing the results across multiple samples and time.

Study design and setting

We performed a single-center parallel unblinded randomized controlled trial to assess the effect of a culture result visualization tool on provider knowledge of previous culture sensitivities. The study hospital emergency department (ED) has approximately 55 000 visits per year and is a tertiary care adult-only academic hospital and Level I trauma center. The EHR in use was custom-developed at the institution.

Simple randomization was performed using a computerized random number generator prepared by the investigators with a 1:1 allocation ratio at the level of the patient visit. Randomizations were assigned through REDCap and the research assistants carried out the assigned intervention. If a patient visit was randomized to the intervention group, the provider who had ordered antibiotics for that patient would be shown a culture visualization tool by a research assistant on a laptop prior to answering questions regarding past sensitivity results. Cases in the control group did not have access to the visualization tool. Both groups had access to the standard EHR while responding to research assistants. Clinicians were presented with 4 antibiotics and asked whether any previous cultures demonstrated resistance or were sensitive to them in the past. Providers may have been approached by research assistants at any time after ordering antibiotics until either the antibiotics were administered or the patient left the department.

Several months before the study, users were given a several minute introduction to the study and later received a brief follow-up e-mail regarding use of the visualization. There was no further training regarding use of the tool.

Selection of participants

Antibiotic orders placed in the ED triggered a page to research assistants who then approached the ordering provider for eligibility screening and trial enrollment. Research assistants were available 8AM–11PM on weekdays and 11AM–8PM on weekends.

Patients met inclusion criteria if they had an order placed for an oral or parenteral antibiotic and also had a previous urine or blood culture with sensitivity results available in the EHR.

Patients were excluded if they were under the age of 18, they had no past urine or blood culture sensitivity results, they had the antibiotic orders cancelled at the time of screening, the antibiotic was administered prior to screening, the provider was unable to be reached before the visit concluded, or participation in the study would impede clinical care. In addition to the prespecified exclusion criteria, to minimize bias, cases were also excluded if the antibiotic in question was not being prescribed as the initial treatment of an acute infection (eg, prophylaxis and ongoing antibiotic courses were excluded) or if the participant was a study investigator.

Intervention

The software to visualize microbiological culture results was developed in an iterative fashion and refined in conjunction with input from informaticists, infectious disease physicians, infectious disease pharmacists, emergency physicians, internists, and surgeons. The tool was built directly into the production version of the EHR and written in Caché (InterSystems Corporation, Cambridge, MA), jQuery, HTML, and JavaScript by the authors (EYK, LAN, and SH).

Rows of the display are ordered by time, with a separate row for each organism that speciated from the past cultures (Figure 1). The remainder of the row is composed of the color-coded sensitivity results (eg, “sensitive,” “resistant”), with the corresponding antibiotic that the culture was tested against listed above the rows as column headers. Functionality to sort and filter based upon the source of the culture, date, and organism was provided. Please see the Supplemental Materials for a sample case and source code.

Methods of measurement

Study data were collected and managed using REDCap electronic data capture tools hosted at our institution.²⁵ REDCap (Research Electronic Data Capture) is a secure, web-based application designed to support data capture for research studies, providing 1) an intuitive interface for validated data entry; 2) audit trails for tracking data manipulation and export procedures; 3) automated export procedures for seamless data downloads to common statistical packages; and 4) procedures for importing data from external sources.

Providers were asked questions by research assistants with responses recorded directly onto iPads using REDCap. If a provider was randomly selected to having the tool available for use, it was provided on a laptop computer by the research assistant. Research assistants were not trained in use of the tool and use of the EHR by the provider was not restricted.

Providers’ knowledge of the patients’ prior culture results was assessed by a questionnaire. The physician was presented with 4 antibiotics and asked if the patient previously had infections resistant to each of the antibiotics at the home institution. The providers could respond that past resistances, including intermediate resistance, were noted to this antibiotic (“resistant”), no past resistances were noted (“sensitive”), or “unsure.”

Both the intervention and control group were also asked to subjectively assess their own knowledge of previous cultures and their confidence in their decision-making. These responses were obtained using a 5-point Likert scale (1 strongly disagree, 5 strongly agree) with the option to respond “unsure” to each question. Providers in the intervention group were also asked about the usability of the tool.

Patient charts were reviewed by the authors to determine the suspected source of infection as per the ED providers, review antibiotic order details, and assess culture results.

Outcome measures

The primary outcome was provider knowledge of patients’ past culture sensitivities. A response of “sensitive” was interpreted as correct when past cultures in the medical record all demonstrated in vitro sensitivity to the antibiotic in question. A response of “resistant” was interpreted as correct if any previous culture data showed organisms with either resistance or intermediate resistance to the antibiotic.

Prespecified secondary outcomes assessed the effectiveness of the ordered antibiotics by the provider. Adequacy of antibiotic choice was determined by comparing the antibiotics prescribed in the ED to cultures drawn during that same ED visit. Antibiotic regimens were judged to be adequate if all cultured isolates from that visit demonstrated in vitro sensitivity to the antibiotics prescribed. Prescribed antibiotics were deemed inadequate if there was demonstrated resistance or intermediate resistance to the antibiotics without otherwise adequate antimicrobial coverage. Patient encounters without cultures drawn or cultures without growth were excluded from this portion of the analysis as were cultures where the antibiotic in question, or another anitbiotic of the same class, was not tested. Charts were reviewed to ensure that the isolates being evaluated reflected the suspected infectious processes of primary concern and were not disregarded (eg, contaminants or isolates incidentally found that were not being considered as an infectious source). Other attributes of antibiotic ordering were not considered (eg, dose, appropriateness of empiric antibiotic regimens).

Additionally, previous resistance demonstrated in the medical record to antibiotics given during the present ED visit was tabulated in 3 time frames: cultures taken from the present visit, cultures from 2 years prior up until and excluding the present visit, and all available cultures preceding the present visit. In this analysis a prescribed antibiotic was noted to be resistant if any isolate within the noted timeframe had in vitro resistance or intermediate resistance to that antibiotic or 1 of the same class. No results were excluded in this analysis.

Cases where antibiotic orders were either cancelled or added were assessed, and survey questions regarding usability of the tool and the providers’ decision-making used 5-point Likert scales.

Sample size calculation

Using a prior randomized controlled trial which also assessed provider knowledge after an intervention, we assumed a conservative standard deviation of 1 for these calculations and a difference of 0.45 out of 4 questions being answered correctly.²⁶ With the above assumptions, a sample size of 105 patients per arm of the study was calculated to provide 90% power at ɑ=0.05.

Primary data analysis

Data were analyzed using Stata 14.2 (State College, TX). Chi-square and Fisher’s exact tests were used in the analysis of categorical variables, while Wilcoxon ranked sum tests were used with continuous data. Likert data were analyzed categorically using Fisher’s exact test when comparing the control group to the intervention group.

Provider knowledge of past sensitivities was evaluated using Cohen’s kappa, where providers’ responses were assessed for agreement with the culture data in the EHR. Provider responses of “unsure” were weighted as half-correct using the weight “w” in Stata. Kappas were considered to be significantly different if the 95% confidence interval in the intervention group excluded the values in the control group. Analysis of agreement was also stratified by residents versus attendings and kappas were compared within and between groups. In addition to Cohen’s kappa, which included all provider responses, we also provided an area under the receiver operating characteristic curve (AUC), sensitivity, specificity, positive predictive value, and negative predictive value for all responses where the provider did not choose “unsure.”

Human subjects research statement

The trial protocol was registered on ClinicalTrials.gov (#NCT03580603) prior to study initiation and approved by the institution’s Committee on Clinical Investigation (#2017P-000432).

Characteristics of study subjects

From 3/5/2018 to 9/30/2018, 518 ED patient visits were assessed by research assistants for potential eligibility (Figure 2). All 273 patient encounters were included in this intention to treat analysis, with 129 encounters randomized to the control group and 144 encounters randomized to the intervention group. Patient demographics are reported in Table 1. 20 (7.3%) encounters were with attending physicians as the ordering provider, the remaining physicians enrolled were resident physicians (see Supplementary Materials). 23 (8.4%) cases had no cultures drawn during the ED visit, and 166 (60.8%) visits had ED cultures with no growth.

Main results

Participants in the control group had agreement with previous culture results with a kappa of 0.16 (95% CI: 0.12–0.20), while those randomized to the intervention group had a kappa of 0.69 (95% CI: 0.65–0.73) (Table 2). The intervention group also had a higher AUC of 0.85 (95% CI: 0.83–0.87) compared to the control group with 0.59 (95% CI: 0.57–0.61) (Table 3). There were no significant differences in changes in antibiotic order or in vitro susceptibility to prescribed antibiotics between the groups (Table 4).

Providers that were randomly selected to use the tool felt it was easy to use and helpful in understanding patients’ culture data. When compared to those not using the tool, providers who used the tool reported that they had a better understanding of patients’ culture data (P < .001) and that they made a more informed choice when compared to providers who did not use the tool (P < .001) (Figure 3 and Supplementary Materials).

Limitations

The study was conducted within a single department of a single institution as a convenience sample, limited to patients with previous data in the EHR. Also, our trial protocol enrolled the ordering physician, which at our institution is usually a junior physician. This environment may not be representative of other practice scenarios. Though providers are expected to take into account past microbiological results, treatment decisions may at times be deferred to other clinicians or treatment teams whose decision-making would not be captured in this study. In other cases, empiric therapy may be guided by institutional guidelines, though best practices would entail checking of previous antibiotic resistances in these situations as well.

Many of the providers approached for the study were not available as the ED is a high-acuity setting and clinical responsibilities were given precedence over study enrollment. Ideally there would have been more extensive evaluation of providers, but this was precluded by the time-sensitive nature of ED care. Given our enrollment, we were not able to control for variation among providers or assess clinical outcomes, such as mortality.

There were no demonstrated differences in the secondary outcomes between the study groups, though the study was not powered or designed to assess these outcomes. Providers more accurately assessed antibiotic sensitivities in the intervention group, yet the benefit of this knowledge was not clearly reflected in antibiotic ordering. Difficulty in randomized trials within informatics to demonstrate clinical outcomes has been previously noted.³⁸ Furthermore, due to technical constraints we were unable to present the visualization prior to antibiotic ordering nor have the tool freely available, thus significantly altering provider workflow. This is inconsistent with several tenets of clinical decision support design,³⁹ likely making the intervention less useful to clinicians. For example, by the time of the intervention, providers have already chosen an antibiotic they are comfortable prescribing and may not elect to validate this decision using the tool. Also, antibiotic selection is a sophisticated medical decision that may be difficult to judge the adequacy of without expert opinion, but this was outside of the scope of this study. Additionally, deficiencies in antibiotic prescribing are a known problem in medicine and the tool does not address factors such as knowledge gaps, faulty decision-making, or inappropriate prescribing habits that may contribute to antibiotic selection.

Though patient-centered outcomes are of paramount importance, the effect of the EHR on the provider was also a motivation for the development of the tool and we would have ideally captured “provider-centered outcomes” such as burnout, medical error, and time spent reviewing culture data. Though not all of these factors directly affect patient care, they may do so indirectly and clinicians may place significant importance on them.