Objective: To systematically review studies reporting problems with information technology (IT) in health care and their effects on care delivery and patient outcomes.
Materials and methods: We searched bibliographic databases including Scopus, PubMed, and Science Citation Index Expanded from January 2004 to December 2015 for studies reporting problems with IT and their effects. A framework called the information value chain, which connects technology use to final outcome, was used to assess how IT problems affect user interaction, information receipt, decision-making, care processes, and patient outcomes. The review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.
Results: Of the 34 studies identified, the majority (n = 14, 41%) were analyses of incidents reported from 6 countries. There were 7 descriptive studies, 9 ethnographic studies, and 4 case reports. The types of IT problems were similar to those described in earlier classifications of safety problems associated with health IT. The frequency, scale, and severity of IT problems were not adequately captured within these studies. Use errors and poor user interfaces interfered with the receipt of information and led to errors of commission when making decisions. Clinical errors involving medications were well characterized. Issues with system functionality, including poor user interfaces and fragmented displays, delayed care delivery. Issues with system access, system configuration, and software updates also delayed care. In 18 studies (53%), IT problems were linked to patient harm and death. Near-miss events were reported in 10 studies (29%).
Discussion and conclusion: The research evidence describing problems with health IT remains largely qualitative, and many opportunities remain to systematically study and quantify risks and benefits with regard to patient safety. The information value chain, when used in conjunction with existing classifications for health IT safety problems, can enhance measurement and should facilitate identification of the most significant risks to patient safety.
Keywords: health information technology, patient safety, adverse events, systematic review, unintended consequences
INTRODUCTIONThe widespread adoption of information technology (IT) brings many potential benefits to health care.1 At the same time, problems with IT can disrupt the delivery of care and increase the likelihood of new, often unforeseen, errors that affect the safety and quality of clinical care and may lead to patient harm.2–7 Our capacity to reap the benefits of IT and manage new threats is contingent upon understanding the ways in which IT problems can disrupt care delivery and pose threats to patient safety.
While previous reviews have looked at the overall effect of IT on patient outcomes,8 less is known about the specific nature of IT problems in clinical settings. A recent review broadly examined 5 categories of sociotechnical factors affecting safety, including people, technology tasks, organization, and environment.9 In this study, we sought to summarize the research literature describing the different problems known to be associated with IT in health care and to more precisely analyze their effects on care delivery and patient outcomes.
METHODS Study identification and selectionHealth IT was broadly defined as computer hardware and software used by health professionals to support patient care. We focused on studies reporting problems with IT and its effects on care delivery and patient outcomes. These effects were examined using a new framework called the information value chain, which connects the use of a technology to final outcome (Figure 1).10,11 The chain is initiated when a user interacts with an IT system. A subset of these interactions will yield new information, only some of which then lead to changed decisions. Next, only some decisions will see changes in the care process, and only some process changes will impact patient outcome. Using this framework, we sought to identify the effects of IT problems on each stage of this chain, from user interaction to clinical outcome.
Fig. 1.The information value chain connects use of a technology to final outcome.10,11 We examined the effects of IT problems on user interaction and information received, as well as effects on decision-making, care process, and patient outcomes. An existing classification was used to categorize IT problems, information received, and contributing factors.12
We searched the bibliographic databases Scopus, PubMed, and Science Citation Index Expanded from January 2004 to December 2015. The search query used was (“health information technology” OR HIT OR “health IT” OR “electronic health record” OR “electronic medical record” OR “decision support” OR CPOE OR “technology induced” OR “computer related”) AND (“medication error*” OR error* OR incident* OR “unintended adverse consequence*” OR consequence* OR “incident report*” OR “patient outcome”) AND (data OR analysis OR qualitative OR quantitative). To be included in the review, studies needed to report problems with IT and their effects on 1 or more stages in the information value chain, ie, interaction, information received, decision-making, care process, or outcome. Study quality was assessed by examining study design, risk of bias, duration, population size, and reporting about successive stages in the information value chain. Only English-language studies were included.
The searches identified 3277 potential articles: Scopus, 1093; PubMed, 1197; and Science Citation Index Expanded, 987. In accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for reporting of systematic reviews, one reviewer first reviewed all titles and abstracts. Seventy-nine papers were selected for full review (Figure 2). Each study was assessed independently by 2 reviewers (MK and FM) against the inclusion criteria. All disagreements were resolved by consensus. After assessment, 34 studies remained.
Fig. 2.Study identification and selection
Data extraction and analysis of the effects of IT problemsFor each included study, both reviewers extracted information to develop an inventory of IT problems. Each identified problem was then labeled using an existing classification for safety problems associated with health IT to describe the nature of the problem.12 The problems were then assigned to the different stages of the value chain, depending on whether they affected user interaction, information receipt, decision-making, care processes, or outcomes. Some of the identified problems had the broad potential to affect events at multiple stages of the chain, and these were classed as either general technical issues, covering problems in the design of software and hardware, or sociotechnical contextual variables related to human or organizational issues that influenced user interaction (contributing factors).
The next stage of analysis sought to further characterize these groupings. Four types of errors in information (information errors) were considered: wrong, missing, partial, and delayed.13,14 These could arise from how software was used (use errors) or software and hardware issues (machine errors). Errors and delays in decision-making were similarly identified. We sought to identify omission errors (ie, when an intended action was not executed) and commission errors (ie, when an action was wrong).
Finally, observable impact on care process and outcomes was examined using a standard approach and categorized as13–16:
Potential or actual harm to a patient. An IT problem led to a clinical error that reached the patient,16 eg, a patient had severe allergic reaction to prescribed medication.
An arrested or interrupted sequence or a near miss. An IT problem led to a clinical error that was detected before reaching the patient,15,16 eg, a prescription in a wrong name was noticed and corrected while printing.
An IT problem with a noticeable consequence but no patient harm. A problem that affected care delivery but involved no harm to a patient, such as delays and rework, eg, a computer network problem resulted in delays or additional phone calls to follow up missing test results.
An IT problem with no noticeable consequence. A problem that did not directly affect the delivery of care, eg, an electronic backup copy of patient records was corrupted, but this was detected and the copy was not needed.
A hazardous event or circumstance. A problem that could potentially lead to an adverse event or a near miss, eg, prescribing software failed to display a patient’s allergy status.
A narrative synthesis then integrated findings into descriptive summaries for each stage of the value chain.
RESULTS Descriptive analysis of all studiesWe identified 34 studies describing the effects of IT problems on care delivery and patient outcomes (Table 1). The majority were analyses of incidents (n = 14, 41%; Table 2), which were reported at varying levels, from a single hospital to nationwide, in 6 countries: the United States, the United Kingdom, the Netherlands, China, Hong Kong, and Australia.12–15,17–26 Nine were ethnographic studies using interviews, surveys, and participant observation27–35 and 7 were descriptive studies using existing data such as prescriptions to examine medication errors.36–42 The remaining 4 studies were case reports.43–46 Of the 34 studies reviewed, more than half examined computerized provider order entry (CPOE) or prescribing systems (n = 19) and 10 (29%) examined all types of health IT systems. Three out of 4 studies were undertaken in inpatient settings (n = 26, 76%) and 15 in outpatient settings.
Table 1.Studies reporting the effects of health IT problems on care delivery and patient outcomes
Authors (year) Study period (no. of months) Method and design Country Sample (incidents) Health IT type Setting Key findings Aarts et al. (2004)43 1988–2000 (144) Case study; longitudinal study Netherlands CPOE Inpatient Hospitalwide implementation of CPOE was halted due to unintended consequences. Software problems including poor user interfaces led to difficulties in entering orders and delayed care processes. Ash et al. (2004)36 Survey and observ (qualitative) United States, Netherlands, Australia >340 h, >77 interviews All health IT Inpatient, outpatient Two categories of unintended effects were identified: errors in processes of entering and retrieving information, and communication and coordination processes. Specific consequences for care delivery included more work for clinicians and communication loss. Technical problems were related to inadequate software functionality such as poor user interfaces, fragmented displays, and inflexibility of system design. System configuration issues were related to problems with decision support including reminders, alerts, and system messages. Contributing factors included poor integration with workflow, cognitive load due to interruptions. Workarounds were used to deal with many IT issues. Ash et al. (2007)28 Survey (qualitative) United States 176 responses CPOE Inpatient, outpatient Representatives from 176 hospitals were surveyed about the extent and importance of CPOE-related unintended consequences. The survey was based on 9 major categories of unintended consequences of CPOE implementation identified by Campbell et al. (2006). All hospitals reported 8 categories of unintended consequences (ie, except category 4, problems related to paper persistence). Seventy-two percent of respondents ranked unintended consequences as moderately to very important, including these categories: (1) more/new work issues, (2) workflow issues, (3) never-ending system demands, (5) communication issues, (6) intense emotions, and (9) overdependence on technology. Campbell et al. (2006)37 2004–2005 (9) Observ (qualitative) United States 324 CPOE Inpatient Unintended consequences associated with CPOE implementation were investigated. Technical problems were related to software functionality, which confused users. Clinical care delivery became dependent upon CPOE technology. System failure and malfunctions delayed patient care and required use of hybrid records systems. Overall, 9 major categories of unintended consequences were identified: (1) more/new work for a clinician, (2) unfavorable workflow issues, (3) never-ending system demands, (4) problems related to paper persistence, (5) untoward changes in communication pattern and practices, (6) negative emotions, (7) generation of new kinds of errors, (8) unexpected changes in the power structure, and (9) overdependence on technology. Cheung et al. (2013)20 2010–2011 (12) Incident analysis (qualitative) Netherlands 668 CPOE, EHR, AutoDisp, PIS, infusion pump, eMAR Inpatient, outpatient pharmacy Half of the incidents were associated with use errors relating to wrong data entry. Technical problems were related to poor design of user interfaces, which led to medication errors in community pharmacies. A total of 360 incidents reached patients. Two deaths and 20 cases of serious but temporary harm were reported. Han et al. (2005)42 2001–2003 (18) Comparative (quantitative) United States 1,942 patients CPOE Inpatient CPOE implementation in a children's hospital and accompanying policy changes were associated with increased mortality from 2.8% to 6.6%. A poor user interface that was not adapted to local requirements led to delays in initiating treatment. The CPOE did not allow entry of orders prior to arrival of critically ill patients, delaying life-saving treatment. New workflow also caused a breakdown in doctor-nurse communication. Hospitalwide implementation over a 6-day period did not allow staff enough time to adapt to new routines and responsibilities. In parallel with CPOE implementation, changes to policies and procedures for dispensing and administering medications exacerbated treatment delays. For instance, all medications including ICU vasoactive drugs were relocated to a central pharmacy. Hanuscak et al. (2009)32 2007 (3) Survey (qualitative) United States 32 responses All health IT Thirty-two individuals who were directly responsible for supporting and maintaining IT systems in 78 hospitals were surveyed to evaluate causes and frequency of medication errors due to downtime over a 12-month period. Technical problems were related to hardware and software issues, including malfunctioning systems, interfaces with other software components, and updates. Downtimes were also linked to use errors. Of the 39 medication errors linked to downtime, 14 reached patients. Horsky et al. (2005)44 Case study United States CPOE Inpatient An elderly patient suffering from hypokalemia (low potassium; serum potassium was 3.1 mEq/L, creatinine 1.7) became severely hyperkalemic (serum potassium 7.8 mEq/L). Wrong, incomplete, and missing information in the hospital order entry system resulted in the patient receiving multiple doses of potassium. In total, 316 mEq potassium chloride was administered over 42 hours. Technical problems were related to software functionality, such as suboptimal screen display and lack of automated checking function. Human factors issues were linked with inadequate training and poor familiarity with the system. Koppel et al. (2005)27 2002–2004 (24) Survey and observ (qualitative) United States 261 responses, 32 interviews CPOE Inpatient Twenty-two CPOE-related mechanisms for medication errors were identified and broadly categorized by information errors and human-machine interface flaws. Technical problems involved software functionality and system configuration. Fragmented displays disrupted user interaction and led to errors in selecting medications. Poor displays delayed time to complete clinical tasks. Problems with process for reapproval of antibiotics led to gaps in therapy. CPOE downtime also contributed to delays in care process. Hybrid record systems were used to deliver care during downtimes. Landman et al. (2013)46 Case study (qualitative) United States EHR Inpatient An update to Web browser software severed the link between an ED tracking board and a Web-based image viewer. Loss of this link resulted in decreased image viewer access rates for ED patients during the 10 days of the incident (2.8 views/study) compared with image review rates for a similar 10-day period preceding this event (3.8 views/study, P < .001) Lei et al. (2013)25 2001–2012 (142) Incident analysis (qualitative) China 116 All health IT Inpatient, outpatient, general practice In total, 66% of downtime events were associated with technical problems, such as hardware and software malfunctions; 36% of events affected more than 100 individuals, and of these, 9 events affected over 1000 individuals. In all, 109 events impacted care delivery. In 21 events, patients were forced to seek care in other hospitals. One death was reportedly linked to downtime. Magrabi et al. (2010)13 2003–2005 (24) Incident analysis (qualitative) Australia 99 All health IT Inpatient, general practice There were 99 incidents describing 117 problems. Overall, 55% of problems were machine-related and 45% were attributed to human-computer interaction. Delays in initiating and completing clinical tasks were a major consequence of machine-related problems (70%), whereas rework was a major consequence of human-computer interaction problems (78%). While 38% (n = 26) of the incidents were reported to have a noticeable consequence but no harm, 34% (n = 23) had no noticeable consequence. Magrabi et al. (2012), (2011)14,50 2008–2010 (30) Analyses of reports about equipment failure and hazards (qualitative) United States 436 All health IT InpatientA total of 436 incidents were associated with 712 problems; 96% of problems were machine related, and the remaining were problems with the human-computer interface. In all, 11% of incidents were associated with patient harm, and 4 deaths were linked to health IT problems:
(1) Entry of a portable X-ray image into a PACS system under the wrong name resulted in a wrong diagnosis and subsequent intubation, which may have contributed to death.
(2) A technician mistakenly entered the date of birth of a baby instead of the study date, making a chest X-ray appear older than it was. A radiologist subsequently viewed the image for peripherally inserted central catheter (PICC line) placement. Seeing that the comparison image did not have the line present, the radiologist concluded that it had been removed. Unfortunately, the line was placed too far in the infant, and the premature baby died.
(3) Orders were not executed and went undetected due to inadequate separation of preoperative orders from postoperative, resulting in a “missed opportunity” to diagnose and treat a life-threatening disease, contributing to death.
(4) A CPOE user interface that did not provide medication doses in milligrams was associated with administration of 3 times the maximum dose of an analgesic drug in 24 hours. This resulted in acute renal failure and death.
Of the 850 events analyzed, 68% (n = 574) described potentially hazardous circumstances, 24% (n = 205) had an observable impact on care delivery, 4% (n = 36) were a near miss, and 3% (n = 22) were associated with patient harm, including 3 deaths (0.35%). Eleven events did not have a noticeable consequence (1%) and 2 were complaints ( < 1%). Among the events, 1606 separate contributing problems were identified. Of these, 92% were predominantly associated with technical rather than human factors.
There were 3 deaths:
(1) A patient who was seen with another patient’s records in general practice was prescribed that patient’s medication and died later the same day from taking it. No further details were available.
(2) A patient suffering from chest pain advised the receptionist in a GP surgery. The receptionist intended to alert the GP about this patient via the practice software, but sent the message to herself instead. The patient later died from a myocardial infarction.
(3) An HIV test ordered during a hospital stay was not followed up after discharge. When the patient was readmitted, the admitting doctors were unable to access the HIV test result, because the test request was hidden from them. The patient developed pneumocystis pneumonia and died.
Characteristics of included studies and reported effects of health IT problems by the stages in the information value chain
Characteristic/Effects of IT problems Studies, n (%) Study design and methods Qualitative 27 (79) Incident analysis 14 (41) Case study 4 (12) Surveys, interviews, participant observation 9 (26) Quantitative, observational studies 7 (21) Descriptive 4 (12) Comparative 3 (9) Setting Inpatient 26 (76) Outpatient 15 (44) Long-term care 4 (12) General practice 6 (18) Country United States 21 (62) Australia 5 (15) United Kingdom 3 (9) Other 5 (15) Type of IT system All health IT 10 (29) CPOE 17 (50) Electronic health record 7 (21) Ambulatory prescribing 2 (6) Dispensing 2 (6) IT problemsa 34 (100) 2.1 Hardware (device) down or slow12–15,20,24–26,32,33 10 (29) 2.2 Data capture/output peripheral device down or slow12–15,24 5 (15) 2.3 Network/server down or slow12–15,20,24,25,33 8 (24) 2.4 Software not available or not licenced12,13,15 3 (9) 2.5 Software not accessible12,13,17,22,24,27,28,33,34,37,42,46 12 (35) 2.6 Software issue13,18,25,26,32 6 (18) 2.6.1 Functionality (including user interface and task fit)12,14,15,17–20,22,23,26–28,30,31,33–44 26 (76) 2.6.2 System configuration (including decision support rules)12,14,15,17,18,20,22,24,26,27,30,31,34,36,38,40,42,44 18 (53) 2.6.3 Interface with devices12,14,15 4 (12) 2.6.4 Interface with other software systems or components12,15,17,20,21,24,26,28,32,37 10 (29) 2.6.5 Increased volume of transaction12,22 2 (6) 2.7 Data storage and backup12–14,24,27 6 (18) 2.8 Record migration12,15,22,23 4 (12) 2.9 Power failure12,25,32,33 4 (12) 2.10 Computer virus12,15,25,32 4 (12) Contributing factorsa 24 (71) 3.1 Staffing/training12,13,15,17–19,21,23,26–29,32,34,35,37,40,43,44 19 (56) 3.2 Cognitive load15,17,27,28,35–37 7 (21) 3.2.1 Interruption12,15,35 3 (9) 3.2.2 Multitasking13,15,35 3 (9) 3.3 Failed to carry out duty15,18,19,21 4 (12) 3.3.1 Failed to log off12,13,27,32 4 (12) 3.4 Information governance12,17,23,26,27,32,34–36,43 10 (29) 3.5 Integration with clinical workflow12,18,26–30,34–37,42–45 15 (44) Information receiveda 33 (97) Use error12–15,17–41,44,45 31 (91) 1.1.1 Wrong12–15,17–24,27–31,34–37,39–41,44,45,47 26 (76) 1.1.2 Partial12–15,18,19,21,22,24,27,29,34,39–41 15 (44) 1.1.3 Missing12–15,18,20,21,24,27,34,40,41 12 (35) 1.1.4 Delayed15 1 (3) Machine error12–15,17,18,20–24,27,28,30,34–38,42–44 22 (65) 1.2.1 Wrong12,15,18,20,22,24,27,28,30,34–38,42,43 16 (47) 1.2.2 Partial12,15,21,34 4 (12) 1.2.3 Missing12,14,15,17,20–22,24,34,38,44 11 (32) 1.2.4 Delayed12,15,27,35 4 (12) Decision changed 27 (79) Omission error12–15,18–21,24,27,30,34–36,38,40,41,45 18 (53) Commission error12–15,17–22,24,27–31,34–41,44,45 26 (76) Delay12,15,35 3 (9) Care process altered 25 (74) Clinical error12–15,17–22,24,25,29,31,32,34,38–41,44 15 (44) Delay12,13,15,22,25,27,29,31,35,39,42–46 15 (44) Outcome changed 21 (62) Near miss12,13,15,17–19,32,38,41,45 10 (29) Actual or potential patient harm12–15,17–20,22,24,25,29,32,38–41,44 18 (53) User interactionThe first stage in the value chain was associated with multiple IT problems, typically technical and sociotechnical or context issues, and these also had the potential to affect multiple stages of the chain. Issues with accessing software were reported in 35% of the studies (eg, software was not available at a particular workstation, was not accessible, or did not have the correct settings). Other commonly reported IT problems were related to interfaces with other software (29%, n = 10), hardware malfunction (29%, n = 10), and network issues (24%, n = 8). Issues with software functionality appeared consistently across a majority of studies (Table 2). In 76% of studies, poor user interfaces and fragmented displays (eg, preventing a coherent view of all of a patient's medications) were associated with errors in selecting and entering information. Other software issues were related to system configuration, especially problems with default settings, which were reported in more than half of the studies (53%, n = 18). System configuration issues were also linked to software updates, eg, decision support errors following updates to a drug database.26
Sociotechnical contextual variables that contributed to information errors were identified in 71% of studies (n = 24). The most commonly reported problem was staffing and training to use IT systems (56%, n = 19). Other contributing factors included integration with clinical workflow (44%) and information governance (29%), eg, procedures to authorize medications27,36 and IT policy.23,34
Information receivedInformation errors arising from the use of software were reported in most studies (91%, n = 31). For example, autopopulated fields in a prescribing system contained incorrect information such as drug dosing directions.31 Problems in data entry and retrieval were linked to wrong (76%, n = 26), partial (44%, n = 15), missing (35%, n = 12), and delayed (3%, n = 1) information. Five studies did not identify specific types of use errors, describing them as keypad or computer entry errors.25,26,32,33,38 In contrast, machine errors were reported in 65% of studies (n = 22). These were due to wrong (47%, n = 16), missing (32%, n = 11), partial (12%, n = 4), and delayed (9%, n = 3) display of information. For example, alerts about drug-drug and drug-allergy interactions failed to display (missing information) or were wrongly displayed (wrong information). As with data entry, some studies did not identify specific types of machine errors, describing them as display or data output errors (n = 2).13,23
Decision changedThe effects of IT problems on errors and delays in clinical decision-making were reported in 76% of studies (n = 26). For example, errors in predefined order sentences led to clinical errors such as wrong dose and wrong route.41 Commission errors were the most commonly reported (76%, n = 26). These were linked to wrong data entry, selection from dropdown menus, and file uploads. In contrast, omission errors were reported in 50% of studies (n = 17) and delays in only 9% (n = 3). For example, users ignored alerts (n = 6) and failed to update information.13,14 Many delays in clinical decision-making were linked to computer network issues.12,15,35
Care process alteredThe effects of IT problems on clinical errors and delays in care process could be identified in 44% of studies (n = 15). Examples of such errors include medication administration errors and failure to follow up test results. Delays in care process were linked to system access24,25 and software functionality issues, including poor user interfaces and fragmented displays.27 Issues with system configuration,43 particularly software updates, were also reported to impact care delivery.46 In 1 study, poor integration of an electronic health record with clinical workflow disrupted care delivery in a long-term care facility.33 Other effects on care delivery included cancellation of patient appointments24,25 as well as unnecessary or emergency clinical procedures and treatment.22 IT issues were reported to create more work for health professionals (21%, n = 7). For example, pharmacists needed to telephone clinicians to clarify IT-related errors and discrepancies in prescriptions.27,29,39 IT problems also wasted time and caused frustration. In 1 study, primary care doctors reported spending 2 hours per week solving IT issues.15 Strategies for dealing with IT problems, including workarounds, were reported in many studies (21%, n = 7). For example, free-text fields were used to enter complex medication regimens when there were difficulties using CPOE systems.27,41 Another commonly reported strategy was to revert to paper, creating a hybrid record system (15%, n = 5). For example, when orders for some medications (eg, those requiring a variable dose regimen) could not be entered electronically, they would be written on paper, creating an opportunity for information to be missed.27,38
Outcome changedActual or potential patient harm was reported in 52% of studies (n = 18; Table 3). Patient deaths were reported in 7 studies; 6 of these were incident analyses.12,14,18,20,22,25 In the seventh study, CPOE implementation in a children's hospital and accompanying policy changes imposed by hospital management were associated with increased mortality, from 2.8% to 6.6%.42 After CPOE implementation, life-saving treatment for critically ill ICU patients was delayed because orders could not be entered unless patients were registered in the system. New workflows caused a breakdown in doctor-nurse communication, and changes to policies and procedures for dispensing and administering medications also delayed treatment. Only 2 studies provided details of patient deaths (Table 3). Of the 7 deaths for which details were available, 2 were linked to patient misidentification; 3 were associated with software use errors that resulted in failure to treat in 2 cases and a wrong procedure. The sixth involved a medication overdose due to poor software functionality. The seventh death was related to a delay in treatment following hospitalization because a pending test result from a previous hospitalization was not visible to the relevant clinicians.
Table 3.Reported effects of incidents and errors on care delivery and patient outcomes
Study Errors/ incidents (n) Potential or actual harm (%) Patient deaths (n) Near miss (%) Noticeable consequence but no patient harm (%) No noticeable consequence (%) Hazard (%) Incident analyses and surveys Cheung et al. (2013)20 668 58 Two deaths, no further details provided Hanuscak et al. (2009)32 39 36 None reported 64 Lei et al. (2013)25 116 1 One death, no further details provided 94 Magrabi et al. (2010)13 68a 3 None reported 4 38 34 13 Magrabi et al. (2012)14 Magrabi et al. (2011)50 436 11 Four deaths linked to patient misidentification, failure to treat, wrong procedure due to software use errors, overdose due to poor software functionality 10 32 46 Magrabi et al. (2015)12 850 3 Three deaths linked to patient misidentification, failure to treat due to software use errors, delay in treatment following hospitalization because a pending test result from a previous hospitalization was not visible to relevant clinicians 4 24 1 68 Magrabi et al. (2015)15 90 7 None reported 27 43 7 17 Myer et al. (2011)22 120 80 Deaths noted but number not specified Samaranyake et al. (2012)19 263a 11 None reported 89 Santell et al. (2009)18 90, 876 43 Three deaths, no further details provided 48 14 Warm et al. (2012)24 149 34 None reported Zhan et al. (2006)17 7,029 5 None reported 32 63b Observational studies Han et al. (2005)42 548 7 Mortality increased from 2.8% to 6.7%, no further details provided Nanji et al. (2011)40 466 35 N/A Palchuk et al. (2010)39 470 84 N/A Singh et al. (2009)29 558a 71 N/A Walsh et al. (2006)38 20 50 N/A 50 Westbrook et al. (2012)51 and (2013)41 493 2 N/AIn 2 studies analyzing safety events reported to the US Food and Drug Administration and from across England’s National Health Service (NHS), human factors issues were proportionally higher in the events involving patient harm.12,14 The potential of IT problems to lead to large-scale adverse events (ie, affecting multiple individuals) was reported in 2 studies.48 One was a study of safety events across England's NHS, where 23% of events (n = 850) affected more than 10 individuals.12 In the second study, 36% of system downtimes (n = 116) in China were estimated to affect more than 100 individuals.25 Near-miss events were reported in 29% of studies (n = 10).
Quality of studies and risk of biasAll 34 studies utilized observational designs. Mean study duration was 41 months, with a range of 1–144 months. Study population was broadly characterized by interviewees and survey respondents. In the 5 studies using interviews, there were 72 participants on average (range: 32–110), and there were 210 respondents (range: 32–369) in the 4 studies using surveys. We found that the average number of errors across the 21 studies was 5,401 (range: 20–90,876).
The main risk of bias was that the majority of studies were not true observational studies where the frequency of events was representative of the population, but were studies of incident reports where frequency could not be correlated with true population incidence. Furthermore, incident reports are potentially biased to events that appear important to the reporter.49 We assessed each study using the Cochrane Collaborations tool for assessing risk of bias. On average, data completeness was 74% across all stages in the information value chain. Machine errors (n = 12) and outcomes (n = 13) were frequently not reported. Only 7 studies (21%) provided information across all stages in the value chain, whereas 10 and 11 studies (29% and 32%) missed 1 and 2 stages, respectively. In 4 out of 5 studies (n = 28), user interactions, consequences, and medication error types were categorized.
DISCUSSIONProblems with IT are pervasive in health care. However, the evidence for IT-related disruptions to care delivery and risks to patient safety still comes largely from qualitative studies. Most of the evidence of patient harm comes from incident reports, with the exception of 1 comparative study where CPOE implementation was associated with an increased risk of mortality in a children's hospital.42 However, this finding was not replicated when the same system was implemented at different sites, and the disparity in outcomes was likely the result of differences in local implementation processes.52 More generally, only 2 other studies identified in this review were comparative, detecting an increase in duplicate medication orders30 and new system–related prescribing errors following CPOE implementation.41,51 While the types of IT problems have been well documented in the literature, further observational studies are required to measure their frequency and the magnitude of their impact on care delivery and patient outcomes.
Implications for measuring the effects of IT problemsThe limited evidence on the magnitude of IT problems and their impact may indicate an underlying problem with measurement.47 Existing classification frameworks tend to identify problems by their cause but not their effects,53 whereas patient safety frameworks do allow us to assign broad categories of consequence, such as whether or not a patient harm is considered severe.
In this review, we have attempted to develop a model that links cause to effect within the clinical decision-making and care process, using the information value chain as a template. The chain assists in identifying which process is impacted by an IT problem, but also shows the many stages through which information errors can then propagate.
The value chain thus offers a simple yet potentially powerful way to pinpoint specific threats to patient safety and identify the effectiveness of existing system defenses and new measures required to deal with clinical errors associated with IT. Were it to be used as a standard template in future studies, it would assist in making comparative assessments between studies. For example, the value chain could be integrated into the Statement on Reporting of Evaluation Studies in Health Informatics guidelines.54
The many reporting gaps identified in this review underline the need for a more structured approach to recording the causes and effects of IT problems in health care. It was not possible, for example, to quantify the propagation of information errors in the included studies, because these effects were not adequately described. Of the 34 studies we reviewed, only 44% (n = 15) reported effects at all stages of the chain. This is an inherent limitation, as each study has its own objective and may not consider all the different effects of IT problems along the value chain. For example, observational studies looking at medication errors may not look at patient outcomes.
Where the effects of IT problems were captured, data quality was poor. For example, IT problems, use errors, contributing factors, and clinical errors were not clearly differentiated.38 In other cases, IT problems were combined with use errors and contributing factors.41 Use errors were also combined with medication errors,39 and information errors were combined with decision-making errors.40 Other issues were related to heterogeneity in measures, even for the reporting of medication errors, which are among the most commonly studied errors in patient safety. For example, some studies reported common clinical error types (eg, wrong dose, wrong timing, wrong route, etc.),41 while others examined the clinical impact of omitted, unclear, and conflicting information in prescriptions.40
One way to improve data quality is to use existing schemas for describing and measuring variables along the value chain. For instance, our earlier classification is based on the natural categories of IT problems described in incidents from Australia, the US, and England.12 It provides a validated and now widely used schema for characterizing IT problems, contributing factors, and information errors and can be used in conjunction with the value chain to enhance measurement. Uniform characterization of information errors and their impact on patient safety can also provide a common language to facilitate collaboration and sharing among organizations with disparate IT implementations so that the most significant risks to patient safety can be identified.
User interactionThe information value chain begins with clinical users interacting with information from IT systems before considering decisions and taking action. The different types of IT problems that could affect user interaction are similar to those described in our earlier classification for safety problems associated with health IT. No new categories were required to code the IT problems, information errors, and contributing factors identified in this review, further validating the classification. The frequency of IT problems and their scale and severity are areas for further investigation.
Information receivedWe found that use errors that interfered with the receipt of patient information were reported in a majority of studies and were commonly linked to poor training and lack of familiarity with the system. Poor user interfaces also contributed to use errors and were sometimes exacerbated by machine errors. For example, clinicians frustrated by multiple irrelevant alerts were reported to disregard all alerts.26,36 Machine errors in displaying alerts and overalerting were reported in 7 studies, and 6 studies reported use errors involving wrongly overridden alerts. Information errors were poorly characterized. This may be inherent to the nature of incident reports, which made up the bulk of the studies reviewed. Incident reports only give a snapshot of safety events and are typically provided by clinicians, who gradually acquire the information required to make decisions and are accustomed to dealing with incomplete information. Therefore, information errors due to missing and partial information may not be available in incident data.
Decision changedThe impact of information errors on clinical decisions was not examined adequately in the studies reviewed here. Only 3 studies reported delays to decision-making due to IT problems. This figure may be underreported, given that software access problems were reported in many of the reviewed studies. For example, errors and delays in decision-making can occur when software is not available12,13,15 or not accessible12,13,17,22,24,27,28,33,34,37,42,46 due to power failure12,25,32,33 or computer viruses,12,15,25,32 but these were not explicitly noted. While observational studies are needed to understand patterns, controlled laboratory experiments can be used to quantify the impact of information errors on decision-making.55
Care process alteredWe found that clinical errors were well characterized in areas such as medications.17–21,29,31,38–41 For example, prescribing error types were neatly identified in 1 study that examined system-related errors.41,51 Taking the likelihood and impact of clinical errors into consideration, another study specifically sought to distinguish errors that were unique to IT as well as those that were made more likely with IT and more likely to cause harm with IT. Such approaches to understanding the nature of IT-related clinical errors will enable better targeting of strategies for prevention and mitigation. For example, errors that are no different from those found with paper records can be addressed by building upon existing patient safety initiatives. Patient identification is one such area where problems have existed with paper records due to gaps in local procedures, and these are likely to persist and propagate via electronic records. However, errors that are unique to IT and those that are more likely to occur or are more likely to cause harm may require new and innovative approaches.
IT problems hindering access to software delayed the initiation of clinical tasks, and software functionality issues delayed completion of clinical tasks. Such problems led to frustration for clinicians, wasted people’s time, and sometimes led to workarounds. Delays in care processes may have been likely in up to 21 studies where software was not available or accessible, or where power failures and computer viruses prevented access. The use of hybrid records may also be underreported. Only 5 studies (15%) reported using hybrid systems to work around IT issues. As part of contingency planning for planned and unplanned downtime, paper forms are generally used to document patient care and communicate with other departments.56 However, use of such procedures was not reported.
Outcome changedStudy designs and issues with data quality did not allow quantitative analysis of outcomes. The majority of studies were incident analyses, which are useful to understand the types and consequences of safety problems with IT and examine typical patterns along the value chain. However, as incident reports do not represent a systematic sample, they cannot be used to quantify the impact of IT problems on care processes and outcomes.57 Further observational studies are required to measure the frequency of the different types of IT problems and quantify their effects.58 While the impact of problems at a large scale such as system downtime might be quantifiable, the effects of low-frequency problems affecting small numbers of patients might be harder to measure than in other domains in patient safety.
Limitations of this reviewThis review has several limitations. It was restricted to studies of IT systems for clinicians that were published in the biomedical literature. We did not include a range of other sources of information about IT problems in health care, such as medical record review, routine data collection, medicolegal investigations, complaints, etc.49 It is thus possible that the IT problem types and effects are not exhaustive. Heterogeneity in study design and IT problem types prevented quantitative examination of effects on care delivery and patient outcomes.
CONCLUSIONThis review confirms that problems with health IT can disrupt care delivery and harm patients. The research evidence is largely qualitative, and there remain many opportunities to systematically study and quantify IT risks alongside its benefits to patient safety. The information value chain can be applied prospectively to quantify the effects of IT problems on user interaction information received, decision-making, care processes, and outcomes.
FUNDINGThis research is supported by Australian National Health and Medical Research Council Centre for Research Excellence in e-Health grant 1032664. The funding source did not play any role in study design, in the collection, analysis, and interpretation of data, in the writing of the report, or in the decision to submit the article for publication.
COMPETING INTERESTSThe authors have no competing interests to declare.
CONTRIBUTIONSFM and EC conceptualized the study. MK and FM led the literature search and data analysis and drafted the paper. MK is responsible for the integrity of the work; she is the guarantor. All authors participated in writing and revising the paper. All aspects of the study (including design; collection, analysis, and interpretation of data; writing of the report; and decision to publish) were led by the authors.
REFERENCESRetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.3
