electronic health record, electronic risk assessment tool, nurse perception, patient safety, risk



  1. Stafos, Andrea MSN, APRN
  2. Stark, Susan MSN, APRN
  3. Barbay, Kathryn MSN, APRN
  4. Frost, Kristen DNP, APRN
  5. Jackel, David MSN, RN
  6. Peters, Lindsey BSN, RN
  7. Riggs, Elizabeth MSN, APRN
  8. Schedler, Susan MSN, APRN
  9. Stroud, Shalan MSN, APRN


Objective: In many hospitals, nurse-led "safety huddles" are used to relay patient safety information, although whether this effectively identifies patients at risk for harm has not been determined. New electronic risk assessment tools are designed to identify patients at risk for harm during hospitalization, based on specific markers in the electronic health record. This study sought to compare the results of both methods. The findings may help to enhance decision making at the level of care delivery.


Methods: A nonexperimental correlational study was conducted over a three-week period in 2015 in a large metropolitan acute care community hospital. Nurses on three units-a medical-surgical unit, a progressive care unit, and an orthopedic unit-constituted the convenience sample. Designated safety huddle leaders collected data using the daily census sheet to record the nurses' perceived risk of harm for each patient and the reason for risk concern. Separately, designated advanced practice nurses collected the electronic risk assessment tool's reports from the same units. Data were paired as they were entered into the database and analyzed to determine correlation. Perceptions of harm from the nurses, recorded as yes or no responses, were compared with the electronic tool's identification of high risk or moderate-to-low risk.


Results: In 746 data pairs, differences between the nurses' harm risk perceptions and the electronic tool's harm risk reports were statistically significant, supporting our prediction that there would be no correlation. The most significant difference was seen in instances when a nurse identified a patient as being at higher risk than the electronic tool did, often citing behavioral or psychosocial issues as the reason for concern.


Conclusions: Nurses perceived harm risk differently than the electronic tool did. In situations when the electronic tool cited risk and the nurse perceived no risk, the risks were currently being addressed in the plan of care. In situations when the nurse perceived higher risk than the electronic tool did, the nurse often cited behavioral or psychosocial issues (which frequently lacked defined data points in the electronic health record and thus were not available to the tool). Changes in data mining algorithms must incorporate and weight the impact of psychosocial and behavioral elements together with other risk factors in order to provide meaningful practice recommendations.


Article Content

After the Institute of Medicine (IOM) published To Err Is Human: Building a Safer Health System in 1999, patient safety efforts increased dramatically as hundreds of toolkits, training sessions, and other publications were developed. In this and subsequent reports, the IOM identified a critical need-the provision of safe, effective, patient-centered, timely, efficient, and equitable care-and emphasized the role of technology and informatics in meeting this need.1 Technologic advances can help health care systems build "a stronger information infrastructure," which can aid in preventing harm and in learning from any errors that do occur.2 And in 2009, the National Quality Forum convened for the sole purpose of standardizing patient safety terminology. It defined harm as "any physical or psychological injury or damage to the health of a person, including both temporary and permanent injury."3


Unfortunately, these efforts toward creating a culture of safety at the national and local levels have not been fully appreciated.4 Despite national and international efforts, health care organizations have struggled to realize the full potential of the strategies recommended by the IOM and the Institute for Healthcare Improvement (IHI), among other agencies, and adverse events continue to occur.5-7 Although harm has primarily been identified retroactively, new efforts are aimed at improving its preemptive identification, thus diminishing risks for patients.


In many hospitals, nurse-led safety briefings or "huddles" are used to relay patient safety information, although whether this effectively identifies patients at risk for harm has not been determined. New electronic risk assessment tools are also available; these are designed to recognize patients at risk for harm during hospitalization, based on specific markers in the electronic health record. We were interested in how these two methods might compare-specifically, how an objective electronic risk assessment tool would compare with nurses' subjective judgment in identifying patients at risk. Through our study, we hoped to provide added insight into how organizations can promote a culture of safety.



Electronic risk assessment tools. The study site is part of a 45-hospital multistate health care system. In 2009, the health care system's quality leadership team began considering system-wide implementation of the Global Trigger Tool for Measuring Adverse Events (known simply as the GTT), which was developed by the IHI. The GTT was created as a way to identify adverse events and patient harms over time, thereby allowing for a review of trends that can direct process improvement.8 The tool uses retrospective chart review of a random sample of inpatient records to identify "triggers" associated with possible adverse events, which can prompt a more in-depth evaluation of actual harms identified in a given patient's record. The GTT has been a part of the IHI's 5 Million Lives Campaign and is now used by hundreds of hospitals in multiple countries.8


In its initial evaluation, the quality leadership team used the GTT to perform systematic reviews of electronic health records from each of its hospitals. In accordance with the GTT, the team then classified identified harms using the National Coordinating Council for Medication Error Reporting and Prevention Index.8 But although research has demonstrated the effectiveness of data mining technologies in retrospectively detecting adverse events, our search of the relevant literature found little research examining the use of such technologies in relation to real-time patient care.9


In 2011, De Wet and Bowie described a process for using a real-time risk trigger tool in the primary care setting.10 The process involves three stages: planning and preparation, which includes selecting specific clinical triggers; systematic review of a random sample of patient records; and reflection and action, which includes possible improvements to reduce harm. Our health care system partnered with a federally certified patient safety organization to explore the potential use of a new tool based on the GTT. Its software performs real-time data mining of electronic health records to identify triggers thought to be predictive of potential patient harm. This represents an evolution from older, manually performed retrospective chart reviews, which were aimed at interrupting cascading adverse events and generating event-specific population analyses. The data mining software incorporates information that reflects measures aimed at both harm risk mitigation (such as falls prevention) and clinical risk assessment (such as infection screening).


In April 2015, the patient safety organization offered our health care system an opportunity to implement the new tool. The tool provides a real-time harm risk score that can focus clinicians' attention on the patients who are most vulnerable. The risk score is determined both by data mining techniques and by the software's proprietary algorithm. The tool captures, but is not limited to, the following factors: fall and skin assessment scores, antiemetic and anticoagulant administration, laboratory values, surgical intervention, restraints, and need for intubation. It uses data from a patient's electronic health record to calculate her or his harm risk score, and categorizes the patient as at low, moderate, or high risk.


Nurses and safety huddles. Early and accurate identification of patients' risk of harm is clearly crucial to ensuring a culture of patient safety in any health care organization. Of all providers, nurses are the most likely to be aware of a patient's change in status or to discover that a medical error has occurred.11 There is undoubtedly a subjective aspect to identifying patients at risk. In one project, Fiandra and colleagues noted that although nurses often cited clinical factors such as diagnostic or therapeutic errors as reasons for concern, they also frequently perceived risks related to nonclinical factors such as poor communication and staffing shortages.12 In a study by Abbasi and colleagues, nurses retrospectively reviewed patient safety events and assigned them to harms categories according to criteria established by the Agency for Healthcare Research and Quality.13 The researchers found that subjective variance was demonstrated by nurses even within this seemingly objective classification system. Nursing concern is crucial to mitigating patients' risk of harm. Yet how nurses evaluate and respond to environmental and physiological stimuli both retrospectively and in real time is somewhat subjective in nature.11, 14


Along with electronic risk assessment tools, the IHI has proposed using safety huddles at shift changes as a strategy for relaying patient information and empowering providers to modify processes, thereby facilitating a culture of safety.15 The literature demonstrates that safety huddles can positively affect a variety of outcomes by improving the efficiency and quality of information sharing, increasing accountability and a sense of empowerment, fostering collaboration, and creating community.16


In 2009, in response to the IOM's To Err Is Human report and the Robert Wood Johnson Foundation's Transforming Care at the Bedside initiative, developed in collaboration with the IHI,17, 18 our health care system began implementing safety huddles. The purpose was to enhance staff communication in order to improve patient care and safety. Implemented as 10-minute team briefings held early in each shift, safety huddles included RNs, care assistants, and other multidisciplinary staff based on patients' needs, and offered staff opportunities to create greater awareness of high-risk situations and discuss patients of concern. Adoption across the system occurred gradually over the course of about 18 months.


In 2012, however, an evaluation of huddle efficacy revealed that the process had eroded, with huddles yielding only insignificant paper documents. Recognizing the need for a more dynamic, interactive process, organization leaders began incorporating the use of dry-erase boards to communicate patient safety matters and concerns about risk of harm. This method was still in place at the time of our study.


Study aim and hypothesis. Knowing that our health care system was planning to implement the use of an electronic risk assessment tool, we were interested in how the tool would compare with nurses' judgment in decision making at the level of care delivery. The primary study objective was to compare patients identified as of concern by bedside nurses during safety huddles with patients identified as at high risk for harm by the electronic risk assessment tool in real time. The secondary objective was to describe the reasons nursing staff gave when they expressed concerns about patients. We hypothesized there would be no correlation between the patients identified by each method.



Design and sample. The study was developed using a nonexperimental correlational study design. It was conducted in a 504-bed nonteaching metropolitan acute care community hospital. The hospital is an accredited center for treating chest pain, stroke, and diabetes, and for performing bariatric surgery, and is currently pursuing Magnet designation. Safety huddles are conducted on patient care units twice daily, and are led by a designated charge nurse or nurse manager. These designated leaders conduct the huddles using a daily census sheet, recording nurses' concerns about patients at risk for harm and their reasons for such concerns.


The convenience sample included nurses in safety huddles on the medical-surgical, progressive care, and orthopedic units. The medical-surgical unit is a 48-bed adult nontelemetry unit serving general medical and surgical patients. It has an average daily census of 40 patients and a five-to-one patient-to-nurse ratio. The progressive care unit is a 33-bed adult intensive care step-down noncardiac telemetry unit serving a large pulmonary care population. It has an average daily census of 32 patients and a four-to-one patient-to-nurse ratio. The orthopedic unit is a 25-bed adult nontelemetry unit serving patients undergoing joint replacement and spinal surgeries or treatment for fractures. It has an average daily census of 20 patients and a six-to-one patient-to-nurse ratio. The experience level of the nurses on the study units ranged from a few weeks to over 40 years.


Procedure. Data were collected from the safety huddles and the electronic risk assessment tool twice daily for a period of three weeks during the fall of 2015. Using the daily census sheet, the designated charge nurses for the huddles recorded nurses' concerns about patients at risk for harm and the reasons for their concerns. Yes was recorded for a patient if concern was expressed by the nurse assigned to that patient. The major reason for concern was also indicated on the sheet. No was recorded for a patient if the nurse expressed an absence of concern. We assumed that concerns were based on the nurse's experience, assessments, practice patterns, intuition, or a combination thereof. Nurses did not differentiate between safety and clinical risks, and were not instructed in definitions of patient harm. The census sheets from the safety huddles were retained in a locked cabinet.


The electronic tool's software automatically refreshed the data and generated new risk scores twice daily at 5 AM and 5 PM, in accordance with the hospital's routine shift changes at 7 AM and 7 PM. Designated safety huddle leaders and bedside staff nurses did not have knowledge of or access to the electronic risk assessment tool, and use of the tool was not incorporated into practice. For our study, we printed the tool's twice-daily reports, which listed the electronically generated risk scores. These reports were retained in a separate locked cabinet.


At the end of the study period, the principal investigator (one of us, Susan Stark) collected the data sheets from the locked cabinets. Patients were coded and pairing was accomplished by viewing both the huddle form and the tool's dashboard simultaneously and entering this information into the study database. Concerns identified by a nurse as yes or no in the huddle were paired in the database with risk identified by the electronic tool, which categorized a patient's degree of risk as high, medium, or low. A high electronic risk score (from 28 to 100 points) was entered by the principal investigator as a yes, indicating that the patient was at high risk. A moderate (from 5 to 27 points) or low (from 1 to 4 points) electronic risk score was entered by the principal investigator as a no. The patient safety organization that created the electronic risk assessment tool recommended that clinicians focus on patients with high risk scores in order to identify trends and facilitate intervention. For this reason, moderate and low risk scores were excluded from the yes risk category. No score available was entered for patients without a score. Reasons for the huddle nurses' concerns were also entered into the database. Data were recorded such that the nurses were not identified. The database automatically assigned a number to each pairing upon data entry, and identifying information for patients was not retained outside of the database. Data collected from the nurses during safety huddles were then compared with the electronic risk scores to determine correlation.


Data analysis. The data were analyzed using IBM SPSS software, version 23, and the McNemar test, in order to determine the significance of paired findings. A sample size of 35 nurse-concern and harm-risk-score data pairs was calculated to achieve a power of 0.80 for the McNemar test at an [alpha] level of 0.05 and medium effect size.


Ethical considerations. The study design was presented to and approved by hospital leadership and the hospital's Council for Research and Evidence-Based Practice and was then submitted for institutional review board (IRB) approval. As this hospital does not have an IRB, a formal contract with the Patient Advisory Council (PAC)-an independent, central IRB-was retained for this purpose.


After careful consideration, the PAC granted an exemption to obtaining the nurses' consent. This was because safety huddle data were obtained from existing documents; nurses were not identified on either the electronic risk assessment tool or the unit census sheets; and study data were paired and recorded such that the nurses could not be identified.



The study yielded 746 data pairs of nurse-identified and electronic-tool-identified harm risks. The McNemar test determined that the distributions of related values across nurse-identified harm risks and the electronic tool's harm risks were significantly different (P < 0.05). The findings showed that, of the 215 nurse-identified yes harm risks, 42 were paired with tool-identified yes (high) harm risks (20% of the total nurse-identified yes's) and 173 were paired with tool-identified no harm risks (80% of the total nurse-identified yes's). Of the 531 nurse-identified no harm risks, 60 were paired with tool-identified yes (high) harm risks (11% of the total nurse-identified no's) and 471 were paired with electronic no harm risks (89% of the total nurse-identified no's). (See Table 1.) The most significant finding was seen in pairings of nurse-identified yes harm risks and tool-identified no harm risks: instances in which a nurse had identified a harm risk and the electronic tool had not.

Table 1 - Click to enlarge in new window Comparison of Nurse-Identified and Electronic Tool-Identified Harm Risk in Hospitalized Acute Care Patients

For patients who were identified by nurses as of concern but who had low-to-moderate electronic harm risk scores, the most common reasons for concern given by the nurses involved behavioral, affective, and psychosocial factors. These included confusion, delirium, family dynamics, discharge planning, infections, anticipated cardiopulmonary arrest, end-of-life issues, and alcohol or drug withdrawal. Such factors likely increase a patient's risk of harm; yet many cannot be captured by data mining software because they lack a defined data point. Furthermore, some factors (such as sedation, delirium, or alcohol withdrawal) do have defined data scores, but were not included in the tool's proprietary algorithm.


In many instances, factors that the tool captured had been anticipated by the nurse or were already addressed in the plan of care. We assumed this was why nurses reported no concern for many patients whom the tool identified as at high risk for harm. Among patients for whom nurses reported no concern but who had high electronic risk scores, many were immediately postoperative and the risks were readily apparent. The tool's software mines data that are already documented in the electronic health record, therefore, any obvious concerns likely have already been accounted for in the nurse's plan of care. For patients who were identified by nurses as of concern but who had low-to-moderate electronic risk scores, there were often minimal diagnostic information and limited assessment data in the electronic health record.



For every patient, the anticipated risks of harm associated with disease and illness, such as risks related to skin integrity, deep vein thrombosis, nutrition, and mobility, were addressed in the standard plan of care. In our study, a patient's harm risk was often elevated for prolonged periods because the electronic risk assessment tool could not take into account the patient's progress toward discharge in those anticipated areas of risk. Continuously high harm predictions despite patient progress can create distractions from the pursuit of patient safety. Although algorithmic data mining in patients' electronic health records may have merit in training providers, our findings indicate that this approach fails to integrate the critical assessments of the bedside nurse.


Limitations. There are inherent limitations associated with a nonexperimental correlational study, such as confounding variables and range restriction. We tried to minimize these limitations by using data that were routinely collected during safety huddles on the three nursing units. Another limitation is that the validity of the electronic risk assessment tool has yet to be established. Moreover, the study design did not lend itself to interrater reliability. Nursing assessment is subjective, and interrater reliability or validity based on actual progression to harm could not be established.


Recommendations. Previous research has explored the use of electronic tools to predict a patient's risk of harm, but has given little attention to the perceptions of the bedside nurse in this area. Our study found that the nurses and the electronic tool identified harm risks differently. Further investigation into objective versus subjective perceptions of harm risk is indicated.


Finding a way to consistently define and document nurses' subjective concerns in the electronic health record might allow such concerns to be integrated into electronic risk prediction models. Nurses often rely on intuition and experience to identify and act on their concerns. Thus studies that explore nurses' psychosocial and intuitive processes in this regard would also be valuable.


Our study did not evaluate actual harm, but rather compared nurses' perceptions of risk with risk scores generated by the electronic tool. Further research is warranted to investigate how well nurses' concerns and electronic tool scores are predictive of instances of actual harm. Research aimed at identifying harm risks along the continuum of patient care, which may contribute to readmission and complications, is also recommended.



The use of electronic health records has led to the development of innovative tools aimed at enhancing patient care, including tools like the one used in this study. Improved data mining software designed to predict and prevent patient harm may revolutionize patient care delivery for the bedside nurse. This study highlighted differences in how nursing intuition and an electronic tool identified patients' risk of harm. The results suggest that, in its current iteration, this tool is unlikely to enhance safety huddles or to improve patient care. Given these findings, the study hospital consequently decided not to implement the use of the tool. Data mining algorithms must incorporate and weight the impact of many more factors-including psychosocial, behavioral, and cognitive elements and end-of-life issues, among others-in conjunction with other harm risks in order to provide meaningful practice recommendations.




1. Committee on Quality Health Care in America, Institute of Medicine. Crossing the quality chasm: a new health system for the 21st century. Washington, DC: National Academy Press; 2001. [Context Link]


2. Aspden P, et al Patient safety: achieving a new standard for care. Washington, DC: National Academies Press; 2004. Quality chasm series; [Context Link]


3. National Quality Forum. NQF patient safety terms and definitions. 2009. [Context Link]


4. Bleich S Medical errors: five years after the IOM report. New York: The Commonwealth Fund; 2005 Jul. Issue briefs; [Context Link]


5. Aranaz-Andres JM, et al Prevalence of adverse events in the hospitals of five Latin American countries: results of the 'Iberoamerican Study of Adverse Events' (IBEAS) BMJ Qual Saf 2011 20 12 1043-51 [Context Link]


6. Classen DC, et al 'Global trigger tool' shows that adverse events in hospitals may be ten times greater than previously measured Health Aff (Millwood) 2011 30 4 581-9 [Context Link]


7. Landrigan CP, et al Temporal trends in rates of patient harm resulting from medical care N Engl J Med 2010 363 22 2124-34 [Context Link]


8. Griffin FA, Resar RK IHI global trigger tool for measuring adverse events (second edition). Cambridge, MA: Institute for Healthcare Improvement; 2009. IHI innovation series white paper; [Context Link]


9. Bakken S Informatics for patient safety: a nursing research perspective Annu Rev Nurs Res 2006 24 219-54 [Context Link]


10. De Wet C, Bowie P. Screening electronic patient records to detect preventable harm: a trigger tool for primary care. Qual Prim Care 2011;19(2):115-25. [Context Link]


11. Despins LA, et al Detection of patient risk by nurses: a theoretical framework J Adv Nurs 2010 66 2 465-74 [Context Link]


12. Fiandra U, et al The perception of health care risk: patients, health care staff and society Blood Transfus 2008 6 2 93-100 [Context Link]


13. Abbasi T, et al Accuracy of harm scores entered into an event reporting system J Nurs Adm 2015 45 4 218-25 [Context Link]


14. Henneman EA, et al Strategies used by critical care nurses to identify, interrupt, and correct medical errors Am J Crit Care 2010 19 6 500-9 [Context Link]


15. Institute for Healthcare Improvement. Develop a culture of safety. 2014. [Context Link]


16. Goldenhar LM, et al Huddling for high reliability and situation awareness BMJ Qual Saf 2013 22 11 899-906 [Context Link]


17. Institute for Healthcare Improvement. Transforming care at the bedside. 2011. [Context Link]


18. Viney M, et al Transforming care at the bedside: designing new care systems in an age of complexity J Nurs Care Qual 2006 21 2 143-50 [Context Link]


For 117 additional continuing nursing education activities on patient safety, go to