eLearning, EMR training, Inpatient nurses, Newly hired, Nursing education, Usability



  1. Smailes, Paula S. DNP, RN
  2. Zurmehly, Joyce PhD, DNP, RN, NEA-BC
  3. Schubert, Carolyn DNP, CNE, RN-BC
  4. Loversidge, Jacqueline M. PhD, RNC-AWHC
  5. Sinnott, Loraine T. PhD


In recent times, policies stemming from the American Recovery and Reinvestment Act of 2009 have served as a stimulus for healthcare organizations to adopt an electronic medical record. As a result, nurses are now more knowledgeable of and experienced with an electronic medical record. In August 2016, our facility converted from instructor-led training to electronic learning for inpatient nurse electronic medical record training, hoping to capitalize on previous experience with the clinical information system. However, a complete program evaluation of this transition had yet to be conducted. The purpose of this study was to evaluate electronic learning usability and the return on investment of an electronic medical record training conversion. Evaluations of electronic medical record electronic learning training were collected from 75 newly hired, inpatient nurses from November and December 2017, and compared to our instructor-led program. Results showed that users found it effective and were satisfied with this training method. The electronic learning had superior efficiency, reducing training time by ~50% compared to instructor-led training, while proving to yield effectiveness and satisfaction. The return on investment was $18 540, with a gain of 593.25 hours in nursing time during the study period of two months. These results support the organizational decision to convert to electronic learning, further supporting the conversion for other clinical roles.


Article Content

With most hospitals now using an electronic medical record (EMR), a new trend has emerged. As organizations hire new nurses, they arrive with previous EMR experience. This has served as a catalyst to rethink how EMR training should be conducted within our organization. A traditional approach is instructor-led training (ILT), which is time-consuming for both information technology (IT) departments and nursing units. A more efficient training methodology that capitalizes on existing EMR knowledge could have many positive effects, including faster onboarding, increased end-user satisfaction, and potential organizational savings. One such training methodology is a self-paced, electronic learning (eLearning) format, which caters to existing EMR use. According to MacDonald et al,1 eLearning involves the use of technologies to deliver learning solutions that enhance knowledge and skills electronically with high levels of interactivity, flexibility, information access, and communication. Evidence suggests that there is no difference in knowledge, skills, and satisfaction between ILT and eLearning.2-6 If organizations can successfully transition to this format of EMR content delivery, there could also be potential for reduced training time, increased user satisfaction, and organizational revenue.



The Health Information Technology for Economic and Clinical Health (HITECH) Act was created in 2009 as part of the American Recovery and Reinvestment Act, which began the push for healthcare organizations to convert from paper to EMRs.7 The Centers for Medicare & Medicaid Services enforced the HITECH Act with incentive payments to eligible hospitals that were meeting the criteria for meaningful use of EMRs.8 Meaningful use relates to using certified health IT to maintain privacy and security; improve care coordination; engage patients and families; improve quality, safety, and efficiency; and reduce patient disparities in hopes of achieving optimal clinical outcomes.9 For those who did not meet meaningful use, financial penalties were enforced. Because of this, organizations began implementing EMRs at a rapid rate.10


At the same time that the HITECH Act prompted organizations to convert to EMRs, The Joint Commission (TJC) began to recognize the inherent risks of conversion. Sentinel Event Alert (SEA) #42 was issued by TJC to address the safe use of IT in healthcare.11 In this alert, TJC11 expressed concern that any form of technology might adversely affect the quality and safety of care if it is implemented improperly. They suggested training programs and refresher courses for all types of clinicians as one of many actions within SEA #42 that may help to prevent patient harm related to health IT.11 As more organizations began utilizing an EMR, an additional alert was issued; SEA #54 addresses the communication workflow and human-computer interface, which relates to usability, ergonomics, and data-related errors.12 The Joint Commission recommends that organizations provide training and have users demonstrate competency before accessing the system.12



The purpose of this project was to evaluate the eLearning usability and return on investment (ROI) of a nursing EMR training conversion from ILT. The International Organization for Standardization has established three goals for usability: effectiveness, or how accurately and completely the user achieved goals in specific environments; efficiency, which entails the resources expended in relation to the accuracy and completeness of goals; and satisfaction, involving the perception of comfort and acceptability associated with the product.13 To improve usability, organizations need to focus on users, along with the processes and procedures for user-centered design, while possessing the funding and resources to make positive change.14




Several studies have indicated there is no difference in effectiveness and knowledge gained between ILT and eLearning.2-5,15-20 However, Sung et al21 found that nurses who received eLearning training had a higher level of knowledge related to medication administration compared to those receiving ILT. In addition, Kontio et al22 found that general self-efficacy strengthened in the eLearning group at the 3-month time point. Wutoh et al6 revealed that Web courses were more effective than ILT formats.



The literature indicates that eLearning may facilitate faster training than traditional, instructor-led methods. When incorporating eLearning methods, Sung et al21 found reduced lecture time, which eased the burden and increased the satisfaction of instructors. Wutoh et al6 also found that learning efficiency was higher with online education. Berke and Wiseman23 noted a transition to eLearning from traditional classroom training could save time by as much as 25% to 60%. In their study comparing classroom training to eLearning, Sheen et al24 discovered that the convenience of eLearning also saved time in the learners' professional lives. They further stated that if nurses could self-initiate and self-direct to improve their knowledge and skills at any time and location with eLearning, the costs to hospitals and nurses for continuing education could be reduced.24



The evidence suggests that nurses are equally satisfied with an eLearning training methodology when compared to ILT.3,4,15,21,24,25 Determinants of satisfaction with eLearning have included learning climate, performance expectations, learner interface, system functionality, intention to use, and computer self-efficacy.26 It was also noted that system quality, information quality, and service quality of a nursing e-learning system have a direct effect on nurse satisfaction with system use.27 The concept of eLearning has been studied across disciplines, resulting in user satisfaction with eLearning approaches.28-30 No evidence was found to suggest that online training methodology had a negative overall impact on nursing satisfaction.


Theoretical Basis

The Information Systems Success Model (ISSM) was developed in 1992 by DeLone and McLean31 to evaluate system quality, system use, information quality, and user satisfaction related to the subsequent success of information systems. They updated the model in 2003 to include the impact of the organization and individual from use of information systems.32 This theory has been used extensively to evaluate eLearning success27,30,33-36 and provides the ideal framework for the project. The conceptual definitions of this theory will be applied to the project interventions and outcomes, while serving as a framework for evaluation (Table 1).

Table 1 - Click to enlarge in new windowTable 1 Theoretical Substruction of the ISSM and EMR Training Conversion to eLearning



This study was conducted at a large, Midwestern academic medical center that comprised seven hospitals on two campuses, with a combined total of 1382 beds.37 The Hospital Feasibility Committee reviewed the project and determined it to be a quality improvement project; thus, institutional review board submission was not necessary. A convenience sample of newly hired, onboarding nurses was used with a mixed-methods design. The eLearning conversion began in August 2016 for inpatient nurses, followed by the evaluation in November and December 2017.



Newly hired nurses working on inpatient units made up the sample. These participants were in their first week of hire, when they were required to begin their EMR training via eLearning in 4-hour time slots on 2 separate days. Participants arrived at a proctored computer laboratory and were given instructions that served as a guide for completion. They were provided initial guidance on how to access the organization's learning management system (LMS) for the completion of the required EMR training sessions in an eLearning format. A proctor was present to assist with answering questions and troubleshooting issues and provide verbal cues to encourage completion of the evaluations.



The usability categories of effectiveness, efficiency, and satisfaction were determined using a post-eLearning evaluation that all end participants took at the completion of training. The evaluation included measures pertaining to demographics, which included years of nursing and Epic EMR experience, satisfaction and effectiveness, and playground feedback. The eLearning evaluation was adapted from the initial ILT evaluations. Both eLearning and ILT evaluations contain content validity, which was determined by experts on the IT Training and Optimization team. The eLearning development process involved several members of this team who ensured quality for the product delivered.


The initial eLearning conversion began in August 2016, and the evaluations were launched into the system in October 2017. Anonymous evaluation data from participants were collected every 2 weeks beginning November 1, 2017, through December 2017. The evaluation contained 13 questions to be answered with a Likert scale or yes/no response. The Likert scale ranged from 5 = strongly agree to 1 = strongly disagree. An additional question collected qualitative responses. The evaluation was distributed in the LMS, and results were exported into an Excel file (Microsoft, Redmond, WA) as aggregate, deidentified data. Evaluations from ILT from November and December of 2015 were used to assess satisfaction and effectiveness of that learning methodology for comparison with eLearning. Deidentified, aggregate data were collected for both ILT and eLearning evaluations.


The LMS captured eLearning efficiency in a course timings report for each eLearning module. Participants were encouraged to limit breaks while a lesson was open, in an attempt to get the most accurate completion time. The eLearning completion times were compared to the time allocation for the same ILT course to determine efficiency and ROI using the average salaries for a newly hired inpatient nurse.


Data Analysis

Data were analyzed using SAS 94 (SAS Institute, Cary, NC). Descriptive statistics were used in the form of mean, SD, and frequency percent to show how the eLearning met nursing perceptions of effectiveness and satisfaction. Inferential statistics in the form of independent-samples t tests were used to determine group differences with respect to confidence, satisfaction, and effectiveness among participants taking the eLearning. Comparisons were made based on nurses with and without Epic EMR experience and their years of nursing experience. Finally, one-sample t tests were used to assess whether eLearning and ILT differed with respect to meeting course objectives and nursing satisfaction.


The last item on the survey was an open-ended question that asked participants to share feedback on the eLearning regarding what they liked and what could be improved for the future. Qualitative data collected from this question were analyzed by coding the information and placing it into themes38 reflective of the concepts of usability.



The eLearning evaluations were completed by 75 participants, all inpatient nurses in their first week of hire. Four separate EMR training sessions were used, three in November 2017 and one in December 2017. Results show that 76% of participants (n = 57) reported coming to the organization with prior experience with an Epic EMR. The range of prior Epic EMR experience was 1 year up to 10 years (mean = 2.49 [SD = 1.90]). Years of nursing experience were reported by 46 participants, with the range from new graduate (n = 10) to 24 years (mean = 6.46 [SD = 6.68]).



Participants were asked whether the eLearning was an effective means to facilitate learning. Eighty-four percent (n = 63) agreed that the eLearning was an effective means for learning. Overall, the respondents were positive about most aspects that made the eLearning an effective tool (Table 2). The playground is an EMR environment where nurses can practice workflows to gain confidence in system use. While the playground exercises were made available to everyone, 93% (n = 70) of the participants indicated they were effective, and 77% reported they had time to access them. Participants were neutral (mean = 3.65 [SD = 0.95]), when asked if the playground exercises enhanced their experience.

Table 2 - Click to enlarge in new windowTable 2 Effectiveness Data From eLearning Evaluation


With ILT, each of the four classes, Nursing Foundations, Admission, Shift, and Discharge, lasted 4 hours, a total of 16 hours to complete. These same classes were converted to eLearning modules and took a combined 485.77 minutes, or 8.09 hours, for an approximate 50% reduction in completion time.



When participants were asked if they were satisfied with the online courses, 76% reported they agreed (n = 41) or strongly agreed (n = 16), while 13% (n = 10) were neutral and 11% (n = 8) disagreed. Although the majority of participants reported satisfaction, the satisfaction mean was 3.87. Participants who disagreed cited reasons in qualitative feedback such as not having enough time to complete the eLearnings, preference of having the content be delivered as instructor-led, proctor issues, and too slow for someone with Epic EMR experience.


Return on Investment

To calculate the ROI for this project, the midpoint of the current pay range for a staff nurse role was provided by human resources. The ROI for the project duration was $18 500 and for the calendar year of 2017 was $156 935. When converted to hours gained, the nursing department gained 593.25 hours during the project period and 5022.85 hours during 2017 (Table 3).

Table 3 - Click to enlarge in new windowTable 3 Return on Investment for Project Duration and for 2017 Calendar Year

Group Comparisons

Independent-samples t tests were conducted to identify any differences among participants taking the eLearning based on demographics. The first group comparison analyzed the variables of confidence, satisfaction, and meeting the learning objectives of participants with (n = 57) and without (n = 18) Epic EMR experience. Results showed that with confidence (P = .44), satisfaction (P = .30), and whether the eLearning met the intended objectives (P = .13), no statistical differences were found (Table 4).

Table 4 - Click to enlarge in new windowTable 4 Participants With and Without Prior Epic EMR Experience Compared With Confidence, Satisfaction, and Meeting Objectives

Next, participants were divided between those with 2 years of nursing experience or less (n = 27) and those with more than 2 years of experience (n = 19) to see if any differences existed among confidence, satisfaction, and meeting learning objectives. Results showed that confidence (P = .71) and satisfaction (P = .52) were not statistically different between the two groups at the 0.05 level, but less experienced participants had higher average scores regarding whether the lessons, topics, and content were directly related to the learning objectives (P = .0.04) (Table 5).

Table 5 - Click to enlarge in new windowTable 5 Nursing Experience Compared With Confidence, Satisfaction, and Meeting Objectives

The ILT and eLearning results were further compared using one-sample t tests based on meeting course objectives and user satisfaction, which were present in both postcourse evaluations. One hundred thirty-seven ILT evaluations were used: 9 from Foundations, 33 from Admissions, 43 from Shift and 52 from Discharge. The ILT satisfaction mean was 4.41, and for meeting course objectives, the mean was 4.61. When the eLearning satisfaction mean of 3.87 was compared to ILT average of 4.41, statistical significance was found (P < .0001). Similarly, when the ILT mean for meeting the learning objectives (4.61) was compared to the eLearning (mean = 4.36), this demonstrated statistical significance (P = .0010) (Table 6).

Table 6 - Click to enlarge in new windowTable 6 Comparison of eLearning and Instructor-Led Sessions in Terms of Objectives and Satisfaction

Qualitative Data

The last item on the survey was an open-ended question that asked the participants to share feedback regarding what they liked about the eLearning and what could be improved for the future. These responses were sorted into common themes based on keywords that reflected effectiveness, efficiency, and satisfaction.


In order to further assess nursing satisfaction, participants offered feedback on what they liked about the eLearning. Eight participants responded to this question and the common theme was the positive review of the self-paced format. One participant stated, "I liked that it was go at your own pace." A related common theme was that experienced users would like the opportunity to test out of content. However, even though self-paced was preferred by most, two participants stated that the content should be delivered as ILT.


Twenty-five participants offered qualitative feedback for what could be improved. The most common theme related to allotted time for completion of the eLearning. A participant stated, "Not enough time for the amount of education required." Another said, "[horizontal ellipsis]no time for playground due to it taking the entire time to complete just the online course." Another common theme related to the proctor overseeing the eLearning. One comment read, "I would have benefitted from communication from the staff person overseeing it at the beginning, giving us a welcome, overview, instructions, telling us what to expect, when to ask for help, basically any (communication)".



Project Outcomes

The eLearning usability goal was met and comparable to the inpatient nursing eLearnings for EMR training. The majority of participants were satisfied with the eLearning, even though the mean of 3.87 shows that there is room for improvement. The eLearning was found to be an effective tool for content delivery. The results confirmed that eLearning is more efficient than an ILT method of delivery. The 50% reduction in training time was achieved compared to ILT, which leads to a substantial ROI for the organization. Upon completion of the inpatient nurse curriculum via eLearning, this project showed no difference between nurses with and without prior Epic EMR experience related to confidence, satisfaction, and meeting eLearning objectives. Furthermore, no statistical difference was found between participants with less than 2 years of experience compared to participants who had more than 2 years of experience regarding satisfaction with the eLearning and future confidence to use the system. However, new participants reported that the eLearning objectives were met more often than did experienced participants.


Compared to eLearning, ILT was superior in terms of satisfaction and meeting course objectives. Qualitative feedback demonstrated that proctor performance affected learner satisfaction and that many participants liked the self-paced format of eLearning. Those who need more time can take it, while those who know the workflows can move faster. Improvements related to this feedback may raise satisfaction with eLearning to the ILT level.



There were several identified limitations to this project. Some participants did not answer all of the questions, leading to missing data. Additionally, the course timings report pulled the amount of time that the learner had the eLearning lesson open. Because of this, it should be noted that the time for completion of the eLearning may actually be less than what was indicated in the course timings report. Finally, while 137 ILT evaluations were used, we do not know how many participants completed evaluations.



With the majority of inpatient nurses onboarding with Epic EMR experience, our inpatient nursing eLearning conversion allows our organization to capitalize on existing experience and increase satisfaction with a self-paced format. In the future, a mechanism to allow users with prior Epic EMR experience to test out of content would be beneficial and allow for an increased ROI. However, with the average time of completion at 8.09 hours, we need to provide some users with additional time. The eLearnings also provide a means for ongoing reference for nurses that was not possible with ILT. The important task of orienting newly hired nurses to an organization's information systems and workflows has grown more challenging in the increasing presence of electronic record technologies utilized with patient care.39 New nurse hires' EMR training has an important function of incorporating workflows and policies, in addition to introducing system functionality. This project has shown that an eLearning format does yield confident nurses who are equipped with the knowledge they need to be successful with patient care.




1. MacDonald C, Stodel EJ, Chambers L. An online interprofessional learning resource for physicians, pharmacists, nurse practitioners, and nurses in long-term care: benefits, barriers, and lessons learned. Informatics for Health & Social Care. 2008;33(1): 21-38. [Context Link]


2. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: a meta-analysis. JAMA. 2008;300(10): 1181-1196. [Context Link]


3. Edwards G, Kitzmiller R, Breckenridge-Sproat S. Innovative health information technology training: exploring blended learning. CIN: Computers, Informatics, Nursing. 2012;30(2): 104-109. [Context Link]


4. Lahti M, Hatonen H, Valimaki M. Impact of e-learning on nurses' and student nurses' knowledge, skills, and satisfaction: a systematic review and meta-analysis. International Journal of Nursing Studies. 2014;51: 136-149. [Context Link]


5. Reis S, Sagi D, Eisenberg O, et al. The impact of residents' training in electronic medical record use on their competence: report of a pragmatic trial. Patient Education and Counseling. 2013;93(3): 515-521. [Context Link]


6. Wutoh R, Boren SA, Balas EA. eLearning: a review of Internet-based continuing medical education. Journal of Continuing Education in Health Professions. 2004;20-30. [Context Link]


7. Pipersburgh J. The push to increase the use of EHR technology by hospitals and physicians in the United States through the HITECH Act and the Medicare incentive program. Journal of Health Care Finance. 2011;38(2): 54-78. [Context Link]


8. Centers for Medicare & Medicaid Services. Electronic Health Records (EHR) Incentive Programs. 2017. Accessed September 11, 2017. [Context Link]


9. EHR Incentives and Certification. February 6, 2015. Accessed February 1, 2017. [Context Link]


10. Henry J, Pylypchuk Y, Searcy T, Patel V. Adoption of Electronic Health Record Systems Among U.S. Non-Federal Acute Care Hospitals: 2008-2015. Health IT Dashboard. May 2016. Accessed March 22, 2017. [Context Link]


11. The Joint Commission. Sentinel Event Alert, issue 42: safely implementing health information and converging technologies. December 11, 2008. Accessed September 13, 2017. [Context Link]


12. The Joint Commission. Sentinel Event Alert 54: safe use of health information technology. The Joint Commission: March 31, 2015. Accessed September 13, 2017. [Context Link]


13. Health Information Management Systems Society. Promoting Usability in Health Organizations: Initial Steps and Progress Toward a Healthcare Usability Maturity Model. Chicago, IL: HIMSS; 2011. [Context Link]


14. Staggers N, Elias B, Hunt J, Makar M, Alexander G. Nursing-centric technology and usability: a call to action. CIN: Computers, Informatics, Nursing. 2015;33(8): 325-332. [Context Link]


15. Atack L, Rankin J. A descriptive study of registered nurses' experiences with Web-based learning. Journal of Advanced Nursing. 2002;40(4): 457-465. [Context Link]


16. Moattari M, Moosavinasab E, Dabbaghmanesh M, ZarifSanaiey N. Validating a Web-based diabetes education program in continuing nursing education: knowledge and competency change and user perceptions on usability and quality. Journal of Diabetes and Metabolic Disorders. 2014;13: 70. [Context Link]


17. McCutcheon K, Lohan M, Traynor M, Martin D. A systematic review evaluating the impact of online or blended learning vs. face-to-face learning of clinical skills in undergraduate nurse education. Journal of Advanced Nursing. 2015;71(2): 255-270. [Context Link]


18. Curtis K, Wiseman T, Kennedy B, Kourouche S, Goldsmith H. Implementation and evaluation of a ward-based eLearning for trauma patient management. Journal of Trauma Nursing. 2016;23(1): 28-35. [Context Link]


19. Kazawa K, Moriyama M, Oka M, Takahashi S, Kawai M, Nakano M. Efficacy and usability of an eLearning program for fostering qualified disease management nurses. Health. 2015;7: 955-964. [Context Link]


20. Sinclair PM, Kable A, Levett-Jones T, Booth D. The effectiveness of Internet-based e-learning on clinician behaviour and patient outcomes-a systematic review. The International Journal of Nursing Science. 2016;57: 70-81. [Context Link]


21. Sung YH, Kwon IG, Ryu E. Blended learning on medication administration for new nurses: integration of e-learning and face-to-face instruction in the classroom. Nurse Education Today. 2008;28(8): 943-952. [Context Link]


22. Kontio R, Hatonen H, Joffe G, Pitkanen A, Lahti M, Valimaki M. Impact of eLearning course on nurses' professional competence in seclusion and restrain practices: 9-month follow up results of a randomized controlled study (ISRCTN32869644). Journal of Psychiatric and Mental Health Nursing. 2013;20: 411-418. [Context Link]


23. Berke W, Wiseman T. The eLearning answer: secure this education solution by setting a vision for its usage and building a sound business plan for its purpose. Nursing Management, IT Solutions. 2003;34(10): 26-29. [Context Link]


24. Sheen ST, Chang WY, Chen HL, Chao HL, Tseng CP. E-Learning education program for registered nurses: the experience of a teaching medical center. Journal of Nursing Research. 2008;16(3): 195-201. [Context Link]


25. Shih Y, Lee T, Liu C, Mills M. Evaluation of an online orientation program for new healthcare employees. CIN: Computers, Informatics, Nursing. 2013;31(7):343-350. [Context Link]


26. Wu JH, Tennyson RD, Hsia TL. A study of student satisfaction in a blended eLearning system environment. Computers and Education. 2010;55(1):155-164. [Context Link]


27. Chang H, Liu C, Hwang H. Exploring nursing e-learning systems success based on information system success mode. CIN: Computers, Informatics, Nursing. 2011;29(12): 741-747. [Context Link]


28. Tse M, Lo L, Chan M. The use of health technology and information: e-learning technological approach. Cyberpsychology and Behavior. 2007;10(6): 821-826. [Context Link]


29. Roach V, Lemasters L. Satisfaction with online learning: a comparative descriptive study. Journal of Interactive Online Learning. 2006;5(3):317-332. [Context Link]


30. Chen H. Linking employees' e-learning system use to their overall job outcomes: an empirical study based on the IS success model. Computers and Education. 2010;55: 1628-1639. [Context Link]


31. DeLone W, McLean E. Information systems success: the quest for the dependent variable. Information Systems Research. 1992;3(1): 60-95. [Context Link]


32. DeLone W, McLean E. The DeLone and McLean Model of Information Systems Success: a ten-year update. Journal of Management Information Systems. 2003;19(4): 9-30. [Context Link]


33. Freeze R, Alshare K, Lane P, Wen H. IS success model in e-learning context based on students' perceptions. Journal for Information Systems Education. 2014;21: 2. [Context Link]


34. Lee-Post A. e-Learning success model: an information systems perspective. Electronic Journal of e-Learning. 2009;7(1): 61-70. [Context Link]


35. Yengin I, Karahoca A, Karahoca D. E-Learning success model for instructors' satisfactions in perspective of interaction and usability outcomes. Procedia Computer Science. 2011: 1396-1403. [Context Link]


36. Lin H. Measuring online learning systems success: applying the updated DeLone and McLean model. Cyberpsychology and Behavior. 2007;10(6):817-820. [Context Link]


37. The Ohio State University Wexner Medical Center. Key Facts and Figures. Accessed September 3, 2017. [Context Link]


38. Willis K. Analysing qualitative data. In: Walter M, ed. Social Research Methods. South Melbourne, Australia: Oxford University Press; 2013: 315-336. [Context Link]


39. Stromberg SC. A training model for orienting newly hired nurses to an organization's electronic health record. CIN: Computers, Informatics, Nursing. 2011;29(6): 321-325. [Context Link]