Nursing professional development (NPD) practitioners are challenged by the question: "What is the return on investment (ROI) for the time and resources spent for professional development?" Although orientation, staff competence assessment, and educational response to critical events are essential, resources are often limited for educational activities. Either the team planning and providing the learning activities is restricted because of staffing, equipment, and supplies, or the clinical staff needing the learning activity is unable to leave the work area. This demonstrates the "push-me-pull-you" life of an NPD practitioner.
NPD practitioners are often the first to be called when a problem surfaces and the first to experience reductions when the budgets are tight. For the purposes of this discussion, the terms learning activities and continuing education of staff are used interchangeably to represent either what has traditionally been called "inservices" and "continuing education." These terms explicitly defined as different in the past are now generally accepted as ongoing development/education of staff.
Using the seven steps for the evidence-based practice (EBP) change process (Melnyk & Fineout-Overholt, 2015) as a framework, the authors evaluated best practices for ROI measurement for professional development activities as found in the literature today. These seven steps of EBP include (0) clinical inquiry, (1) population-intervention-comparison-outcome (PICO) question, (2) search for evidence, (3) critical appraisal of evidence, (4) integration into practice, (5) evaluation of outcomes, and (6) dissemination of results.
Step 0: Clinical Inquiry
The Association for Nursing Professional Development 2014 learning needs assessment of its membership identified great interest in determining best practices regarding impact evaluation and ROI for professional development activities. In response to this identified need, members of the Association for Nursing Professional Development Education Committee set out to identify these best practices. A task force was formed with a goal to clarify how ROI has been measured and used in decision-making regarding professional development activities based on the current available literature. Therefore, the clinical inquiry that motivated this project was to determine if routine ways to build measurement of ROI into the work of professional development already exist.
Kirkpatrick's levels of evaluation
Kirkpatrick and Kirkpatrick (2009) introduced the well-known educational program evaluation methodology consisting of four hierarchal levels for conducting evaluation in a business partnership approach. Kirkpatrick's levels of evaluation consist of reaction, learning, behavior, and results.
In Level 1, reaction, the lowest and easiest level of evaluation, the focus is on the learner's perceptions of the training (Kirkpatrick & Kirkpatrick, 2009). This level often uses surveys, questionnaires, or other rating scales for measurement (DeSilets, 2009). The next level, learning, measures the increase in knowledge, skill, or capacity due to the learning event. This level often uses posttests, checklists, or return demonstration for measurement. The third level, behavior, considers the extent of behavior change and the improvement achieved by application of the educational content. This level may be measured through observation of behavior, chart audits, or self-report of learners at a later date, after the content of the learning activity has had an opportunity to be applied in the work setting. Finally, the highest level of evaluation, results, considers the effect on the business or environment based on performance. This level is the most time-consuming to measure because it often takes a long time to actually see changes in outcomes as a result of the learning activity. Applying Kirkpatrick's four-level evaluation model builds a compelling chain of evidence for the value relationship between the learning activity and the organization's financial bottom line (Kirkpatrick & Kirkpatrick, 2007).
Kirkpatrick and Kirkpatrick (2009) state that formal educational activities alone do not deliver optimal bottom-line outcomes and results. The educational event is typically just one factor influencing the outcome. In addition to education, reminder systems, physical environment changes, unit culture, and being held accountable by leadership are examples of other influences on outcomes.
To account for additional system influences, Kirkpatrick and Kirkpatrick (2009) expanded their emphasis to focus on collective efforts to identify the return on stakeholder expectations. With the return on stakeholder expectations, outcome indicators are more holistic and focus on both qualitative and quantitative components of a variety of interventions. Using questions to clarify and refine, stakeholders are able to adapt expectations into measurable outcomes that meet the bottom line of the organization.
Phillips's ROI methodology
Phillips (2003) built upon the foundation of Kirkpatrick's model of evaluation by adding a fifth level, ROI. Phillips five-level model begins with reaction/satisfaction/planned action by measuring the learners' satisfaction and the plans to incorporate new learning into practice. Next, assessing the amount of information acquired describes the learning level. The third level, behavior/application/implementation, measures actual implementation into practice, whereas the fourth level, results, focuses on measuring the organizational impact by determining the output, quality, costs, and time in relation to the benefit/cost ratio. The last and most critical level is ROI, which evaluates the ultimate value with potential investment opportunities (Phillips & Phillips, 2002). Throughout the evaluation process, Phillips states that credibility depends on the isolation techniques and intangibles identified. The focus is on normalizing so that benefits and costs can be compared to all factors influencing measureable performance outcomes.
Paramoure's measurable instructional design
Paramoure (2013) presents a model of Measurable Instructional Design to facilitate measurement of ROI of education. This model begins with identification of the key performance metric (KPM) that is the target of an educational activity. Specific skills associated with the KPM are delineated and lead to the development of measureable performance outcomes. Teaching methods are created to align with these measureable objectives and ultimately lead to the evaluation of the learner's achievement of the objectives. Finally, application of new behaviors in the work setting is evaluated along with its impact on the KPM. Key components of the Measurable Instructional Design model include collaboration with managers to ensure that educational activities align with organizational and departmental goals and prospectively developing a "chain of evidence" (p. 36) to measure the impact of education. ROI is subsequently measured by quantifying the cost of education and the monetary value of the changes in the KPM.
Step 1: PICO Question
To guide the search for external evidence, the next step for EBP changes is to create a PICO question to inform the literature search (Melnyk & Fineout-Overholt, 2015). This project had a meaning-type PICO question, which does not incorporate the comparison component.
"How do healthcare organizations (Population) with effective professional development programs (Intervention) measure the return on investment for professional development activities (Outcome)?"
Step 2: Search for Evidence
After development of the PICO question, the search for evidence began. The keywords used to search for evidence in the literature included nurse educator, staff development, professional development, education, return on investment, cost-benefit analysis, impact, and educational outcome measurement. MEDLINE, CINAHL, and ERIC databases were searched from 2004 to 2014, followed by additional review of references from articles. From that further review, one more study from 2002 was identified and included in review (Zack et al., 2002).
To guide the search for evidence, inclusion criteria for articles were that the article must (a) identify an educational intervention for professionals, not patient educational activity; (b) describe the outcomes measured before and after the educational activity; and (c) provide a calculation for ROI. When few articles describing ROI were found, the search was broadened to include measurement of other financial impacts such as cost-effectiveness and benefit-cost analysis. A total of 69 articles were identified. Of these, 58 described educational impacts without including either financial analysis or outcomes measurement. The remaining 11 studies met the criteria for inclusion in the review of the literature.
Step 3: Critical Appraisal of Evidence
Once articles that met the inclusion criteria were identified, critical appraisal of the evidence began. Both qualitative and quantitative studies were identified during the literature search. The Melnyk and Fineout-Overholt (2015) rapid critical appraisal tools consider description of study; validity, reliability, and applicability of questions; and strength of the study design in appraising a study. The team discovered that commonly used evidence appraisal tools like Melnyk and Fineout-Overholt's (2015) Rapid Critical Appraisal and the AGREE II (Brouwers et al., 2010) only included a cursory analysis of financial considerations. In these instruments, cost is mentioned under applicability of the study but does not evaluate financial impact measurement. As a result, an article lacking financial data may receive a "good" rating using these appraisal tools whereas an article achieving a "fair" rating may contain essential financial data. This discrepancy caused the review team to realize that they must look beyond the scores or word ratings (e.g., excellent, good, fair, poor) in standard evidence appraisal tools to assess the true value of the article for this project's purpose. When completing the appraisals, additional notes were included as represented in Table 1.
Sample description
The review of literature found 11 studies between 2002 and 2013 that evaluated the financial impact of educational interventions. The educational interventions identified in these 11 studies were computer modules (1), newly licensed nurse residency or fellowship programs (4), customized orientation plan (1), blended learning approach orientation program (1), and multifaceted programs (4). The content of the educational interventions included topics such as medication safety, ventilator-associated pneumonia (VAP) prevention, pressure ulcer prevention, ergonomics, and a wellness program.
Educational activities comparison
In the studies reviewed, the measurement of the financial impact of professional development activities varied, including comparing cost of educational activities to the following:
1. cost of preventing an error resulting in litigation;
2. cost of avoiding expenses from error;
3. cost of employee turnover;
4. previous educational program costs and outcomes;
5. payments for injuries, personal days, and workers compensation; and
6. expense of employing agency nurses.
In these 11 studies, the calculation of financial impact measured efficiency and/or effectiveness of the educational interventions using a variety of calculations including cost-effectiveness analysis, benefit-cost ratio, and ROI.
Appraisal of evidence
Dennison (2007) calculated the cost savings including per error annual impact, length of stay, and litigation concluding that the minor expense of education was well worth the investment. Using a quasiexperimental design and providing detailed information of how calculations were performed contributed to the credibility of the conclusions.
Hillman and Foster (2011) calculated turnover cost savings over 4 years resulting from implementation of a newly licensed nurse residency program. Although significant cost savings due to reduced turnover was realized, this study was conducted in a children's hospital and should be replicated in other settings to ensure the generalizability of the findings.
Morris et al. (2009) evaluated program costs for a revised critical care orientation focused on critical thinking and competence validation. Outcomes measured included retention, vacancy rates, and staff satisfaction. The authors used a mixed-methods research design. The discussion of costs was limited, making it difficult to evaluate financial impact of the revised orientation.
Nelson et al. (2006) described benefit-cost comparisons by calculating injury prevention education and capital equipment costs, then comparing it to the treatment expenses, payments for injuries, personal days, and workers compensation savings. The authors provided detailed information, raising awareness of comprehensive cost analysis and validating their ROI.
Pine and Tart (2007) calculated total expenses, expenses per nurse resident, net program benefit, and cost savings of their Baccalaureate Nurse Residency Program. Their thorough ROI analysis provides a model for evaluating financial impact of professional development programming. A 13% reduction in turnover showed how a program that requires an initial financial outlay can result in long-term savings for the organization.
Ryan and Tatum (2013) used a Prerequisite Exam for Pediatrics (PREP) exam to individualize orientation. They calculated potential savings from the reduction of the average number of weeks for orientation paired with a reduced attrition rate at 1 year. This study had a small number of participants and used an estimated cost for calculations as opposed to actual costs, challenging the generalizability of the findings.
A premier example of a study that evaluated both the benefit-cost and ROI analysis is Sandhusen, Rusynko, and Wethington (2004). They calculated the cost savings of implementing a 16-month nursing fellowship program. Using the Phillips ROI model, key areas of cost reduction were identified. The savings justified the fellowship program intervention by reducing the costs of agency nurses.
Sendelbach, Zink, and Peterson (2011) described statewide reduction of pressure ulcers after a bundle intervention program was implemented in 10 different hospitals. They showed a wide variation in estimated cost savings, reducing the value of the study results.
Ulrich et al. (2010) used replacement cost of new RNs to calculate benefit-cost of the newly licensed nurse residency program over 10 years. Although this longitudinal study showed significant reduction of turnover, the authors did not offer actual costs and savings to delineate ROI.
Val Palumbo, Sikorski, and Liberty (2013) calculated the total program costs and percentage difference in unscheduled absence days over 3 years to demonstrate ROI as money saved from their multifaceted wellness program. This well-designed study provides a clear description of ROI.
Finally, Zack et al. (2002) estimated cost savings following an educational intervention on prevention of VAP. The financial impact was reported by calculating the potential savings resulting from the reduction in the number of VAP episodes. This is a significant study with broad clinical implications not only for the reduction of VAP in other institutions but also because the way the financial impact was reported could be used as a template for other educational interventions.
Synthesis of evidence
Synthesis of these 11 studies showed that describing the financial impact of professional development activities is more meaningful than simply measuring participant satisfaction in the current healthcare environment. However, a lack of consistent methods to describe financial and clinical impact of professional development activities was found.
Step 4: Integration Into Practice
In the current healthcare environment where all interventions are being evaluated for contributions to organizational outcomes, NPD practitioners must quantitatively demonstrate their value. Although this project was initiated to identify best practices related to measuring ROI of NPD activities, the lack of articles on ROI resulted in expanding the search to include other financial measures. As a result, cost per participant and benefit-cost analysis, were also found to be useful methods of demonstrating financial impact. The NPD practitioner can use all of these calculations to demonstrate the value for a variety of educational programs. Although time constraints prevent calculating financial impact of all educational interventions, realistically, considering stakeholders, values, and environmental factors, approximately 5%-10% of programing could be a goal for this analysis (DeSilets, 2010).
Although NPD practitioners may not have historically conveyed these financial impacts, routine reporting of the value of professional development activities to the organization's senior leadership is essential. Understanding the value of professional development activities guides senior leadership decisions in lean economic environments.
In addition, NPD practitioners need to publish data regarding financial impact of programs to advance the specialty practice of NPD. These publications can help others justify the need for and quantify the value of their own professional development activities.
Step 5: Evaluation of Outcomes
When implementing an EBP practice change, whether clinical or leadership focused, measuring the variables of interest before and after implementation is important. If positive outcomes are found, changes are supported. If negative outcomes are identified, problems with the interventions can be resolved. The outcomes measured in the 11 articles included in this review included employee behavior (e.g., retention, satisfaction, absenteeism) and/or patient care (e.g., number of infusion pump alerts, rate of VAP). Table 2 lists the 20 different outcomes reported in the literature reviewed. Overall, the financial outcomes measured in the reviewed articles showed that educational interventions yielded positive clinical and/or financial impacts.
Step 6: Dissemination of Results
This final step is essential for all EBP projects to minimize the resources consumed by repeating the work on projects already completed in other settings. Dissemination can be oral, written, or poster formats and may be internal (within the organization) as well as external (outside the organization) such as presentations at conferences and in professional journals. For this EBP project, one method of dissemination is this publication.
Conclusion
In this first part of the series on measuring ROI for professional development activities, a review of the literature was completed using the seven-step EBP process. Only 11 of 69 studies found included an analysis of financial impact of professional development activities. Furthermore, the reporting of financial impact was inconsistent and, in some cases, incomplete. In the current healthcare environment, NPD practitioners must demonstrate their financial impact to the organization to demonstrate their value. To facilitate this process, Part II of this series describes ways of calculating financial impact of educational activities including ROI. In addition, suggestions are provided for integrating financial impact measurement as an outcome of educational activities.
ACKNOWLEDGMENTS
Special thanks to the membership of Association for Nursing Professional Development for their requests for content on ROI as reported in the 2014 Annual Needs Assessment. Thanks also to the Association for Nursing Professional Development Education Committee for their support and guidance throughout this project.
References