1. Campbell-Yeo, Marsha PhD, MN, NNP-BC, RN
  2. Carrier, Leah RN
  3. Benoit, Britney PhD, RN
  4. Kim, Theresa PhD
  5. Bueno, Mariana PhD
  6. Rao, Megha MA
  7. Riahi, Shirine MN, RN
  8. Stevens, Bonnie PhD, RN, FAAN, FCAHS


Background: Electronic health (e-health) learning is a potential avenue to educate health professionals about accurately using infant pain assessment tools, although little is known about the impact of e-health interventions on clinical competence.


Purpose: To evaluate whether an e-health learning module for teaching the accurate use of the Premature Infant Pain Profile-Revised (PIPP-R) pain assessment tool results in immediate and sustained competency to assess infant pain.


Methods: Neonatal intensive care unit (NICU) nurses who participated in a larger study across 2 tertiary NICUs in Canada examining the implementation and clinical utility of the PIPP-R e-learning module completed 2 follow-up evaluations at 1 week and 3 months. Participants were asked to view a video recording of an infant undergoing a painful procedure and to assess the infant's pain intensity response using the PIPP-R measure. Immediate and sustained competency was assessed via interrater consensus of participant-reported PIPP-R scores compared with those of an experienced trained coder.


Results: Of the 25 eligible nurses, 22 completed 1-week and 3-month follow-up evaluations. At the 1-week follow-up, 84% of nurses scored the video accurately compared with 50% at 3 months. Behavioral pain indicators were more likely to be scored incorrectly than physiological indicators.


Implications for Practice: Follow-up training after completion of the initial e-learning module training may improve competency related to the clinical use of the PIPP-R tool to assess infant pain over time.


Implications for Research: Additional study regarding the need and timing of e-health training to optimize sustained competency in infant pain assessment is warranted.


Article Content


Incidence and Consequences of Untreated Neonatal Pain

Infants admitted to the neonatal intensive care unit (NICU) undergo multiple invasive and painful procedures during their routine medical care. A recent systematic review indicated that infants admitted to the NICU experienced between 7.5 and 17.3 painful procedures daily.1 The most recent audit conducted in 14 Canadian NICUs demonstrated that infants receive an average of 5.8 tissue-breaking procedures each week, ranging from 0 to 89 procedures.2 Only half of the 582 infants included in this Canadian study received a pharmacological or nonpharmacological intervention to relieve their pain.2 Furthermore, infants may experience repeated painful procedures or continuous pain with no or limited recovery time between the events.3 This is significant because early pain exposure in preterm infants is associated with immediate- and long-term negative consequences2,4-9 such as increased physiological responses to painful stimuli, blunted behavioral responses, and alterations in neural development.6,10-15


Importance of Accurate Pain Assessment to Prevent Untreated Neonatal Pain

In children and adults, pain is typically assessed through self-report measures. However, this is not possible in the preverbal infant population and health professionals must interpret pain levels by observing contextual, behavioral, and physiological indicators.3 One of the challenges in ensuring the optimal treatment of neonatal pain is the accurate use of assessment tools by healthcare professionals. There has been substantial research on developing pain assessment tools and their validity and reliability.3 However, the Premature Infant Pain Profile-Revised (PIPP-R) tool is the only pain assessment tool that has been evaluated in extremely low gestational age (ELGA) infants within the first days to weeks of age and hospitalization.3


Premature Infant Pain Profile-Revised

The PIPP-R is a composite measure of pain indicators in infants: behavioral (brow bulge, eye squeeze, and nasolabial furrow facial actions); physiological (heart rate and oxygen saturation); and contextual (gestational age and behavioral state).3,16 The 7 indicators are scored on the basis of a 4-point (0-3) Likert-type scale, with possible scores ranging from 0 to 21. Clinical cutoff scores indicate no pain (0) or mild (1-6), moderate (7-12), or severe (>12) acute biobehavioral pain responses. It is a valid and reliable measure for both preterm and term infants up to 44 weeks' postnatal age.3,16 To enhance clinical utility and to incorporate findings of its use with ELGA infants, the PIPP-R was revised in 2010 from the original Premature Infant Pain Profile (PIPP) tool developed by Stevens and colleagues in 1996.3,8,17 The PIPP-R includes the same 7 behavioral, physiological, and contextual pain indicators. Specific changes made to the PIPP-R included changes in the scoring of the oxygen saturation and facial grimacing indicators to improve the feasibility of the PIPP-R for clinical use, and revision of the order of scoring for the contextual indicators of behavioral state and gestational age. These indicators are modifying variables of pain responses rather than contextual variables that are inherent to the infant, thus minimizing the likelihood of overinflated scores based on baseline characteristics of the infant.16



Electronic Learning in the Health Professions

While there has been substantial research and validation of clinical tools used in neonatal pain, educating health professionals to accurately use these measures has been a barrier to pain prevention and treatment. e-Health technology has been identified as one potential avenue for teaching, where learners access the material online either individually or in synchronous courses with others.18-20 The benefits of electronic learning (e-learning) include flexibility,21-23 convenience,18,21,22,24 lower cost,22,23,25,26 and the ability to distribute the content widely.26 While many studies have reported high levels of satisfaction by users of e-learning modules,19,20,22,23,25,26 other authors have asserted the importance of examining knowledge uptake and skills acquisition following e-learning interventions.19,20 In a study that examined the effectiveness of e-learning on nursing assessment skills and knowledge, McDonald and colleagues27 suggested that e-learning alone does not exceed the benefits gained from face-to-face interactions. They recommended a scaffolding approach in which e-learning initiatives were integrated with traditional teaching methods.27


In a systematic review and meta-analysis of studies comparing e-learning interventions with no intervention among medical professionals, Cook and colleagues24 found that e-learning had a significant positive effect on learners' knowledge, skills, and behaviors compared with no educational intervention. In a subsequent review, they reported that feedback, repetition, and practice exercises improved the learning outcomes of participants who used an e-learning tool.21 Using Kirkpatrick's model as a guide,28 Lahti and colleagues19 conducted a systematic review examining the impact of nurses' and nursing students' knowledge, skills, and satisfaction following e-learning interventions. In their review of 11 studies, they found that the e-learning approaches were not statistically significant at improving the knowledge levels of participants when compared with traditional learning methods and that more research was needed.19 Although the body of evidence examining e-learning is growing, many authors have emphasized the need for more research examining the immediate knowledge uptake and long-term skills acquisition and competency following e-learning initiatives19,20,27,29-31 and whether e-learning modules result in a positive impact on patient outcomes.20,32


What This Study Adds


* The development of an effective e-health PIPP-R learning module for teaching pain assessment strategies is a promising avenue to improve competency related to the accuracy and reliability of pain assessments and improve the clinical outcomes of hospitalized infants.


* The use of an e-health learning module to educate nurses on how to use the PIPP-R assessment tool resulted in immediate improvement in competency to accurately assess infant pain, which suggests that e-health technology is effective for improving nurses' skills and competency in pain assessment in the NICU.


* Sustained infant pain assessment competency was lower than immediate competency following training. Further investigation regarding sustained competency and potential need and timing of training updates is warranted.



Thus, the aim of this study was to evaluate the immediate and sustained pain assessment competency of NICU staff nurses following completion of an e-learning module about the PIPP-R pain assessment tool. The results of this study will guide further refinements to the e-learning module and decisions made on how to optimally use the module as a training tool in clinical practice.




This descriptive repeated-measures study is a single-center follow-up study to a previous research project that was developed to evaluate implementation (ie, acceptability, feasibility, usability) and clinical utility of the PIPP-R e-learning module as assessed by 98 nurses from 2 tertiary NICU settings in Canada, The Hospital for Sick Children (SickKids) in Toronto, Ontario, and IWK Health in Halifax, Nova Scotia.33 The study received approval from the Quality Management Department at SickKids and the Research Ethics Board at IWK Health.


Preliminary Work Informing This Study

The larger previous study employed a user-centered design approach3 with 3 phases. In phase 1, between May and September 2015, feedback was collected from clinical and knowledge translation experts at a large tertiary care facility on the module content in terms of clinician learning needs and design preferences. Four experts reviewed the module during individual audio-recorded "think aloud" sessions.34 In phase 2, 22 NICU nurses reviewed the module and completed an 11-item survey on its feasibility, effectiveness, and potential clinical utility.


The initial 2 phases of the study produced a prototype of a self-administered e-learning module with clear prompts, navigation bars, immediate/real-time feedback, and engaging and user-friendly features such as diverse virtual characters, colorful visuals and illustrations, quizzes, and clinical case scenarios. Content included information on the similarities and differences between the original (PIPP) and revised (PIPP-R) versions of the measures, definitions, and illustrations of each behavioral and physiological pain indicator, and directions on how to use the PIPP-R measure to score pain in term, preterm, and ELGA infants. Phase 3 of the larger project sought to evaluate the latest e-learning module prototype in the clinical setting using a mixed-methods convergent approach and following quality improvement methods. The study was conducted at 2 tertiary centers, one in Ontario and the other in Atlantic Canada, with 50 staff nurses from the NICU participating from each site. Additional details on previously published findings are available in Bueno et al.33



Registered nurses who participated in phase 3 of the larger study described earlier were able to opt in or opt out, through provision of consent, to be contacted to participate in the current follow-up study. Consenting participants were asked to provide data on PIPP-R scoring at 1 week and 3 months following completion of the e-learning PIPP-R module. Nurses were eligible to participate if they were providing neonatal care at the IWK Health NICU, had greater than 6 months of NICU experience, could read and write English, and provided informed consent. The goal was to recruit a minimum of 25 nurses.



This article reports on data from nurses from IWK Health. IWK Health is a tertiary-level antenatal and pediatric referral center with a 45-bed NICU providing level 3-4 inborn and outborn care.


At IWK Health, the PIPP is the standard pain assessment measure used in the NICU. IWK nurses are educated on how to use the PIPP tool during their orientation training for the NICU. There are plans to switch to using the PIPP-R tool in the unit. There is significant interest in the potential of a self-administered e-learning module, accessible to nurses at any time, to standardize training and clinical application of the PIPP-R measure to improve pain assessment practices in the NICU.


Data Analysis

All data were stored in an encrypted secure file at the study center made only available to the research team. Data were analyzed using the Statistical Package for the Social Sciences (SPSS) 20 (IBM Corporation, Armonk, New York). Demographic data were analyzed using basic descriptive statistics. Categorical data were analyzed by reporting frequency counts and percentages. Differences in nurses' scores from the criterion were calculated for each pain indicator, and frequency counts were done to show which individual indicators were being scored incorrectly.



To determine immediate pain assessment competency acquisition, participants were required to view one of 6 possible vignette videos where a neonate was undergoing a heel lance within 1 week of completion of the e-learning module. Participants were then asked to assess and score the infant's pain level using the PIPP-R measure. This procedure was repeated 3 months following the initial 1-week assessment using an alternate video with a different neonate undergoing a heel lance procedure. All videos were previously scored by an independent, experienced, and reliable PIPP-R coder blinded to the study purpose or design. These pretest scores were considered to be criterion for the purpose of the study and used for comparison of participants' scores to determine reliability of score consensus. Consensus between the study participants and the criterion scores was obtained for both time points.


The vignette videos used for scoring pain intensity were obtained during a medically indicated heel lance procedure in the clinical setting. Parental consents were obtained for their use for training, education, and research. To minimize response bias in the scoring of the videos at study follow-up testing, the research assistant provided the case study, initiated the recorded film, and asked each participant to complete a PIPP-R assessment form, with no prompting or reinforcement from the research assistant. If a participant did not completely fill in the assessment form, the research assistant asked the participants to complete all sections. Only the research assistant and research coordinators had knowledge of participant responses, and the research assistant de-identified all responses prior to analysis.



Sample Demographics

Of the 25 nurses who consented to participate in the follow-up study, 25 completed the follow-up at 1 week and 22 completed the follow-up at 3 months. Participant demographics are reported in Table 1.

Table 1 - Click to enlarge in new windowTABLE 1. Participant Demographics


At the 1-week follow-up, a majority of nurses scored the video perfectly compared with half of the nurses at the 3-month follow-up. At 1 week, 16% of participant scores differed from the expert criterion scores: 12% by 1 point on the PIPP-R scale, and 4% differed by 2 or more points. At 3 months, 50% of participant scores differed from the expert criterion scores: 27% by 1 point on the PIPP-R scale, and 23% differed by 2 or more points. Of the participants whose scores differed at the 1-week follow-up, all of the participants scored the infant's pain lower than the criterion PIPP-R coder. While a greater number of participants underscored the infant's pain response at 3 months (27% vs 16%), the percentage of participants over scoring pain response (calculating a PIPP-R score higher than the criterion PIPP-R coder) was similar at 3 months (23%). See Table 2 for a complete description of differences in total PIPP-R scores over time.

Table 2 - Click to enlarge in new windowTABLE 2. Number of Raters With Interrater Exact Agreement in Total PIPP-R Score

When examining the 7 unique pain indicators of the PIPP-R, participants were able to accurately score the physiological indicators (heart rate, oxygen saturation, and gestational age) without error when compared with the criterion scores. In contrast, discrepancies in PIPP-R scores were mainly based on participants' low reliability in scoring the behavioral pain indicators. These included the infant's baseline behavioral state and 3 facial expressions: brow bulge, nasolabial furrow, and eye squeeze. See Table 3 for a description of behavioral pain component scores at 1-week and 3-month follow-ups.

Table 3 - Click to enlarge in new windowTABLE 3. Raters' Differences in Scores for Subjective PIPP-R Indicators


The aim of this study was to examine the impact of an electronic health (e-health) intervention on NICU staff nurses immediate and sustained calculation competency of PIPP-R scores and accurate assessment of infant pain. Competency to perform accurate pain assessment of infants undergoing a medically indicated tissue-breaking procedure was high immediately-1 week-following completion of the PIPP-R e-learning module.


The authors found that immediate knowledge acquisition and accuracy of total scoring were considerably higher at 1 week following completion of the e-learning module than sustained accuracy of pain scoring at 3 months, with the latter decreasing by almost one-third. This suggests that the e-learning module may not promote sustained competency for assessing pain using the PIPP-R. Further research is warranted with a more adequately powered sample to confirm and generalize these results.


Our findings are consistent with others examining the effectiveness of using e-learning to improve NICU care provider knowledge35 as well as online pain resources to potentially improve pain care.32 In a systematic review of 32 studies examining online pain resources, reporting on knowledge, skill acquisition, confidence/competence, and attitudes/beliefs, findings demonstrated that participants receiving online training had higher knowledge than those receiving training through traditional or usual training methods. Of note, while a majority of studies in this review reported knowledge to be higher immediately following online pain training, few studies assessed knowledge at a subsequent follow-up and findings were variable regarding the sustained impact of e-learning on knowledge. Of the 4 studies including follow-up, 2 reported that the participant's level of knowledge was sustained36,37; one reported knowledge scores remained higher than preintervention scores, but it was not clear if the level of sustained knowledge was significant38; and one reported significantly lower scores during a retention test.39 Of the 2 studies reporting on the impact of online resources on attitudes/beliefs and/or confidence/competence compared with those receiving training as usual/alternative training (health professionals and students), no significant differences were found. It is important to note that large variation in rigor, design, and outcomes limited conclusions from this review related to overall impact on patient receipt of care.32


While differences in overall pain scores were small, a decrease in one-third of the participants accurately scoring in a 3-month period would be a clinically meaningful finding, as inaccurate assessment of neonatal pain may lead to inadequate pain treatment, which has both immediate- and long term adverse consequences.40,41 Previous studies using the PIPP or PIPP-R have a 1-point difference in scores to evaluate potential differences in efficacy of pain-relieving interventions in superiority trials of multiple known pain-relieving interventions.42,43 A 2-point difference is considered a clinically meaningful difference when determining effectiveness of pain-reducing interventions when compared with no-treatment controls.44 These authors' findings provide support for the importance of ongoing continuing education of healthcare providers to ensure sustained competency related to infant pain assessment. To our knowledge, while there are no studies examining the immediate and sustained competency of users of different pain tools or other assessment measures in different contexts regarding sustained competency, our findings are consistent with others reporting a decline in sustained knowledge over time using both traditional and e-learning methods.20


When examining the specific components of the PIPP-R score, some components were more likely to be incorrectly scored than others. This was most pronounced at the 3-month follow-up time point compared with at 1 week. Nurses maintained near-perfect scoring on the PIPP-R physiological indicators such as heart rate, oxygen saturations, as well as gestational age at both 1 week and 3 months. However, there was more variability and more incorrect scores on the behavioral indicators such as facial responses of pain and infant behavioral state. These findings likely reflect the objective nature of the physiological findings obtained from monitors and the slightly more subtle behavioral changes that require greater ongoing focused observation of the infant and greater observer's judgment. Moreover, when coding behavioral responses, the nurse needs to provide an estimate of the length of time that the infant was exhibiting the behavioral responses. Therefore, there is a potential higher likelihood of error based on subjective evaluation of behavior and duration. Differences may also reflect other subconscious biases of acknowledging behavioral responses. Sensitivity to behavioral indicators could be lost over time, and there could be a need for continuing education on scoring those over time-this is consistent with literature that asserts that e-learning modules are more effective when coupled with evaluation and follow-up training.45 Given that behavioral indicators of pain have been reported to be the most sensitive with respect to infant pain response,46 the current findings highlight the importance of targeted efforts related to the accuracy of training related to these indicators.



While the current findings provide valuable knowledge to support future interventions aimed at using e-health technology to improve pain assessment and treatment by healthcare professionals, there are some limitations that must be noted. The unit from which study participants were recruited places significant emphasis on the importance of optimal newborn pain prevention and treatment and already uses the PIPP tool, which is the predecessor of the PIPP-R. Given this, nurses may have had preexisting knowledge and competency related to infant pain assessment that could have influenced the results. As this study is a subset of a larger research initiative with a small sample size and was only conducted in one study setting, it is difficult to make statistical conclusions about the effectiveness of the e-learning module or its generalizability. However, the preliminary results can inform future studies that evaluate the clinical effectiveness of using e-learning modules and inform curriculum development and timing of use for education related to neonatal pain assessment. It is also possible that using 2 different videos for scoring purposes at each time point impacted consensus across scores. Another limitation was that we did not measure the amount of exposure the nurses had using the PIPP-R tool between evaluation periods, which may have affected our findings. Finally, in keeping with any study examining outcomes over time in a nonsimultaneous way, it may be possible that a unit contextual factor varied between the 1-week and 3-month study time points. Further examination of the influence of contextual factors such as unit culture, leadership engagement, time, and resources would add to a better understanding of the effectiveness of e-learning tools in the future.




1. Cruz MD, Fernandes AM, Oliveira CR. Epidemiology of painful procedures performed in neonates: a systematic review of observational studies. Eur J Pain (Lond Engl). 2016;20(4):489-498. doi:10.1002/ejp.757. [Context Link]


2. Johnston C, Barrington KJ, Taddio A, Carbajal R, Filion F. Pain in Canadian NICUs: have we improved over the past 12 years? Clin J Pain. 2011;27(3):225-232. doi:10.1097/AJP.0b013e3181fe14cf. [Context Link]


3. Stevens BJ, Gibbins S, Yamada J, et al The Premature Infant Pain Profile-Revised (PIPP-R): initial validation and feasibility. Clin J Pain. 2014;30(3):238-243. doi:10.1097/AJP.0b013e3182906aed. [Context Link]


4. Andrews K, Fitzgerald M. Cutaneous flexion reflex in human neonates: a quantitative study of threshold and stimulus-response characteristics after single and repeated stimuli. Dev Med Child Neurol. 1999;41(10):696-703. doi:10.1017/s0012162299001425. [Context Link]


5. Bartocci M, Bergqvist LL, Lagercrantz H, Anand KJS. Pain activates cortical areas in the preterm newborn brain. Pain. 2006;122(1/2):109-117. doi:10.1016/j.pain.2006.01.015. [Context Link]


6. Grunau RE, Holsti L, Peters JWB. Long-term consequences of pain in human neonates. Semin Fetal Neonatal Med. 2006;11(4):268-275. doi:10.1016/j.siny.2006.02.007. [Context Link]


7. Stevens BJ, Johnston CC, Horton L. Factors that influence the behavioral pain responses of premature infants. Pain. 1994;59(1):101-109. doi:10.1016/0304-3959(94)90053-1. [Context Link]


8. Stevens B, Johnston C, Taddio A, Gibbins S, Yamada J. The Premature Infant Pain Profile: evaluation 13 years after development. Clin J Pain. 2010;26(9):813-830. doi:10.1097/AJP.0b013e3181ed1070. [Context Link]


9. Anand KJS, Eriksson M, Boyle EM, et al Assessment of continuous pain in newborns admitted to NICUs in 18 European countries. Acta Paediatr. 2017;106(8):1248-1259. doi:10.1111/apa.13810. [Context Link]


10. Fitzgerald M, Millard C, McIntosh N. Cutaneous hypersensitivity following peripheral tissue damage in newborn infants and its reversal with topical anaesthesia. Pain. 1989;39(1):31-36. doi:10.1016/0304-3959(89)90172-3. [Context Link]


11. Fitzgerald M, Walker SM. Infant pain management: a developmental neurobiological approach. Nat Clin Pract Neurol. 2009;5(1):35-50. doi:10.1038/ncpneuro0984. [Context Link]


12. Grunau RE, Holsti L, Haley DW, et al Neonatal procedural pain exposure predicts lower cortisol and behavioral reactivity in preterm infants in the NICU. Pain. 2005;113(3):293-300. doi:10.1016/j.pain.2004.10.020. [Context Link]


13. Brummelte S, Grunau RE, Chau V, et al Procedural pain and brain development in premature newborns. Ann Neurol. 2012;71(3):385-396. doi:10.1002/ana.22267. [Context Link]


14. Ranger M, Chau CMY, Garg A, et al Neonatal pain-related stress predicts cortical thickness at age 7 years in children born very preterm. PLoS One. 2013;8(10):e76702. doi:10.1371/journal.pone.0076702. [Context Link]


15. Vinall J, Grunau RE. Impact of repeated procedural pain-related stress in infants born very preterm. Pediatr Res. 2014;75(5):584-587. doi:10.1038/pr.2014.16. [Context Link]


16. Gibbins S, Stevens BJ, Yamada J, et al Validation of the Premature Infant Pain Profile-Revised (PIPP-R). Early Hum Dev. 2014;90(4):189-193. doi:10.1016/j.earlhumdev.2014.01.005. [Context Link]


17. Stevens B, Johnston C, Petryshen P, Taddio A. Premature Infant Pain Profile: development and initial validation. Clin J Pain. 1996;12(1):13-22. doi:10.1097/00002508-199603000-00004. [Context Link]


18. Belcher J, Vonderhaar K. Web-delivered research-based nursing staff education for seeking Magnet status. J Nurs Adm. 2005;35(9):382-386. doi:10.1097/00005110-200509000-00004. [Context Link]


19. Lahti M, Hatonen H, Valimaki M. Impact of e-learning on nurses' and student nurses knowledge, skills, and satisfaction: a systematic review and meta-analysis. Int J Nurs Stud. 2014;51(1):136-149. doi:10.1016/j.ijnurstu.2012.12.017. [Context Link]


20. Sinclair PM, Kable A, Levett-Jones T, Booth D. The effectiveness of Internet-based e-learning on clinician behaviour and patient outcomes: a systematic review. Int J Nurs Stud. 2016;57:70-81. doi:10.1016/j.ijnurstu.2016.01.011. [Context Link]


21. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in Internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med J Assoc Am Med Coll. 2010;85(5):909-922. doi:10.1097/ACM.0b013e3181d6c319. [Context Link]


22. Wutoh R, Boren SA, Balas EA. eLearning: a review of Internet-based continuing medical education. J Contin Educ Health Prof. 2004;24(1):20-30. doi:10.1002/chp.1340240105. [Context Link]


23. Smith CM. e-Orientation: a cyber approach to orienting per diem and temporary nurses. J Nurses Staff Dev. 2005;21(5):204-212; quiz 213-214. doi:10.1097/00124645-200509000-00004. [Context Link]


24. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: a meta-analysis. JAMA. 2008;300(10):1181-1196. doi:10.1001/jama.300.10.1181. [Context Link]


25. Ward R, Stevens C, Brentnall P, Briddon J. The attitudes of health care staff to information technology: a comprehensive review of the research literature. Health Inf Libr J. 2008;25(2):81-97. doi:10.1111/j.1471-1842.2008.00777.x. [Context Link]


26. Sumak B, Hericko M, Pusnik M. A meta-analysis of e-learning technology acceptance: the role of user types and e-learning technology types. Comput Hum Behav. 2011;27(6):2067-2077. doi:10.1016/j.chb.2011.08.005. [Context Link]


27. McDonald EW, Boulton JL, Davis JL. e-Learning and nursing assessment skills and knowledge-an integrative review. Nurse Educ Today. 2018;66:166-174. doi:10.1016/j.nedt.2018.03.011. [Context Link]


28. Galloway D. Evaluating distance delivery and e-learning is Kirkpatrick's model relevant? Perform Improv. 2005;44:21-27. doi:10.1002/pfi.4140440407. [Context Link]


29. Curran VR, Fleet L. A review of evaluation outcomes of Web-based continuing medical education. Med Educ. 2005;39(6):561-567. doi:10.1111/j.1365-2929.2005.02173.x. [Context Link]


30. Curran VR, Fleet LJ, Kirby F. A comparative evaluation of the effect of Internet-based CME delivery format on satisfaction, knowledge and confidence. BMC Med Educ. 2010;10(1):10. doi:10.1186/1472-6920-10-10. [Context Link]


31. Voutilainen A, Saaranen T, Sormunen M. Conventional vs. e-learning in nursing education: a systematic review and meta-analysis. Nurse Educ Today. 2017;50:97-103. doi:10.1016/j.nedt.2016.12.020. [Context Link]


32. Liossi C, Failo A, Schoth DE, Williams G, Howard RF. The effectiveness of online pain resources for health professionals: a systematic review with subset meta-analysis of educational intervention studies. Pain. 2018;159(4):631-643. doi:10.1097/j.pain.0000000000001146. [Context Link]


33. Bueno M, Stevens B, Rao M, et al Implementation and evaluation of the Premature Infant Pain Profile-Revised (PIPP-R) e-learning module for assessing pain in infants. Clin J Pain. 2021;37(5):372-378. [Context Link]


34. Stevens B, Dinsdale S, Promislow S, Yamada J, Dionne K. Evaluation of the Premature Infant Pain Profile: revised eLearning module for enhancing clinical practice and research. Council of International Neonatal Nurses 9th Conference 2016. Vancouver, BC, Canada; August 14-17, 2016. [Context Link]


35. Poon WB, Tagamolila V, Toh YP, Cheng ZR. Integrated approach to e-learning enhanced both subjective and objective knowledge of aEEG in a neonatal intensive care unit. Singapore Med J. 2015;56(3):150-156. doi:10.11622/smedj.2015041. [Context Link]


36. Harris JM, Elliott TE, Davis BE, Chabal C, Fulginiti JV, Fine PG. Educating generalist physicians about chronic pain: live experts and online education can provide durable benefits. Pain Med (Malden, Mass). 2008;9(5):555-563. doi:10.1111/j.1526-4637.2007.00399.x. [Context Link]


37. Trudeau KJ, Hildebrand C, Garg P, Chiauzzi E, Zacharoff KL. A randomized controlled trial of the effects of online pain management education on primary care providers. Pain Med. 2017;18(4):680-692. doi:10.1093/pm/pnw271. [Context Link]


38. Pelayo-Alvarez M, Perez-Hoyos S, Agra-Varela Y. Clinical effectiveness of online training in palliative care of primary care physicians. J Palliat Med. 2013;16(10):1188-1196. doi:10.1089/jpm.2013.0005. [Context Link]


39. Jordan J, Jalali A, Clarke S, Dyne P, Spector T, Coates W. Asynchronous vs didactic education: it's too early to throw in the towel on tradition. BMC Med Educ. 2013;13:105. doi:10.1186/1472-6920-13-105. [Context Link]


40. Campbell-Yeo ML. Reaffirming that every poke counts! Higher repeated pain exposure in early life linked with greater short- and long-lasting alteration of the nociceptive system. Pediatr Res. 2020;87(1):15-16. doi:10.1038/s41390-019-0565-7. [Context Link]


41. Eriksson M, Campbell-Yeo M. Assessment of pain in newborn infants. Semin Fetal Neonatal Med. 2019;24(4):101003. doi:10.1016/j.siny.2019.04.003. [Context Link]


42. Campbell-Yeo M, Johnston C, Benoit B, et al Trial of repeated analgesia with Kangaroo Mother Care (TRAKC Trial). BMC Pediatr. 2013;13:182. doi:10.1186/1471-2431-13-182. [Context Link]


43. Stevens B, Yamada J, Campbell-Yeo M, et al The minimally effective dose of sucrose for procedural pain relief in neonates: a randomized controlled trial. BMC Pediatr. 2018;18(1):85. doi:10.1186/s12887-018-1026-x. [Context Link]


44. Johnston CC, Filion F, Campbell-Yeo M, et al Kangaroo mother care diminishes pain from heel lance in very preterm neonates: a crossover trial. BMC Pediatr. 2008;8:13. doi:10.1186/1471-2431-8-13. [Context Link]


45. Rholdon R, Lemoine J, Templet T, Stueben F. Effects of implementing a simulation-learning based training using a train-the-trainer model on the acquisition and retention of knowledge about infant safe sleep practices among licensed nurses. J Pediatr Nurs. 2020;55:224-231. doi:10.1016/j.pedn.2020.08.014. [Context Link]


46. Pillai Riddell R, Fitzgerald M, Slater R, Stevens B, Johnston C, Campbell-Yeo M. Using only behaviours to assess infant pain: a painful compromise? Pain. 2016;157(8):1579-1580. doi:10.1097/j.pain.0000000000000598. [Context Link]


For more than 135 additional nursing continuing professional development activities related to Neonatal topics, go to


e-health technology; e-learning; infant; neonatal intensive care unit; neonate; pain assessment; procedural pain