Evidence-informed practice in nursing implies that decision making is continually informed by the critical and conscious use of the most recent research results, while considering patients' preferences and values, professional expertise, and available resources.1,2 Recent studies show that evidence-informed nursing practices can lead to significant improvements in healthcare.3-5

However, nurses seldom use scientific evidence in their practices.6-8 The assessment of scientific evidence currently poses particular challenges for nurses. A lack of education in regard to critical appraisal of the research as is the case with evidence-based practice has been identified as one of the most important barriers to the use of research findings in nursing practice.9-13

Furthermore, the time required and the attitude and motivation of nurses are other important barriers to assessing the scientific evidence.14,15 Lastly, nurses must be aware of potential limitations in published literature and develop skills for critical appraisal to determine whether to trust articles they read on a given health issue.16-18

In the current media-driven world, e-learning is increasingly used for professional training, especially in continuing education.19-22 However, very few structured interventions, such as e-learning courses for increasing evidence-informed nursing practice, have been evaluated, and their effectiveness remains uncertain.7

LITERATURE REVIEW

The potential of the Internet as an instructional means for the health professions is widely recognized.20,23-26 A comprehensive review by Cook et al27 summarizes the evidence on the effectiveness of Internet-based learning in the health professions. In two systematic reviews, which included 201 studies,27,28 they compared the effects of Internet-based learning to no intervention and to non-Internet interventions and found positive effects associated with Internet-based learning when compared with no intervention. Compared with non-Internet learning methods, effects were inconsistent across the studies and generally small, a finding that may be due to different learning contexts and objectives or to different methods of implementing a given Internet-based course. In another systematic review on Internet-based learning, Cook and colleagues28 found a wide range of approaches, notably as regards the configuration of courses, and also highlighted that the instructional methods encompassed in Internet-based learning varied considerably.

A recent systematic review15 assessed the effectiveness of Web-based distance education for nursing students and employed nurses. The results showed that Web-based distance learning produces equivalent or better effects in knowledge acquisition. This review also found that participants generally accept Web-based education with high satisfaction rates. According to a cross-sectional survey of healthcare professionals from six hospital systems in an integrated delivery network,29 the three key predictors of satisfaction with Web-based training in continuing education are instructional design effectiveness, Web site usability, and course usefulness.

With e-learning, the delivery and receipt of course content may be asynchronous, shifting the educational model from a passive, teacher-centered one to an active, learner-centred one.25 E-learning has many advantages: (1) it increases accessibility to educational materials, personalized instruction, and standardization of content; (2) it allows students to choose when to study, and is a good option for motivated individuals who work and want to study during their spare time; and (3) it helps users learn new concepts and consolidate their knowledge and skills, thus increasing the independence and motivation of students in different areas.14,27,30-36 A two-group randomized controlled study found that motivation and a teaching method using Internet-based modules had an interaction effect on knowledge acquisition by nursing students. Among less motivated students, those in the intervention group performed better than did those who received traditional training.14

Self-directed learning (SDL) is a central element in e-learning and is seen as a promising method for continuing education in medicine37 and nursing.38 It has been recommended to achieve effective and efficient training of health professionals, including nurses.39-41 However, a literature review by Murad and Varkey37 demonstrated that while SDL has been used in the education of a variety of healthcare professionals, there is little evidence showing the efficacy of individual aspects of SDL.37,42 Most studies have focused on evaluating the learner's acceptance and satisfaction with SDL rather than its impact on learning outcomes.37 Moreover, very few studies have been done in continuing nursing education.

Some authors have explored the concept of SDL readiness (SDLR), defined by Wiley in 1983 (as cited in Fisher et al in 2001)38 as "the degree [to which] the individual possesses the attitudes, abilities, and personality characteristics necessary for self-directed learning." Some scales have been developed to measure SDLR in educational and nursing research,43-45 including the Self-directed Learning Readiness Scale for Nursing Education (SDLRSNE) developed by Fisher et al.38 This scale contains 40 items to assess the three underlying factors of SDLR in nursing students. Recently, Fisher and King46 confirmed the factor structure of the subscales of the SDLRSNE, supporting the tool's validity.

The present study was conducted within a research project based on an international collaboration between Quebec (Canada) and the Basque Country (Spain). It aims to evaluate the effectiveness of a self-directed, Internet-based learning intervention on three outcomes: nurses' knowledge, satisfaction, and SDLR. The main research hypothesis is that online self-learning modules enable nurses to increase their readiness for self-learning. We also posit that nurses will show a high level of satisfaction with respect to online learning and that they will acquire knowledge in the critical reading of scientific articles.

METHODS

Ethical Approval

Approval for the study was obtained from the Ethics in Human Experimentation Committee of a university hospital center in Quebec, Canada. For the Spanish study, ethical approval was not required given that the study was an evaluation of a learning activity. Written informed consent was obtained from each participant. All the information related to the project was kept confidential; only members of the research team had access to it.

Design and Setting

Two organizations participated in this study, namely, a university hospital center from Quebec, Canada, and the Basque Country Public Health Service (Osakidetza) in Spain. In both cases, an evaluative study was conducted, using a quasi-experimental design including a pre-post study with a single group.

Intervention

The InfoCritique program was developed by Laval University (Quebec, Canada) to train nursing, occupational therapy, and medical students in the critical appraisal of scientific literature and assesses them based on tests focusing on the content of each module. It comprises self-learning online modules on critical appraisal skills for different health disciplines, including medicine and nursing. In 2004, six SDL modules aimed at graduate nursing students were developed. These online nursing modules were adapted from similar modules used with medical students and medical residents at Laval University.47 In 2011, four of these six modules were translated into Castilian Spanish in the context of this project.

The purpose of these online nursing modules is the acquisition of the skills required to efficiently select medical information and critically analyze scientific publications. Their specific objectives are (a) to use different electronic sources of information for access to sound medical information and for keeping up to date regarding scientific data and (b) to independently and critically assess the following types of scientific data: (1) systematic reviews and meta-analysis, (2) experimental studies, (3) nonexperimental studies as descriptive studies, (4) measurement and evaluation tests, (5) practice guidelines, and (6) qualitative studies.

Basque participants were asked to complete four modules (the practice guidelines and nonexperimental studies were not identified as a priority in their context and were not translated) and had a 3-month period to do so. In Quebec, participants were asked to complete all six modules, with a 5-month period to do so.

Sample and Recruitment

The target population was composed of nurses with a basic knowledge of research methodology from a university hospital center in Quebec, Canada, and from different areas of the Basque Health Service, Osakidetza. Nurses in all specialties and of any training background were included. The evaluation of the self-learning modules in the Basque Country took place in fall 2011. Participants were selected through various departments of nursing (primary care, specialized care, and mental health); the departments were asked to identify nurses who had a basic knowledge of research and who were interested in learning more about it and were willing to participate in the educational intervention. A convenience sample of 50 nurses was invited to take part in the study. Participation in the study was voluntary, and participants received continuing education credits for completing modules.

In Quebec, the goal was also to recruit 50 nurses working in a Centre Hospitalier Universitaire (CHU) in Quebec. The recruitment of Quebec nurses took place with the help of the Innovation Assistance Office (Bureau d'Aide a l'Innovation [BAI]) of the participating CHU. The head of the BAI invited members of the Nursing Care Administration (1) to identify nurses who could have an interest in the project and (2) to share information with the nurses of their work team. In addition to the nonprobability sampling by purposeful sampling method, the "snowball method" also proved effective with regard to participants, who encouraged their colleagues to take part in the study.

Participation in the study took place on a voluntary basis. Certain incentives were put in place to facilitate recruitment. In addition to the accreditation of training hours by the Ordre des infirmieres et infirmiers du Quebec (Quebec Nurses Corporation) and the acquisition of continuing education credits for the completion of modules, participants from Quebec received the equivalent of 2 days of work pay as compensation.

Measures

Before the course, nurses were asked to complete two questionnaires: the knowledge questionnaire and the SDLRSNE. After the course, they were asked to repeat the same two questionnaires and, in addition, to fill out a final satisfaction questionnaire. This questionnaire also included open-ended questions allowing participants to comment on different aspects of the modules.

1. Self-directed learning readiness: In this study, SDLR was measured using the 40 items of the SDLRSNE assessed on a five-point Likert scale with scores from strongly disagree to strongly agree. The SDLRSNE was specifically designed to measure SDLR in nursing students.38,46,48 The French version of the SDLRSNE scale was validated in a previous study14 following the steps proposed by Vallerand49 for transcultural validation. The French version of the SDLRSNE had high internal consistency, with a Cronbach's [alpha] of .94 and acceptable temporal stability with an intraclass correlation coefficient of 0.67. A Spanish version of the SDLRSNE was developed50 by researchers in the Basque Country for the purpose of this project, based on the same method for transcultural validation of instruments.49

2. Satisfaction: A questionnaire, consisting of 16 questions, was developed, first in Spanish and then translated into French, as inspired by 20 questions measuring satisfaction of Laval University (Quebec) students (using four-point Likert-type statements: 1 = totally agree to 4 = totally disagree). It was validated during the experimental research of Gagnon et al14 and demonstrated good internal consistency, with a Cronbach's [alpha] of .83.

3. Knowledge acquisition: A knowledge questionnaire (Table 1) with 20 questions (Basque Country) and 25 questions (Quebec) randomly selected from those found in the evaluation tests of the various self-learning modules (four modules for the Basque Country and six modules for Quebec) was developed. The questionnaire was pretested on nurses in training to ensure that the questions were clear and to determine the time required. Some sociodemographic data concerning the participants were also collected, that is, gender, age, year of obtention of first nursing diploma, level of education, number of hours devoted to research training, putting into practice of methodological research knowledge, and current occupation.

Table 1 Knowledge Evaluation Test (25 Questions for Quebec and 20 Questions for Basque Country)

Data Analysis

Descriptive analyses were carried out first. The effect of the control variables on the three outcomes was also explored. The McNemar test for qualitative variables and Student t test for paired samples, or the Wilcoxon tests, depending on whether data were normally distributed, were performed to assess the effects of online self-learning modules on knowledge, SDLR, and satisfaction. Normality was tested using the Shapiro-Wilk test. The confidence level was set at 95% (0.05). Additional analyses were also carried out to explore relationships between measured variables and characteristics related to participants and their experience. All statistical analyses were performed using the SAS statistical package (SAS, Cary, NC). Qualitative analyses were also conducted to supplement the documentation of the data collected and made it possible to consider areas for improvement both in terms of the training offered and the development of modules.

The reliability of the SDLRSNE was assessed using Cronbach's [alpha] both for the overall questionnaire and for its three subscales (self-management, desire for learning, and self-control). Finally, univariate general linear models were built to assess the association of some potentially confounding variables, such as previous training and computer skills, with changes in scores on the knowledge questionnaire before and after the modules and with satisfaction scores.

RESULTS

Sociodemographic Characteristics

Of the 58 nurses initially recruited in Quebec, only 36 completed the training, despite the incentives proposed for nurses to complete the modules (continuing education credits and wage compensation). The majority of participants (89%) were women, and more than half of participants (66%) were 40 years or younger. Also, 17% of participants had a college degree, and 17% a master's degree, whereas 66% of participants had a bachelor's degree. Of the 22 people who quit the training program, 41% were younger than 30 years, and 46% had a college degree, 18% a master's degree, and 36% a bachelor's degree. The reasons for quitting were lack of time, the difficulty of the content, and issues of interest (or motivation) concerning the reading of research articles. As a matter of fact, 55% of those who did not complete training initially claimed to have very little experience in the field of research, that is, fewer than 15 hours.

Of the 50 Basque nurses who started the modules, 46 were women (92%), and four were men, and more than a third of participants (38%) were between 41 and 50 years of age. All participants had an academic degree in nursing, and 84% were working in a clinical setting, 12% in management, and 4% in teaching. Three (6%) did not finish all the modules for personal or work-related reasons. These three participants were women between 50 and 54 years of age who reported in the precourse questionnaire that they had very little experience of research-related training.

Knowledge

The results obtained in Quebec with regard to the Knowledge Evaluation Questionnaire generally show that participants from Quebec improved their knowledge between the pretest and posttest for all modules (Table 2). Indeed, they were, on average, more successful after having completed the module corresponding to the evaluation test, with an average difference of 9.5%. However, for Module 4, Measurement and Evaluation Tests, we note a decrease in knowledge between the pretest and posttest. This module was considered the most difficult in both Quebec and the Basque Country.

Table 2 Percentage of Pretest and Posttest Success for Each Module (Quebec and Basque Country)

Given the paired nature of the samples, the McNemar test was used (Table 3A) to determine whether the training received resulted in a significant improvement in the participants' knowledge. Test results indicate that there was a statistically significant increase in knowledge (P = .0001) after the intervention for all modules. This increase was also significant for Module 2, Experimental Studies (P = .0295), and Module 3, Nonexperimental Studies (P < .0001), whereas there is no significant increase for other modules. The results of these tests are consistent with the data presented in Table 2, indicating a decrease in knowledge after the intervention for Module 4, Measurement and Evaluation Tests, a decrease that is not, however, significant.

Table 3 Difference in the Number of Incorrect and Correct Answers to Knowledge Questions Overall and for Each Module, Pretest and Posttest (Quebec [A] and Basque Country [B])

Subsequently, the possibility that there is a relationship between sociodemographic variables and the success rate of the training modules was examined. Initially, the Shapiro-Wilk test was performed to check whether the difference between preintervention and postintervention success stemmed from a normal distribution, which proved to be the case (P = .3009). Thus, a univariate generalized linear regression with each of the sociodemographic variables at our disposal for this study was conducted. The results then showed that the only variable with a significant influence on the success of the knowledge evaluation test after the intervention is the number of hours devoted to research training during the last 5 years.

Among nurses in the Basque Country, a significant change in the mean score (16.1%) of the knowledge questionnaire was found following the course (Table 2). After confirming that the postcourse scores were not normally distributed, the Wilcoxon test was used and the preintervention-postintervention change was found to be statistically significant (P < .001). The McNemar test was also used to assess whether the results of the knowledge questionnaire improved after the intervention in the Basque Country. As shown in Table 3B, the number of correct answers was significantly higher after the intervention, in general and for each individual module. Following the training program, results indicate a clear improvement in knowledge, but one question (question 11) continued to have a very low success rate (14.9%), which is not, however, statistically significant according to McNemar test (Table 4). In Quebec, the same question (corresponding to question 13) obtained the same success rate (50%) before and after training. This leads to reflection about the wording of the question and the theoretical content related to this question.

Table 4 Difference in Pretest and Posttest Response to Knowledge Questions (Quebec and Basque Country)

The questions for which there were statistically significant differences in the precourse and postcourse answers were precisely those in which there were the most errors in the initial questionnaire, that is, those with the greatest margin for improvement.

Self-directed Learning Readiness

Both in Quebec and in the Basque Country, the Cronbach's [alpha] for each of the 3 dimensions (self-management, desire for learning, and self-control) of the SDLRSNE exceeded .70 before and after the intervention. According to the Shapiro-Wilk test, the distribution of the self-learning score in Quebec was not normal. For this reason, we used the Wilcoxon signed rank test, according to which the difference in scores before and after the operation was not statistically significant to a threshold of 0.05.

In the Basque Country, the SDLRSNE precourse mean score was 157.92. The average postcourse score was 163.06. After confirming the normality of the data, the t test for paired samples was applied to the mean scores, with results indicating that the change was statistically significant (P < .001).

Satisfaction

Participants also filled out a satisfaction assessment questionnaire, evaluating, among other things, module content, participation, the online platform used, and the general level of satisfaction regarding the training program. As a general rule, participants from Quebec (83%) said they liked the training program, giving it an average rating of seven out of 10 for all six modules. For 90% of respondents, the content of modules seemed adequate or very adequate, mainly as regards Module 1, Systematic Reviews; Module 5, Practice Guidelines; and Module 6, Qualitative Studies. However, results indicate that Module 4, focusing on Measurement and Evaluation Tests, seems more difficult or less suitable for participants. The majority of participants (83.3%) would not eliminate any module, but half of them agreed that modifications should be made to Module 2, Experimental Studies; Module 3, Nonexperimental Studies; and Module 4, Measurement and Evaluation Tests. The main reasons given concerned the difficulty of the theoretical content. Participants indicated that they would like more examples related to the concepts presented.

For the opened questions, qualitative data analysis has also identified aspects that pleased and displeased participants during training. The main criticisms concern the selection of mainly English-language articles and the complexity of the concepts presented, considering the theoretical information provided and the basic knowledge of participants with respect to the research. On a more positive note, the pace of the training, the user-friendliness of the Web site used for training, the subject matter of the articles selected, and the feedback provided on the Web site via information capsules are elements that participants particularly appreciated.

Like participants from Quebec, those from the Basque Country gave an average grade of seven out of 10 as regards their satisfaction for all four modules. In general, participants reported good or very good levels of motivation and commitment. On the other hand, 25% of them recognized that they had not assimilated the content very well (low to moderate rating). In general, the four modules were considered to be adequate or very adequate in terms of content. Notably, however, a quarter of participants considered that the Measurement and Evaluation Tests module was not adequate.

When participants from the Basque Country were asked if they would eliminate a module, the majority (96%) of them answered no. Although few participants would add new modules, several areas, such as literature searches and types of experimental design, were mentioned as possible additional content. However, most of them (75%) agreed that changes could be made to the Measurement and Evaluation Tests module, in particular by adding more theoretical explanations. According to participants, the main difficulty with the training program was the lack of basic knowledge in the field of statistics. Conversely, the case histories presented in the modules appeared to be the most appreciated because of their usefulness in understanding the material taught. The most popular modules were Measurement and Evaluation Tests, despite its being the most difficult, and Qualitative Studies.

The most valued characteristics of the Web site included the fact that it was simple and user-friendly; it loaded rapidly; it was interactive; and it was well structured. On the other hand, the weakest aspects identified were the design, its perceived unattractiveness, and the lack of a space for interacting with tutors and other participants. The vast majority of participants thought that navigation of the Web site was easy and was no obstacle to the practical functioning of the course. Overall, there was a high level of satisfaction with the Web site.

DISCUSSION

This quasi-experimental study supports the use of self-directed educational modules for increasing nurses' knowledge and skills in relation to evidence-based practice, in both settings (Quebec and Basque Country). As shown in the literature,10,20,21,41 e-learning could be an effective approach for continual professional development for nurses. Du et al15 found in their systematic review that Web-based education has encouraging effects in improving participants' knowledge and skills performance and in enhancing self-efficacy in performing nursing skills, with a high satisfaction rate expressed by participants. But such educational interventions should consider not only increasing knowledge and skills but also how to transfer them into practice in order to have an impact on health outcomes in patients.39 These findings are consistent with the literature on e-learning and with other similar studies that have demonstrated that training in evidence-based practice is necessary.1,7,34,48

Two of the three hypotheses were statistically significant: (1) participants significantly improved their overall knowledge score after the educational intervention; and (2) they were, in general, satisfied with the course, giving it a rating of seven out of 10 for both Quebec and the Basque Country. A high satisfaction rate in Web-based education has also been identified in the systematic review of Du et al.15 However, participants in the Basque Country reported a greater readiness and ability for SDL, whereas this result was not significant in Quebec.

Participants from both countries commented on the usefulness of the course for clinical practice and professional development. These findings are consistent with the three key predictors of satisfaction with Web-based training in continuing education discussed in Atreja et al29: instructional design effectiveness, Web site usability, and course usefulness. According to our results, it seems to be difficult for nurses to put the acquired knowledge into practice, but some participants reported feeling that they were able to transfer this knowledge to colleagues.

This study also confirms that e-learning is an effective means for continuing training and provides greater independence with regard to the organization of learning time.20,22 The results add to knowledge on the effectiveness of different forms of e-learning for nurses in continuing education and also provide evidence to support the use of online self-learning modules to teach critical appraisal, as suggested by two systematic reviews, Cook et al28 and Du et al.15

Findings from this study could be transferable to similar nurse populations, but more research is needed to evaluate the effects of a self-directed course in other nurse populations. This research also contributes to the knowledge of e-learning in continuing education in nursing by providing a specific method for the evaluation of an Internet-based learning course.

LIMITATIONS

A first limitation to this study is the lack of control group and random assignment, making it impossible to determine the specific contribution of the online self-learning modules on critical appraisal skills. However, the pretest measures provide a comparative basis for some of the data. Furthermore, the fact that a very similar process was applied to evaluate online modules in the two countries offers an interesting way to assess the external validity of the intervention.

A second limitation is the small size of the groups, especially that of Quebec. A sample size of 50 was targeted in both settings. However, in Quebec, 38% of nurses withdrew from the study before completing all modules. For many of these nurses (32%), time constraints (or a lack of availability) caused by an excessive workload or by balancing work and studies are the causes of this withdrawal. Others (18%) reported personal reasons, whereas 50% of dropouts did not mention a reason. Unlike in the Basque Country, a minimum level of training was not required in Quebec to participate in the study. As such, because the concepts presented in the modules were largely at a university level, we anticipated that those who dropped out would mainly be holders of a junior college degree. However, no less than 64% of those who withdrew from the study before the end of the project held a university degree (50% with a bachelor's degree and 14% with a master's degree).

The important difference between the experience of the Basque Country and Quebec concerning participant withdrawals needs to be underscored. The fact that continuing education is mandatory in Spain for health professionals may well have had a positive effect on the participation rate. On the other hand, participants from Quebec had to complete two more modules compared with those from the Basque Country, albeit with 2 additional months to do so.

CONCLUSION

An online self-learning course for teaching critical appraisal skills and evidence-based practice, developed through international collaboration, was implemented and evaluated among practicing nurses in Quebec and the Basque Country. Our results provide support for the use of e-learning to increase nurses' knowledge of research methodology, as well as a critical approach to decision making based on scientific evidence. This educational intervention responds to the need for specific training in critical appraisal skills for nurses in order to emphasize and strengthen the relationship between research and their clinical practice and promote a research culture in nursing.

REFERENCES