Digital learning, Higher education, Intervention study, Scoping review, Technological outcomes



  1. Sormunen, Marjorita PhD, RN
  2. Saaranen, Terhi PhD, RN, PHN
  3. Heikkila, Asta PhD, RN
  4. Sjogren, Tuulikki PhD, PT
  5. Koskinen, Camilla PhD, RN
  6. Mikkonen, Kristina PhD, RN
  7. Kaariainen, Maria PhD, PHN
  8. Koivula, Meeri PhD
  9. Salminen, Leena PhD, RN, PHN


This scoping review was undertaken to synthesize and describe research related to digital learning interventions in higher education, focusing on technological outcomes. Five electronic databases were searched, and 86 articles were included in the review. The data related to positive and negative technological outcomes and authors' suggestions were analyzed using inductive content analysis. The articles represented six disciplines across six continents and included quantitative (n = 65), qualitative (n = 3), and mixed-methods (n = 18) intervention studies. For positive technological outcomes, digital formats of learning were considered effective and participatory forms of learning in a majority of the articles. The students appreciated individualized and self-paced learning, and the digital form increased their motivation to learn. Automatized technical solutions that enabled learning and teaching had several advantages, and digital learning was believed to save the resources of students, teachers, and organizations. For negative technological outcomes, the technical difficulties in using the digital devices or platforms were described the most, and a need for resources was identified. Feedback from teachers was considered important from positive and negative viewpoints. Authors' suggestions for future digital teaching and learning as well as related interventions consisted of various activities, resources, environments, and methods.


Article Content

A worldwide consensus has emerged regarding the benefits that can be brought to education through the appropriate use of evolving digitalization, including information and communication technologies. The range of possible benefits covers practically all areas of activity in which knowledge and communication play a critical role, from improving teaching and learning processes to better learning outcomes.1,2 Clearly, the way digitalization is used will depend on the subject being taught, the learning objectives, and the nature of the students.


The use of new technologies in education implies new teacher roles, new pedagogies, and new approaches to teacher education. The successful integration of digital technologies in the classroom depends on the teacher's ability to structure the learning environment to merge new technology with pedagogy and develop socially active classrooms, encouraging cooperative interaction, collaborative learning, and group work.3 These goals require a different set of classroom management skills, including the ability to develop innovative ways of using technology to enhance the learning environment and encourage technology literacy, knowledge deepening, and knowledge creation.


Digital competence has been defined as one of the eight key competencies for lifelong learning. The Council of the European Union4 recommends member states develop digital competence by promoting and taking advantage of appropriate digital learning strategies and environments in education. Digitalization rebuilds education, learning, and research and requires development of new skills from teachers. For example, recent studies among healthcare faculty have shown that the use of digital technology in teaching to ensure ongoing information exchange and to maintain contact between faculty and students poses a challenge.5 The need for changes in pedagogical practices, such as the use of digital technology, is highlighted. Digipedagogical competence requires pedagogical competence, the knowledge to utilize digital technology in teaching and learning, use of digital technology, and consideration of student-centered learning during digitalization without forgetting the ethical issues.


With digital technology, it is possible to improve efficiency in education and offer students better possibilities for distance and lifelong learning.6 The use of technologies, such as e-learning, computer-assisted learning, Web-based applications, virtual reality, and augmented reality, has increased learner satisfaction7,8 and motivation8,9 and developed students' problem-solving10 and critical-thinking skills7 better than conventional education methods11,12 even as it has been said slow and laborious work for teachers.9,13 Quite often, faculty feel a lack of confidence using new technologies in their teaching.14


The European Commission has published the Digital Education Action Plan,15 which covers schools, vocational education, and higher education. It defines how education can make better use of digital technology and how digital competencies can be supported. The report suggests supporting digital readiness by strengthening schools' digital capacities.15 Many studies have been conducted showing how to use various technologies in teaching and learning, but the consensus and summary of those studies are missing. Moreover, we lack summarized research knowledge about the technology-related aspects or conclusions of digital interventions, which we address in this review.



The aim of this review is to synthesize and describe published research related to digital learning interventions in higher education, referring to their technological outcomes. The focus is on technology-related aspects or conclusions that the study participants (students) and their teachers have considered beyond the learning outcomes. The questions that guided the review process were: (1) What are the reported positive and negative technological outcomes of digital learning interventions in higher education? and (2) What suggestions and recommendations did the authors of the reviewed studies present regarding the future development of digital teaching and learning, as well as related interventions?



The scoping literature review was chosen because it is used to identify the nature and extent of research evidence16 and the gaps in the existing research literature.16,17 The methodology for this scoping review was based on the framework outlined by Arksey and O'Malley,17 which comprises five key phases: (1) identifying the research question; (2) identifying relevant studies; (3) selecting studies; (4) charting the data; and (5) collating, summarizing, and reporting the results. A descriptive summary and inductive content analysis were used to summarize and disseminate the findings. This review is intended to draw attention to the comprehensive coverage of the available literature and therefore does not exclude studies of lower scientific quality, as it is intended to provide an account of available research.17,18



Articles were identified by searching five electronic databases: CINAHL (EBSCO); MEDLINE/PubMed; Eric (ProQuest); Scopus; and a Finnish database, Medic. A university librarian was consulted for searches. The searches were conducted on May 2018, including all articles from the beginning of 2015. The following similar search terms were used across databases: digital, mobile, virtual, games, social media, Web-based, Internet, information technolog*, ICT, communication technolog*, technology uses in education, educational technology, E-learning, electronic learning, online courses, teach*, learn*, education, higher education, universities, postsecondary education, interve*, teaching method, and teaching technique. Two searches were conducted in four international databases, the first including the search term "intervention" and the second including the search term "teaching method." In the Medic database, the search terms were digital, mobile, virtual games, social media, Web-based, Internet, information technology, ICT, communication technology, technology uses in education, educational technology, e-learning, electronic learning, eoppiminen, digitaal*, virtuaal*, mobiil*, opetus, teach*, learn*, education, and koulutu*. No additional manual search was executed because the number of existing articles was large. Criteria for inclusion and exclusion were established (Table 1).

Table 1 - Click to enlarge in new windowTable 1 Search Criteria

Search Outcome

The initial search yielded 1252 titles, which were imported to the bibliography management tool RefWorks (Ex Libris, Jerusalem, Israel), and duplicate entries were eliminated, leaving 1159 titles. Based on the inclusion and exclusion criteria, two authors (M.S. and A.H.) scrutinized the titles, abstracts, and content to reduce the number of articles to 132. After the full-text articles were screened, 91 were accepted, of which five additional articles were excluded with consensus discussions between four authors (M.S., A.H., L.S., T.S.). Eighty-six articles were eventually included in the review (Figure 1).19

Figure 1 - Click to enlarge in new windowFIGURE 1. A flowchart of the process of study selection (PRISMA [Preferred Reporting Items for Systematic Reviews and Meta-analyses]).

Data Analysis

To obtain an overview of the selected articles, they were summarized in a data extraction matrix according to the first author, year, country, study purpose, academic discipline(s), main digital means of intervention, methodology or design, and data collection method(s). For the first research question about the reported technological outcomes of digital learning interventions, the outcomes were first divided into two categories: positive and negative. Then an inductive content analysis was conducted in the two categories. For the second research question, the authors' suggestions and recommendations for future interventions from a technological viewpoint were searched, mainly in the discussion, limitations, and conclusion sections, and sentences including the words and phrases "should," "need," "recommend," "to meet the needs," "can be used," and "in the future" were searched for and categorized inductively.



Characteristics of the Included Studies

Several characteristics of the studies (n = 86) included in this review were categorized and presented (see Supplemental Digital Content 1,, including the study purpose, academic disciplines, main digital means of intervention, methodology and design, and data collection method(s). The studies were conducted in 30 countries on six continents (Asia n = 32, North America n = 24, Europe n = 18, Australia n = 4, Africa n = 2, and South America n = 3). Three studies were conducted in two countries and continents (Sweden and Hong Kong, United States and Lebanon, UK and Somaliland). Quantitative (n = 65), qualitative (n = 3), and mixed-methods (n = 18) studies were included, with various designs.


The studies represented six academic disciplines: health sciences (n = 44), social sciences (n = 13), humanities (n = 12), engineering and technical sciences (n = 12), physical sciences (n = 2), and life sciences (n = 2). One article included social sciences and health sciences (n = 1). The participants were students from these disciplines. Intervention duration varied from several hours to one academic year, although it was not reported in all articles, and in some articles, it was unclearly described. The categorization of technological outcomes, including authors' suggestions, is presented in Figure 2.

Figure 2 - Click to enlarge in new windowFIGURE 2. The categorization of results.

Positive Technological Outcomes

Positive outcomes were sorted into seven categories: (1) effective and participatory form of learning, (2) individualized and self-paced learning, (3) motivation to learn, (4) resource saving, (5) satisfaction with the technological method, (6) automatized technical solutions enabling learning and teaching, and (7) enabling a wide learning environment. As a most often mentioned outcome, digital formats of learning were seen as an effective and participatory form of learning.20-52 Although learning in a virtual environment can be time consuming, the engagement and effectiveness were reported as high.28,33,40 Improved confidence,27,41,46 reduced anxiety,41,46 enhanced active participation21,24 and problem-solving capacity,42 augmented analogical reasoning,20 and stimulated information retrieval skills23 were mentioned as benefits of digitally derived education. Digital devices allowing for synchronous and asynchronous interaction21,22,35 and multiple opportunities to use text, audio files, and videos22,26 or to have discussion environments integrated with contents,35 as well as the game's repeatability,50 were seen as advantages. Having the possibility to contribute to co-construction of knowledge with a teacher or with peers was expressed positively.21,22 as was the possibility of sharing material with others.31 Furthermore, learning clinical topics and complex treatment protocols29 and the ability to use learned skills in various future situations23,25,51 in a clinical setting during the studies40 were emphasized. A digital method combined with traditional learning was suggested as useful in reinforcing learning.38,45,47 The digital method's effectiveness in professional training of students26 and its ability to encourage the students to take risks (such as in practicing a foreign language) while learning the issue27 were highlighted. A platform supporting interactivity also was reported to help the students share their thoughts and exchange ideas, even for those not used to doing so publicly,39 and digital technologies were seen as diminishing differences between the students, thereby increasing equality in learning.21


The students mentioned individualized and self-paced learning often.32,34,36,39,41,52-62 It gave them access to the relevant information outside the allocated teaching period41,52,53,55,57-62 and unrelated to the place,32,41,55,56,58,61,62 thus increasing their autonomy.34,58 In case of individual learning-based problems, the mechanism of the tool in identifying problems through statistics was considered helpful.36 Students also valued meeting the various needs and preferences that the digital forms of learning made possible.32,54,59


The use of digital resources enabled a wide learning environment.23,36,59,63-65 This finding was manifested, for example, in the educational module's strength in providing authentic clinical scenarios65 and presenting real-world problems23 or using videos, which gave learners access to additional material.59 Moreover, a digital tool complementing theory material was considered beneficial.36 The fact that a small mobile device can contain persuasive technology for educational interventions was considered innovative, flexible, and promising.63 Some interventions contained a variety of online classroom strategies, providing a large set of methods.64 Based on the students' feedback and teachers' observations, the digital method used provided students a new way of learning and increased their motivation to learn.23,28,30,32,33,36,38,41,42,46,49,60,63,65-68 They said learning was more exciting, paid more attention to instruction, and were more interested in the class.30 The students were even reported to show extra involvement in their learning due to their high interest in game-based learning.28


Automatized technical solutions enabling learning and teaching and instruction being integrated into the content54 had clear advantages,31,33,36,41,53-55,62,65 as the students, for example, mentioned the benefit of immediate or provided timely feedback,31,36,53,55,65 and teachers emphasized the advantages of the automatic-correction technique.36 Standardized content was also considered beneficial, minimizing the variation in teaching.62 In general, students indicated their satisfaction with the technological method of use.29,35-37,44,51,52,58,63,69-72 The satisfaction was expressed regarding usefulness in learning,36,44,51,70,71 general user friendliness, and ease in using the tool.37,44


Digital learning was also seen as a way to save resources,24,32,33,36,45,48,50,52,55,70,73-75 as some learning events could be arranged in a seminar room instead of simulation facilities,50 or using free24 or relatively low-cost48 digital applications for learning. In many cases, digital learning allowed students living far from campus to study from home32,45 or from their home countries rather than traveling abroad.74 Also, for example, students listened to audio files during their free time or when traveling.52


From the teachers' perspective, some digital learning platforms planned short and standardized, reducing the running time and lecture preparation.75 Teachers' reduced time in monitoring and correcting students' problems was considered important because the digital tool monitored the students' progress continuously.36 From the educational organization's perspective, running costs and contributions to savings of revenue were identified,32,73 as was the relatively small number of faculty members teaching.70 Also, one digital device's smaller cost in comparison to another's was noticed.33


Negative Technological Outcomes

Outcomes that represented negative aspects were sorted into five categories: (1) technical problems, (2) need for resources, (3) degree of difficulty, (4) insufficient instructions and feedback, and (5) undesirable structure and design. Technical problems were mentioned most often as a disturbing factor in learning interventions.51,67,73,74,76-83 The quality of the sound,73,76,82 small screen size,78 or other quality problems in visual presentation73,82 were reported, as were poor Internet connections67,74,77,79 or lack of Internet access outside the university campus.79 Managing multiple devices was reported as difficult,81 as was using all the digital environments' components.83 Moreover, a need for resources was reported.28,38,51,52,83-85 In addition to the labor and financial resources that are needed to develop a digital application or e-learning material,38,85 virtual teaching requires time and effort.52 Time is also needed for learning,28 as well as time and support to enhance knowledge and skills necessary to use the digital platform before its implementation.83 Some digital applications require ongoing Internet connectivity or a fast or at least sufficient Internet connection.51,84


The learning module's degree of difficulty42,51,80,86-88 was manifested in various ways. Regarding games, the students can become uninterested due to low demand for or inconveniences in playing the game87 or experience insufficient and too simple feedback from the online tool.80 On the other hand, the system can also be too complex, requiring lot of learning before one can use the tool.51 Moreover, the game can turn out to be insufficiently realistic to create a sense of immersion,42 and as the authors indicated, the games created by specialists can restrict the students' creativity, or the students are able to memorize the game's content.86 In general, the students may feel that the module is too overwhelming and not adequately targeted to them.88Insufficient instructions and feedback30,58,88 from instructors was reported; learners hoped for more support from teachers58 and appropriate feedback,58,88 as well as more guidance in the learning platform.30 Eventually, the students were discouraged from participating by the learning platform's undesirable structure and design30,65,88; some students also indicated their own preferences in using mobiles, such as viewing landscape versus portrait views or using another user interface.30


Suggestions and Recommendations From Authors of the Reviewed Studies

Based on tested interventions, the reviewed articles' authors gave their suggestions or presented recommendations for future digital teaching and learning and related interventions. They were divided into five categories: (1) learning environments; (2) learning/teaching techniques, applications, methods, and devices; (3) teacher activities; (4) learner resources and characteristics; and (5) duration and timing of intervention. Suggestions related to learning environments36,45,52,83,89 varied. The importance of an authentic learning context (eg, a museum) was highlighted,83 but a fully equipped laboratory for teaching language was considered unnecessary.52 Authors also pointed out that using only the game to teach demanding tasks is difficult,36 indicating the need for more versatile learning environments. Furthermore, authors recommended the use of technological platforms that students already use,89 along with paying attention to collaborative aspects of interaction in mobile learning environments.83 Last, the virtual learning environment was recommended to train students based at remotely located institutions.45


Concerning learning/teaching techniques, applications, methods, and devices,23,25,29,34,41,42,47,48,52,56,79,83,84,86-94 the importance of selecting an appropriate tool or method for a certain group of students was mentioned.52 Furthermore, assessing the impact of mobile learning tests on student performance89 and adding more features in assessment90 were proposed. Future developments for the tested application or platform included adding transitional stages between components86; increasing attractiveness, using an access monitoring system, and assessing individual performance87; including listening,34 practice exercises,91 or cases29 in the platform; adding repetitive features and feedback,42,91 interactivity,91 or more intervention characters94; developing additional video recordings47 or free access to them41; and adding voice recognition48 or multimedia.29 Moreover, a combination of several applications or content-learning methods was suggested,42,94 as was using multiple types of applications for the students to learn before they entered the workforce.89 The authors also emphasized the importance of the developmental phase of application; for example, it is important to pay attention to instructional design and testing, as well as how the learner is guided effectively to the content and given meaningful feedback.88 Improving the ICT infrastructure79 and high-quality Internet connections84 were recommended. Some comments were related to gathering more learner-related analytics data.23,83,92 Some suggestions were contradictory. For example, some suggested that faculty should be cautious when selecting an application to make sure it can be used to meet the students' learning objectives,56 and others stated that concern over learning outcomes alone should not dissuade educators from employing games,25 implying that digital methods can also help enhance skills besides the stated learning outcomes. According to Millis et al,93 it is essential to know how the game-like elements contribute or detract from learning, because student engagement in the learning activities is important.


Teacher activities32,37,40,52,53,59,62,72,79,80,82,83,88,95-99 contained several suggestions, such as capacity building for lecturers to adopt a blended learning approach79 or devoting time to learning and gaining experience in running webinar sessions.82 The authors emphasized that the benefits do not depend only on the technology but on how well it is used to promote reflection in the learners,37 and they strongly recommended teachers redesign their courses and teaching methods when moving from traditional to digital presentation methods to achieve satisfactory learning effects.97 A teaching model for using certain applications was recommended.52 Many suggestions concerned the guidance or feedback,32,40,53,72 and advice for setting up virtual office hours where students could ask for teacher assistance was given.59 Besides feedback, technical support and content expertise were also considered important.80,88 A virtual interaction to enhance reflection with the expert teacher was suggested during the student field trips.83 Authors presented some advice for the selection of contents for educators96 and made remarks related to a lack of quality criteria for the application (app).95 Authors also suggested that educators meet students' needs by choosing multiple learning strategies or methods to teach them (medical) content,62,98 allowing customization of learning, thereby increasing students' motivation,53 or having Web-based learning available as on-demand.99


Learner resources were also emphasized.73,91,94,100-102 Fleming et al100 mentioned that it should be kept in mind that not all economically disadvantaged students have digital resources off campus. Similarly, Carlson et al73 brought up financial issues and family considerations but favored ICT solutions to help with internationalization at home. Adjusting teaching methods to serve students' needs and profiles was considered important in some articles,91,94,101 and paying attention to students' possibly inadequate computing skills when they enter universities was brought out.102


Park et al103 brought up the need for sufficient time designated for intervention, stating that the extension for the education period and repeated application would be needed in the future. Similarly, Lee et al41 pointed out the duration of intervention to improve the students' skills and knowledge competencies and suggested more measurement during the intervention, such as finding out how many times or how frequently students watch the video and the frequency with which it affects the outcomes. Time between the performance test and end of intervention was considered important,42 as was the experiment's duration.80 Hence, duration and timing of intervention and its activities41,42,80,103,104 were considered important factors in the future. Last, Thalluri and Penman105 presented a checklist for the use of Facebook for learning and teaching, examining the application from a variety of aforementioned perspectives.



The findings in this study contribute to the body of knowledge about digital interventions that are conducted among a variety of disciplines in higher education. The number of included studies (n = 86) was large, and their geographical area broad, implying that many interventions are being conducted globally and the need to find alternative or complementary technology-based methods or digital devices for already established teaching is evident. The digital learning interventions contained a variety of content, resulting in a mixture of digital methods, which were implemented in a various innovative ways. Based on the results, the positive outcomes outnumbered the negative ones, and the authors presented plenty of suggestions and recommendations based on their experiences with digital learning interventions.


Several articles reported the perceived effects of digitally enriched teaching. Digital learning methods encouraged students to take risks and enabled them to repeat the learning content to increase their skills and knowledge. Practicing communication or clinical skills with virtual patients or rehearsing challenging situations using a serious game can complement theory-based learning. However, we might ask whether combining multiple digital methods to obtain learning objectives is effective from the viewpoint of resources from teachers and students, as we can learn from these articles.


One of the most visible advantages of using digital platforms was the possibility to have control of one's own learning. The independence of the time and place of studying is essentially important to students who live far from their campuses but was also valued by other students for giving them the freedom to study when it best suited them. Using mobile devices in learning while traveling or during free time shows goal orientation and can eventually even influence learning outcomes. One interesting feature from the viewpoint of students and teachers was the automatized technical solutions that were built into applications or platforms. They allowed immediate feedback for students and provided teachers real-time information on students' performance. Although these articles do not go deeper into learning analytics, it is useful to know that applications are able to produce information about students' processes and are an evolving area of practice.106 The question is, are teachers skilled and motivated to use this information and how can they use it to benefit learners? Therefore, educational organizations are encouraged to offer digipedagogical support to teachers to gain full benefit from learning analytics to further develop digital learning and teaching processes.


Negative outcomes, although far fewer than positive ones, are still important for educators to reflect on. Technical difficulties, which were the most criticized in included articles, are very common and stem from variety of reasons. Irrespective of their origin, they caused students to have decreased learning experiences or, in worst cases, no learning experience at all if, for example, the Internet connection did not work. Such problems can decrease the motivation to study or cause feelings of inequality. Not only the students are harmed by inoperative devices or connections; every teacher knows the feeling of being unable to help students in need, and, in many cases, they have to give students a replacement task (and read it) afterward. When the synchronic learning situation requires student interaction, as described in many of these articles, or even live international collaboration of students, that experience cannot be replaced easily. Another important reminder for educators is the students' need for sufficient feedback and support in digital learning environments. These interventions tested digital means of learning, and it is assumed that the teachers were active in promoting learning. Hence, the students' complaints of insufficient instructions and feedback were fairly low.


The authors of these reviewed studies highlighted multiple, important issues that are good to keep in mind when the teaching is digitally modified. These issues concerned a wide range of factors from environmental opportunities to learner resources. Even though most suggestions were related to the methods or related technologies, the teacher's role and responsibility were highly emphasized. Thus, a competent teacher, who has technological and pedagogical skills,14 is always key in facilitating student learning processes, implying that up-to-date knowledge and skills must be a requirement for educators at all levels.107


Almost 10 years ago, UNESCO3 emphasized the use of new technologies in education. According to the results of this review, it can be said that this aim was fulfilled. A vast number of interventions have been conducted presenting varying integrations of digital technologies into teaching. Almerich et al14 disclosed that teachers feel a lack of confidence when using new technologies in their teaching. According to this review, however, we can say that teachers are willing to test new technologies.



As the number of the included review articles was substantial, a quality appraisal was not conducted. As Arksey and O'Malley17 point out, it is not a "must" in scoping reviews. A positive outcome of not setting strict criteria can increase the variety of methodologies and range of study designs, which allows for a more comprehensive review of the digital intervention literature. The gray literature was not searched due to the large volume of selected studies.


Second, although this review did not specifically favor health-related interventions, more than half of the studies (n = 44) were from the area of health sciences. Thus, the outcome of the review was dependent on the databases selected for this review, of which three were related to health (CINAHL, PubMed, Medic). Searching additional or more varied databases might have identified additional relevant studies and balanced the variety of disciplines. However, as the aim of the review did not include deeper analysis of content, examining digital learning interventions from the point of presenting an overview of the existing literature was deemed feasible.


Third, this review communicates the self-reported technological outcomes of students and teachers described in the digital learning intervention articles. The exploration of the learning outcomes was left out to be reported at later phase. Therefore, a comprehensive synthesis of the literature reviewed was not a goal. The balance between breadth and depth of analysis was decided among the authors.


Fourth, the technological outcomes and authors' suggestions and implications for future digital learning interventions from a technological viewpoint may overlap slightly, as they sometimes represented fairly similar contents. In addition, authors of reviewed articles may have proposed further insights or recommendations for readers, but all of them were not interpreted as suggestions related to future digital learning interventions or from a technological viewpoint. Furthermore, very broad and general suggestions-or very detailed ones-examining specific technological solutions were not exported from the articles. If the suggestions were more related to learning outcomes, they were omitted, as were suggestions related to study design or research methodologies.


Last, although this review included 30 countries in six continents, it is worth noting that educational organizations throughout the world have significant variations in their technological capacities and infrastructure. Therefore, these findings are relevant in educational environments where appropriate digital resources for teaching and learning are available.



Higher education faculty are in a key position in educating future professionals who are expected to master digital environments as skillfully as other competencies in today's demanding and constantly changing workforce. The results of this scoping review reflect the importance of systematic digital learning intervention planning and provide information to educators and other professionals who are looking for effective and student-friendly ways to modify their teaching based on evidence. Overall, the main aim for educators to modify their teaching methods or carry out new technical solutions is to help students gain deeper understanding, improve skills, increase knowledge, remain motivated to learn, gain awareness, and learn to think critically.



The authors thank the Ministry of Education and Culture, Finland, for funding the study.




1. Hoffmann RL, Dudjak LA. From onsite to online: lessons learned from faculty pioneers. Journal of Professional Nursing. 2012;28(4): 255-258. doi:. [Context Link]


2. van Laar E, van Deaursen AJAM, van Dijk JAGM, de Haan J. The relation between 21st-century skills and digital skills: a systematic literature review. Computers in Human Behavior. 2017;72: 577-588. doi:. [Context Link]


3. UNESCO. UNESCO ICT Competency Framework for Teachers. Paris, France: United Nations Educational, Scientific and Cultural Organization; 2011: Accessed March 14, 2019. [Context Link]


4. Council recommendation. 2018: The Council of the European Union. Council recommendation of 22 May 2018 on key competences for lifelong learning. Official Journal of the European Union. Council Recommendations C189. Published June 4, 2018. Accessed February 26, 2019. [Context Link]


5. Zlatanovic T, Havnes A, Mausethagen S. A research review of nurse Teachers' competencies. Vocations and Learning. 2017;10(2): 201-233. doi:. [Context Link]


6. Minedu. 2019: Ministry of Education and Culture. Digital learning environments in higher education institutions. Accessed February 26, 2019. [Context Link]


7. Lee H, Parsons D, Kwon G, et al. Cooperation begins: encouraging critical thinking skills through cooperative reciprocity using a mobile learning game. Computers in Education. 2016;97: 97-115. doi:. [Context Link]


8. Bacca J, Baldiris S, Fabregat R, et al. Augmented reality trends in education: a systematic review of research and applications. Journal of Educational Technology & Society. 2014;17(4): 133-149. [Context Link]


9. Stott A, Mozer M. Connecting learners online: challenges and issues for nurse education-is there a way forward?Nurse Education Today. 2016;39: 152-154. doi:. [Context Link]


10. Scherer R, Gustafsson J-E. The relations among openness, perseverance, and performance in creative problem solving: a substantive-methodological approach. Thinking Skills and Creativity. 2015;18: 4-17. doi:. [Context Link]


11. Bloomfield JG, While AE, Roberts JD. Using computer assisted learning for clinical skills education in nursing: integrative review. Journal of Advanced Nursing. 2008;63(3): 222-235. doi:. [Context Link]


12. Byrne AJ, Pugsley L, Hashem MA. Review of comparative studies of clinical skills training. Medical Teacher. 2008;30(8): 764-767. doi:. [Context Link]


13. Arthurs J. A juggling act in the classroom: managing different learning styles. Teaching and Learning in Nursing. 2007;2: 2-7. doi:. [Context Link]


14. Almerich G, Orellana N, Suarez-Rodriguez J, Diaz-Garcia I. Teachers' information and communication technology competences: a structural approach. Computers in Education. 2016;100: 110-125. doi:. [Context Link]


15. COM. 2018: European Commission. Communication from the commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions on the digital education action plan. 22 final. Published January 17, 2018. Accessed February 26, 2019. [Context Link]


16. Pham MT, Rajic A, Greig JD, Sargeant JM, Papadopoulos A, McEwen SA. A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Research Synthesis Methods. 2014;5(4): 371-385. doi:. [Context Link]


17. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005;8(1): 19-32. doi:. [Context Link]


18. Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implementation Science. 2010;5: 69. doi:. [Context Link]


19. Moher D, Liberati A, Tetzlaff J, Altman DGPRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA Statement. PLoS Medicine. 2009;6(7): e1000097. doi:. [Context Link]


20. Achuthan K, Francis SP, Diwakar S. Augmented reflective learning and knowledge retention perceived among students in classrooms involving virtual laboratories. Education and Information Technologies. 2017;22(6): 2825-2855. doi:. [Context Link]


21. Blau I, Shamir-Inbal T. Digital technologies for promoting "student voice" and co-creating learning experience in an academic course. Instructional Science. 2018;46(2): 315-336. doi:. [Context Link]


22. Bozoglan H, Gok D. Effect of mobile-assisted dialect awareness training on the dialect attitudes of prospective English language teachers. Journal of Multilingual and Multicultural Development. 2017;38(9): 772-787. doi:. [Context Link]


23. Evens M, Larmuseau C, Dewaele K, Van Craesbeek L, Elen J, Depaepe F. The effects of a systematically designed online learning environment on preservice teachers' professional knowledge. Journal of Digital Learning in Teacher Education. 2017;33(3): 103-113. doi:. [Context Link]


24. SFESA Fattah. The effectiveness of using WhatsApp messenger as one of mobile learning techniques to develop students' writing skills. Journal of Education and Practice. 2015;6(32): 115-127. [Context Link]


25. Franciosi SJ. The effect of computer game-based learning on FL vocabulary transferability. Journal of Educational Technology & Society. 2017;20(1): 123-133. [Context Link]


26. Gartmeier M, Bauer J, Fischer MR, et al. Fostering professional communication skills of future physicians and teachers: effects of e-learning with video cases and role-play. Instructional Science. 2015;43(4): 443-462. doi:. [Context Link]


27. Gokturk N. Examining the effectiveness of digital video recordings on oral performance of EFL learners. Teaching English With Technology. 2016;16(2): 71-96. [Context Link]


28. Ijaz K, Bogdanovych A, Trescak T. Virtual worlds vs books and videos in history education. Interactive Learning Environments. 2017;25(7): 904-929. doi:. [Context Link]


29. Leung JYC, Critchley LAH, Yung ALK, Kumta SM. Evidence of virtual patients as a facilitative learning tool on an anesthesia course. Advances in Health Sciences Education: Theory and Practice. 2015;20(4): 885-901. doi:. [Context Link]


30. O'Bannon BW, Skolits GJ, Lubke JK. The influence of digital interactive textbook instruction on student learning preferences, outcomes, and motivation. Journal of Research on Technology in Education. 2017;49(3). doi:. [Context Link]


31. Oner D, Adadan E. Are integrated portfolio systems the answer? An evaluation of a Web-based portfolio system to improve preservice teachers' reflective thinking skills. Journal of Computing in Higher Education. 2016;28(2): 236-260. doi:. [Context Link]


32. Pellas N, Kazanidis I. On the value of second life for students' engagement in blended and online courses: a comparative study from the higher education in Greece. Education and Information Technologies (Dordrecht). 2015;20(3): 445-466. doi:. [Context Link]


33. Remon J, Sebastian V, Romero E, Arauzo J. Effect of using smartphones as clickers and tablets as digital whiteboards on students' engagement and learning. Active Learning in Higher Education. 2017;18(2): 173-187. doi:. [Context Link]


34. Wang YH. Integrating self-paced mobile learning into language instruction: impact on reading comprehension and learner satisfaction. Interactive Learning Environments. 2017;25(3): 397-411. doi:. [Context Link]


35. Yilmaz EO, Yurdugul H. Design and effects of a concept focused discussion environment in e-learning. Eurasian Journal of Educational Research. 2016;(63): 353-374. doi:. [Context Link]


36. Boada I, Rodriguez-Benitez A, Garcia-Gonzalez JM, Olivet J, Carreras V, Sbert M. Using a serious game to complement CPR instruction in a nurse faculty. Computer Methods and Programs in Biomedicine. 2015;122(2): 282-291. doi:. [Context Link]


37. Fernandez-Aleman JL, Lopez-Gonzalez L, Gonzalez-Sequeros O, Jayne C, Lopez-Jimenez JJ, Toval A. The evaluation of i-SIDRA-a tool for intelligent feedback-in a course on the anatomy of the locomotor system. International Journal of Medical Informatics. 2016;94: 172-181. doi:. [Context Link]


38. Schneider AT, Albers P, Muller-Mattheis V. E-learning in urology: implementation of the learning and teaching platform CASUS(R)-do virtual patients lead to improved learning outcomes? A randomized study among students. Urologia Internationalis. 2015;94(4): 412-418. doi:. [Context Link]


39. Chao SY, Chang YC, Yang SC, Clark MJ. Development, implementation, and effects of an integrated Web-based teaching model in a nursing ethics course. Nurse Education Today. 2017;55: 31-37. doi:. [Context Link]


40. Choi M, Lee H, Park JH. Effects of using mobile device-based academic electronic medical records for clinical practicum by undergraduate nursing students: a quasi-experimental study. Nurse Education Today. 2018;61: 112-119. doi:. [Context Link]


41. Lee NJ, Chae SM, Kim H, Lee JH, Min HJ, Park DE. Mobile-based video learning outcomes in clinical nursing skill education: a randomized controlled trial. Computers, Informatics, Nursing. 2016;34(1): 8-16. doi:. [Context Link]


42. Tan AJQ, Lee CCS, Lin PY, et al. Designing and evaluating the effectiveness of a serious game for safe administration of blood transfusion: a randomized controlled trial. Nurse Education Today. 2017;55: 38-44. doi:. [Context Link]


43. Azman AM, Esteb JJ. A coin-flipping analogy and Web app for teaching spin-spin splitting in 1H NMR spectroscopy. Journal of Chemical Education. 2016;93(8): 1478-1482. doi:. [Context Link]


44. Chingos MM, Griffiths RJ, Mulhern C. Can low-cost online summer math programs improve student preparation for college-level math? Evidence from randomized experiments at three universities. Journal of Research on Educational Effectiveness. 2017;10(4): 794-816. doi:. [Context Link]


45. Agrawal N, Kumar S, Balasubramaniam SM, et al. Effectiveness of virtual classroom training in improving the knowledge and key maternal neonatal health skills of general nurse midwifery students in Bihar, India: a pre- and post-intervention study. Nurse Education Today. 2016;36: 293-297. doi:. [Context Link]


46. Dankbaar ME, Richters O, Kalkman CJ, et al. Comparative effectiveness of a serious game and an e-module to support patient safety knowledge and awareness. BMC Medical Education. 2017;17(1, 30): 1-10. doi:. [Context Link]


47. Fernandez-Lao C, Cantarero-Villanueva I, Galiano-Castillo N, Caro-Moran E, Diaz-Rodriguez L, Arroyo-Morales M. The effectiveness of a mobile application for the development of palpation and ultrasound imaging skills to supplement the traditional learning of physiotherapy students. BMC Medical Education. 2016;16(1): 274. doi:. [Context Link]


48. Foster A, Chaudhary N, Kim T, et al. Using virtual patients to teach empathy: a randomized controlled study to enhance medical students' empathic communication. Simulation in Healthcare. 2016;11(3): 181-189. doi:. [Context Link]


49. Kim H, Suh EE. The effects of an interactive nursing skills mobile application on nursing students' knowledge, self-efficacy, and skills performance: a randomized controlled trial. Asian Nursing Research. 2018;12(1): 17-25. doi:. [Context Link]


50. Kow AWC, Ang BLS, Chong CS, Tan WB, Menon KR. Innovative patient safety curriculum using iPAD game (PASSED) improved patient safety concepts in undergraduate medical students. World Journal of Surgery. 2016;40(11): 2571-2580. doi:. [Context Link]


51. Pusponegoro HD, Soebadi A, Surya R. Web-based versus conventional training for medical students on infant gross motor screening. Telemedicine Journal and E-Health. 2015;21(12): 992-997. doi:. [Context Link]


52. Shih R-C. The effect of English for specific purposes (ESP) learning-language lab versus mobile-assisted learning. International Journal of Distance Education Technologies. 2017;15(3): 15-30. doi:. [Context Link]


53. Davidson SJ, Candy L. Teaching EBP using game-based learning: improving the student experience. Worldviews on Evidence-Based Nursing. 2016;13(4): 285-293. doi:. [Context Link]


54. Swart R. Critical thinking instruction and technology enhanced learning from the student perspective: a mixed methods research study. Nurse Education in Practice. 2017;23: 30-39. doi:. [Context Link]


55. Avci H, Adiguzel T. A case study on mobile-blended collaborative learning in an English as a foreign language (EFL) context. International Review of Research in Open and Distributed Learning. 2017;18(7): 45-58. [Context Link]


56. Diliberto-Macaluso K, Hughes A. The use of mobile apps to enhance student learning in introduction to psychology. Teaching of Psychology. 2016;43(1): 48-52. doi:. [Context Link]


57. Thomas RL, Fellowes MDE. Effectiveness of mobile apps in teaching field-based identification skills. Journal of Biological Education. 2017;51(2): 136-143. doi:. [Context Link]


58. Virtanen MA, Kaariainen M, Liikanen E, Haavisto E. The comparison of students' satisfaction between ubiquitous and Web-based learning environments. Education and Information Technologies. 2017;22(5): 2565-2581. doi:. [Context Link]


59. Yang L. The effects of L2 proficiency on pragmatics instruction: a Web-based approach to teaching Chinese expressions of gratitude. L2 Journal. 2017;9(1): 62-83. doi:. [Context Link]


60. Yilmaz O. The effects of "live virtual classroom" on students' achievement and students' opinions about "live virtual classroom" at distance education. Turkish Online Journal of Educational Technology-TOJET. 2015;14(1): 108-115. [Context Link]


61. Steinke EE, Barnason S, Mosack V, Hill TJ. Baccalaureate nursing students' application of social-cognitive sexual counseling for cardiovascular patients: a Web-based educational intervention. Nurse Education Today. 2016;44: 43-50. doi:. [Context Link]


62. Pourmand A, Tanski M, Davis S, Shokoohi H, Lucas R, Zaver F. Educational technology improves ECG interpretation of acute myocardial infarction among medical students and emergency medicine residents. The Western Journal of Emergency Medicine. 2015;16(1): 133-137. doi:. [Context Link]


63. Alvarez AG, Dal Sasso GTM, Iyengar MS. Persuasive technology in teaching acute pain assessment in nursing: results in learning based on pre and post-testing. Nurse Education Today. 2017;50: 109-114. doi:. [Context Link]


64. Rojjanasrirat W, Rice J. Evidence-based practice knowledge, attitudes, and practice of online graduate nursing students. Nurse Education Today. 2017;53: 48-53. doi:. [Context Link]


65. Velan GM, Goergen SK, Grimm J, Shulruf B. Impact of interactive e-learning modules on appropriateness of imaging referrals: a multicenter, randomized, crossover study. Journal of the American College of Radiology. 2015;12(11): 1207-1214. doi:. [Context Link]


66. Bai H, Aman A, Xu Y, Orlovskaya N, Zhou M. Effects of Web-based interactive modules on engineering students' learning motivations. American Journal of Engineering Education. 2016;7(2): 83-96. [Context Link]


67. Hazaea AN, Alzubi AA. The effectiveness of using mobile on EFL learners' reading practices in Najran University. English Language Teaching. 2016;9(5): 8-21. doi:. [Context Link]


68. Lanken PN, Novack DH, Daetwyler CR, et al. Efficacy of an Internet-based learning module and small-group debriefing on trainees' attitudes and communication skills toward patients with substance use disorders: results of a cluster randomized controlled trial. Academic Medicine. 2015;90(3): 345-354. doi:. [Context Link]


69. Salajegheh A, Jahangiri A, Dolan-Evans E, Pakneshan S. A combination of traditional learning and e-learning can be more effective on radiological interpretation skills in medical students: a pre- and post-intervention study. BMC Medical Education. 2016;16: 46-47. doi:. [Context Link]


70. Djukic M, Adams J, Fulmer T, et al. E-learning with virtual teammates: a novel approach to interprofessional education. Journal of Interprofessional Care. 2015;29(5): 476-482. doi:. [Context Link]


71. Desplaces D, Blair CA, Salvaggio T. Do e-learning tools make a difference? Results from a case study. The Quarterly Review of Distance Education. 2015;16(4): 23-34. [Context Link]


72. Lameris AL, Hoenderop JG, Bindels RJ, Eijsvogels TM. The impact of formative testing on study behaviour and study performance of (bio)medical students: a smartphone application intervention study. BMC Medical Education. 2015;15: 72-78. doi:. [Context Link]


73. Carlson E, Stenberg M, Chan B, et al. Nursing as universal and recognisable: nursing students' perceptions of learning outcomes from intercultural peer learning webinars: a qualitative study. Nurse Education Today. 2017;57: 54-59. doi:. [Context Link]


74. Murphy R, Clissold E, Keynejad RC. Problem-based, peer-to-peer global mental health e-learning between the UK and Somaliland: a pilot study. Evidence-Based Mental Health. 2017;20(4): 142-146. doi:. [Context Link]


75. Carpenter R, Estrada CA, Medrano M, Smith A, Massie FS Jr.A Web-based cultural competency training for medical students: a randomized trial. The American Journal of the Medical Sciences. 2015;349(5): 442-446. doi:. [Context Link]


76. Woodham LA, Ellaway RH, Round J, Vaughan S, Poulton T, Zary N. Medical student and tutor perceptions of video versus text in an interactive online virtual patient for problem-based learning: a pilot study. Journal of Medical Internet Research. 2015;17(6): e151. doi:. [Context Link]


77. Karabag Aydin A, Dinc L. Effects of Web-based instruction on nursing students' arithmetical and drug dosage calculation skills. Computers, Informatics, Nursing. 2017;35(5): 262-269. doi:. [Context Link]


78. Davie E, Martin M, Cuppett M, Lebsack D. Effectiveness of mobile learning on athletic training psychomotor skill acquisition. Athletic Training Education Journal. 2015;10(4): 287-295. doi:. [Context Link]


79. Gyamfi SA, Gyaase PO. Students' perception of blended learning environment: a case study of the University of Education, Winneba, Kumasi-campus, Ghana. International Journal of Education and Development Using Information and Communication Technology. 2015;11(1): 80-100. [Context Link]


80. Shang H-F, Chen Y-Y. The impact of online autonomous learning on EFL students' reading skills. International Journal on E-Learning. 2018;17(2): 227-249. [Context Link]


81. Wake D, Dailey D, Cotabish A, Benson T. The effects of virtual coaching on teacher candidates' perceptions and concerns regarding on-demand corrective feedback. Journal of Technology and Teacher Education. 2017;25(3): 327-357. [Context Link]


82. Alnabelsi T, Al-Hussaini A, Owens D. Comparison of traditional face-to-face teaching with synchronous e-learning in otolaryngology emergencies teaching to medical undergraduates: a randomised controlled trial. European Archives of Oto-Rhino-Laryngology. 2015;272(3): 759-763. doi:. [Context Link]


83. Martin F, Ertzberger J. Effects of reflection type in the here and now mobile learning environment. British Journal of Educational Technology. 2016;47(5): 932-944. doi:. [Context Link]


84. Tezer M, Cimsir BT. The impact of using mobile-supported learning management systems in teaching Web design on the academic success of students and their opinions on the course. Interactive Learning Environments. 2018;26(3): 402-410. doi:. [Context Link]


85. Long JD, Gannaway P, Ford C, et al. Effectiveness of a technology-based intervention to teach evidence-based practice: the EBR tool. Worldviews on Evidence-Based Nursing. 2016;13(1): 59-65. doi:. [Context Link]


86. Chaves RO, von Wangenheim CG, Furtado JCC, Oliveira SRB, Santos A, Favero EL. Experimental evaluation of a serious game for teaching software process modeling. IEEE Transactions on Education. 2015;58(4): 289-296. doi:. [Context Link]


87. Nguyen TN. Motivational effect of Web-based simulation game in teaching operations management. Journal of Education and Training Studies. 2015;3(2): 9-15. doi:. [Context Link]


88. Liu M, McKelroy E, Corliss SB, Carrigan J. Investigating the effect of an adaptive learning intervention on students' learning. Educational Technology Research and Development. 2017;65(6): 1605-1625. doi:. [Context Link]


89. Nickerson C, Rapanta C, Goby VP. Mobile or not? Assessing the instructional value of mobile learning. Business and Professional Communication Quarterly. 2017;80(2): 137-153. doi:. [Context Link]


90. Basitere M, Ndeto Ivala E. An evaluation of the effectiveness of the use of multimedia and Wiley plus Web-based homework system in enhancing learning in the chemical engineering extended curriculum program physics course. Electronic Journal of e-Learning. 2017;15(2): 156-173. [Context Link]


91. Van Lancker A, Baldewijns K, Verhaeghe R, et al. The effectiveness of an e-learning course on medication calculation in nursing students: a clustered quasi-experimental study. Journal of Advanced Nursing. 2016;72(9): 2054-2064. doi:. [Context Link]


92. Lu OH, Huang JC, Huang AY, Yang SJ. Applying learning analytics for improving students engagement and learning outcomes in an MOOCS enabled collaborative programming course. Interactive Learning Environments. 2017;25(2): 220-234. doi:. [Context Link]


93. Millis K, Forsyth C, Wallace P, Graesser AC, Timmins G. The impact of game-like features on learning from an intelligent tutoring system. Technology, Knowledge and Learning. 2017;22(1): 1-22. doi:. [Context Link]


94. Bellhauser H, Losch T, Winter C, Schmitz B. Applying a Web-based training to foster self-regulated learning-effects of an intervention for large numbers of participants. The Internet and Higher Education. 2016;31: 87-100. doi:. [Context Link]


95. Fernandes Pereira FG, Afio Caetano J, Marques Frota N, Gomes da Silva M. Use of digital applications in the medicament calculation education for nursing. Investigacion y Educacion en Enfermeria. 2016;34(2): 297-304. doi:. [Context Link]


96. Carter EB, Wiles JR. Comparing effects of comedic and authoritative video presentations on student knowledge and attitudes about climate change. Bioscene. 2016;42(1): 16-24. [Context Link]


97. Chyr W-L, Shen P-D, Chiang Y-C, Lin J-B, Tsai C-W. Exploring the effects of online academic help-seeking and flipped learning on improving students' learning. Journal of Educational Technology & Society. 2017;20(3): 11-23. [Context Link]


98. Sobocan M, Turk N, Dinevski D, Hojs R, Pecovnik Balon B. Problem-based learning in internal medicine: virtual patients or paper-based problems?Internal Medicine Journal. 2017;47(1): 99-103. doi:. [Context Link]


99. Rolskov Bojsen S, Rader SB, Holst AG, et al. The acquisition and retention of ECG interpretation skills after a standardized Web-based ECG tutorial-a randomised study. BMC Medical Education. 2015;15: 36. doi:. [Context Link]


100. Fleming R, Stoiber LC, Pfeiffer HM, et al. Using U-pace instruction to improve the academic performance of economically disadvantaged undergraduates. Journal of Computer Assisted Learning. 2016;32(4): 304-313. doi:. [Context Link]


101. Chang L-C, Guo JL, Lin H-L. Cultural competence education for health professionals from pre-graduation to licensure delivered using Facebook: twelve-month follow-up on a randomized control trial. Nurse Education Today. 2017;59: 94-100. doi:. [Context Link]


102. Tsai C-W. Exploring the effects of online team-based learning and co-regulated learning on students' development of computing skills. Interactive Learning Environments. 2016;24(4): 665-680. doi:. [Context Link]


103. Park JY, Woo CH, Yoo JY. Effects of blended cardiopulmonary resuscitation and defibrillation e-learning on nursing students' self-efficacy, problem solving, and psychomotor skills. Computers, Informatics, Nursing. 2016;34(6): 272-280. doi:. [Context Link]


104. Sabatino JA, Pruchnicki MC, Sevin AM, Barker E, Green CG, Porter K. Improving prescribing practices: a pharmacist-led educational intervention for nurse practitioner students. Journal of the American Association of Nurse Practitioners. 2017;29(5): 248-254. doi:. [Context Link]


105. Thalluri J, Penman J. Social media for learning and teaching undergraduate sciences: good practice guidelines from intervention. Electronic Journal of e-Learning. 2015;13(6): 455-465. [Context Link]


106. Viberg O, Hatakka M, Balter O, Mavroudi A. The current landscape of learning analytics in higher education. Computers in Human Behavior. 2018;89: 98-110. doi:. [Context Link]


107. Redecker C. European Framework for the Digital Competence of Educators: DigCompEdu. Punie Y, ed. EUR 28775 EN. Publications Office of the European Union, Luxembourg, 2017. Doi:. [Context Link]