Skip to main content
  • Research Article
  • Open access
  • Published:

Process evaluation of E-learning in continuing medical education: evidence from the China-Gates Foundation Tuberculosis Control Program

Abstract

Background

E-learning is a growing phenomenon which provides a unique opportunity to address the challenges in continuing medical education (CME). The China-Gates Foundation Tuberculosis (TB) Control Program implemented online training for TB health workers in three provinces of China. We aim to evaluate the implementation of E-learning CME programs, analyse the barriers and facilitators during the implementation process, and to provide policy recommendations.

Methods

Routine monitoring data were collected through the project office from December 2017 to June 2019. In-depth interviews, focus group discussion with project management personnel, teachers, and trainees (n = 78), and staff survey (baseline n = 555, final n = 757) were conducted in selected pilot areas at the provincial, municipal, and county/district levels in the three project provinces (Zhejiang, Jilin, and Ningxia). Descriptive analysis of quantitative data summarized the participation, registration, and certification rates for training activities. Thematic approach was used for qualitative data analysis.

Results

By the end of June 2019, the national and provincial remote training platforms had organized 98 synchronous learning activities, with an average of 173.2 people [standard deviation (SD) = 49.8] per online training session, 163.3 people (SD = 41.2) per online case discussion. In the pilot area, 64.5% of TB health workforce registered the asynchronous learning platform, and 50.1% obtained their professional certifications. Participants agreed that E-learning CME was more economical, has better content as well as more flexible work schedules. However, the project still faced challenges in terms of unmet learning needs, disorganized governance, insufficient hardware and software, unsupported environment, and lack of incentive mechanisms.

Conclusions

Our results suggested that it’s feasible to conduct large scale E-learning CME activities in the three project provinces of China. Training content and format are key facilitators of the program implementation, while the matching of training supply and demand, organizational coordination, internet technology, motivations, and sustainability are key barriers.

Background

E-learning is a broad concept that involves the provision of educational programmes through electronic systems [1]. Since the 1990s, with the great progress of information technology, the E-learning market has been quickly growing around the world. In 2002, the International Data Corporation report estimated the world E-learning market was only USD 6.6 billion [2]. However, in 2018, the market size has grown to USD 190 billion and is projected to flourish into USD 423 billion by 2023 [3]. In China, the number of online education users in 2018 reached 144 million, and they downloaded online education applications (apps) more than 276 million times, which was an increase in both by a yearly rate of 20% [4]. Existing evidence indicates that E-learning could reduce cost [5, 6], improve the access to education [7] as well as provide more flexibilities for students who have work and family commitments [8, 9]. In this way, E-learning is considered to have a unique advantage in the implementation of continuing education. However, a huge volume of research on the effects did not provide information on whether effective interventions (such as E-learning) would be reproduced in specific context [10, 11]. There has been little research on E-learning implementation in continuing medical education (CME) programs, especially in low- and middle-income settings [12,13,14].

In the context of China, increasing demands for tuberculosis (TB) CME programs have taken place since the significant reform of China’s TB control system [15,16,17]. Under the new TB service delivery model, a large number of tuberculosis-related services have been transferred from the Centres for Diseases Prevention and Control (CDCs) to the designated hospitals, which highlighted a great shortage of the qualified TB health workforce [18]. At the same time, with the fast technology development in TB diagnosis, treatment, and prevention, the knowledge and skills of the existing personnel also need to be updated. These two factors have created a huge demand for CME among China’s TB control institutions. E-learning tools are increasingly available around the world. For example, The International Union Against Tuberculosis and Lung Disease (The Union) and Médecins Sans Frontières (MSF) have developed the Structured Operational Research Training Initiative (SORT IT) course, which has shown an encouraging and sustained research capacity improvement among participants [19, 20]. There is little evidence on how these E-learning tools are applied in different settings.

Since 2012, China CDC and Bill & Melinda Gates Foundation are planning to introduce and expand the new comprehensive model of TB control in China. To achieve this goal, two phases of program focused on multidrug resistant tuberculosis (MDR-TB) control, and accessibility and affordability for TB care have been implemented in 2009–2012 and 2012–2015 [21]. This project, started in 2017, named the China-Gates Foundation TB Control Program Phase 3, covered multiple complex interventions including E-learning CME for TB health personnel. We used both qualitative and quantitative approaches, to assess the participation to the E-learning CME program, to identify barriers and facilitators to implementing E-learning CME, and to provide related policy recommendations.

Methods

Intervention design

The E-learning project was implemented from May 2016 to June 2019 among three project provinces (Zhejiang represents the most developed eastern area in China, while Jilin is from less developed central area and Ningxia represents the least developed western area). In each province, the China-Gates program management office selected two cities as pilot areas for the E-learning project according to their level of socioeconomic development and TB health service capacity [for example, number of TB health workers, gross domestic product (GDP) per capita, level of network hardware, etc.].

Two key interventions were designed in the E-learning project of the China-Gates Foundation TB Control Program Phase 3 [22]. First, the national and provincial remote training and medical consultation mechanism was established using the "National TB Telemedicine Consultation and Training Platform", a live streaming platform developed by Clinical Centre on Tuberculosis in China CDC. This platform focused on the training of advanced diagnosis and treatment of complex clinical conditions for TB clinical staff at the county level and above (hereinafter referred to as "synchronous E-learning"). The average duration of each lecture of the synchronous E-learning is 1–1.5 h. Second, an online training and qualification system (China TB prevention Online Training Website) was developed for all TB health workers, including clinical doctors, public health physicians, and primary care medical staff. The system focused on basic knowledge and theory of TB control, with pre-recorded video tutorials. Participants can watch the video in their convenient time (hereinafter referred to as "asynchronous E-learning"). Each module of the asynchronous E-learning consists of a 25–30 min video and 5–10 questions.

The development process of these E-learning tools was led by the Clinical Centre on Tuberculosis of China CDC, which is China’s national centre for TB clinical treatment and prevention. For the synchronous E-learning, national and international experts on specific areas of TB diagnosis and treatment were selected and invited to give live lectures. For the asynchronous E-learning, a national expert group on TB control was organized to develop teaching materials based on an updated national TM control guideline. The whole process of intervention design and implementation took more than three years. The China-Gates Program has set up an advisory committee to review and evaluate the training contents.

Study design: mixed methods of qualitative and quantitative study

This study was a qualitative study with a quantitative supplementary data. Our study results were reported according to the Consolidated criteria for Reporting Qualitative research (COREQ) reporting guidelines [23]. Site selecting criteria for intervention and sampling methods are reported in detail elsewhere [18]. Briefly speaking, the evaluation program conducted three waves of field trips (baseline, process, and final evaluation) in 2017, 2018 and 2019, respectively. We selected two cities in each of three provinces (Zhejiang, Jilin, and Ningxia) as study sites according to their level of socioeconomic development (for example, GDP per capita, types of TB health service delivery model, etc.).

Three different types of data were used in our study. First, qualitative data (key informant interviews and focused group discussions) were collected to explore the barriers and facilitators of the implementation of project. Key informant interviews (30–45 min) were conducted with project management personnel at national and provincial levels, teachers who attended training sessions, and primary care sectors. In each designated hospital and CDC, 45–60-min focused group discussions (FGDs) were conducted with 6–8 TB-related doctors, 2–3 public health physicians, or 1–3 primary care workers. The interview guide included issues on barriers and facilitators to the TB health workers’ engagement in the E-learning, the subjective effectiveness of E-learning, and the feedback in the synchronous/asynchronous E-learning from the trainers and trainees, especially compared with the traditional face to face training. The participants were recruited by coordinators from provincial and county level CDC with a purposive sampling method [24]. The interviews and FGDs were conducted in a quiet and private meeting room or office room without any other irrelevant person. A senior researcher conducted the interviews and FGDs as the interviewer or facilitator, with a junior researcher as an observer and notetaker. All interviews were recorded the after the consent form were signed. In total, we interviewed 78 participants with 17 key informant interviews and 14 FGDs in July and August 2018 (Table 1).

Table 1 Sample size for key informant interviews and focused group discussions

In addition to qualitative data, the routinely collected data from the project management office monitored all the TB institutions in the polit area from January 2017 to June 2019, which provided quantitative data for the program implementation and participation. We also performed staff questionnaires in TB health workers from three provinces during the baseline and final evaluation. In the baseline questionnaire, we collected quantitative data on participants’ demographic variables, participation and feedback of their existing CME activities (face-to-face), training need, and a quiz on TB knowledge. In the final questionnaire, we further added several questions on their participation and feedback of the synchronous and/or asynchronous online learning activities. In total, 555 TB-related health workers completed the baseline questionnaire and 757 completed the final questionnaire.

Data analysis

For both quantitative and qualitative data, analysis was conducted for two dimensions. The first dimension was the implementation of synchronous/asynchronous E-learning (descriptive statistical analyses on participation rate of health facilities and individual health workers in the synchronous training sessions, the registration rates/certification rates for asynchronous training activities). The second dimension was the barriers and facilitators to implement the E-learning program. We transcribed the qualitative data and used a hybrid approach in thematic analysis [25]. The analytical framework was developed based on the topic guides and emerging issues from the interviews and FGDs. The quantitative data were analysed using Stata 14.0 (StataCorp, College Station, TX, USA) and the qualitative data were analysed in MAXQDA 2018 (VERBI GmbH, Berlin, German).

Results

Implementation of E-learning

Implementation of synchronous E-learning

By the end of June 2019, the national-level and three provincial training centres had conducted 98 synchronous learning activities, including 47 national-level activities (32 remote training activities and 15 case discussions). The three provincial platforms organized 51 events. Jilin had organized more activities (20) than Ningxia (13) and Zhejiang (12) (Table 2).

Table 2 Implementation of synchronous E-learning activities through the China-Gates Foundation Tuberculosis Control Program

Thirty-nine TB institutions in the pilot area have access to the remote synchronous learning platform (14 in Jilin, 9 in Ningxia, and 16 in Zhejiang). On average, 23.5 TB institutions and 173.2 TB health workers accessed the national-level platform for each synchronous E-learning activity, while an average of 22.5 units and 163.3 TB health workers participated in each case discussion. Jilin had the highest participation rate followed by Ningxia. The participation in Zhejiang was relatively low (Table 3).

Table 3 Participation of synchronous E-learning in three provinces

Implementation of asynchronous E-learning

By the end of June 2019, there were 23 online asynchronous training modules uploaded to the training system (7 for public health personnel, 8 for doctors, 4 for primary care workers, and 4 optional courses). Nine hundred and six (66.9%) TB health workers from the pilot area in three provinces registered for the system, and 681 (50.1%) TB health workers have obtained the qualification certificates. Zhejiang has the highest level of participation, followed by Jilin and Ningxia (Table 4).

Table 4 Participation of asynchronous E-learning activities in three provinces (%)
Table 5 Barriers and facilitators to implement synchronous and asynchronous E-learning activities

Key informants reported that different provinces had different patterns of training organization. In Zhejiang Province, asynchronous learning activities were mainly organized by the CDCs with the "province-city-county" three-tier administrative management mode, while synchronous learning activities were TB designated hospitals’ responsibility. The organization of synchronous learning activities was led by provincial designated hospital. Since there is no vertical administrative relationship between the designated hospitals, the provincial hospitals could not effectively coordinate the training activities. Instead, they had to rely on local CDCs as intermediary partners to send training reminders and other messages (Fig. 1). In Ningxia and Jilin, all institutions were notified of synchronous/asynchronous learning activities by provincial hospitals (including CDCs, hospitals, and TB dispensaries), while trainees at the township level were notified by county-level CDCs (Fig. 2).

Fig. 1
figure 1

Organizational structure of capacity building in Zhejiang. CDC Centre for Disease Control and Prevention

Fig. 2
figure 2

Organizational structure of capacity building in Jilin and Ningxia. CDC Centre for Disease Control and Prevention, TB tuberculosis

Facilitators of E-learning CME

Facilitators of the training format

In the interviews, most TB health workers agreed that the E-learning model is economical and timesaving compared to traditional face to face training. In the past, only a few face-to-face training opportunities were available for TB health workers, especially for those from lower-level institutions. The TB institutions could only send one or two doctors to attend the training and forward the training contents for those who were not able to attend. They often used meetings as a substitute for on-job training. E-learning have expanded access to high-quality continuing medical educational resources, and reduced its costs.

Compared with the traditional training model, the online and remote training are timesaving and more efficient. Traditional methods are not able to invite people come over frequently, and most hospitals can't afford the human and financial costs. (Provincial TB health professional FGD in Zhejiang)

Facilitators of training content

In general, the training contents of the China-Gates Program were regarded as high-quality by the TB health workers. The trainers of E-learning were well-known experts in China. The lower-level health workers could receive training and technological guidance from the national experts, which was unlikely in traditional face-to-face training. Different health workers have different preferences: provincial and city-level TB health workers preferred the frontiers and advances in synchronous E-learning activities, while county-level TB health workers favoured sections about routine clinical practice in asynchronous E-learning activities.

It's really nice to stay at home and have direct access to teachers from Beijing or across the country, teaching us the basic knowledge or cutting-edge ideas. It’s brilliant. (Provincial TB health professional FGD in Jilin) (Table 5)

Barriers in the implementation of E-learning

Barriers in matching the supply and demand of training

Although China-Gates Program provided TB staff with high-quality training contents, they were not evenly spread out among different types of TB health workers. In general, more courses were available to the clinical doctors: the total training loads including synchronous session once a week (or twice monthly) and nine modules of asynchronous sessions. However, the training loads for public health physicians (seven modules in total) and primary care workers (four modules in total) was relatively small.

The uneven supply and demand was another key concern. Staff survey showed that clinical doctors thought they need more training in the clinical diagnosis and treatment of TB (83.5% for multidrug-resistant TB, MDR-TB, 78.4% for TB radiology and 70.1% for common TB), which perfectly matched with their training supplies (Table 6). Public health physicians expected more training in TB infection control (76.2%), planning (73.4%), and surveillance (63.1%), they mostly received training in planning (91.8%), TB management (89.8%) and treatment of common TB (75.5%). Similar results were observed in FGDs. Some TB health workers complained that the training materials were too difficult to follow and some sessions were not related to their daily practices.

The content of remote (synchronous) training is too deep to be used in clinical practice, which is beyond our competence, especially for county-level personnel. (County-level TB health professional FGD in Jilin)

Table 6 Supply and demand of training among tuberculosis (TB) health workers.

Barriers in organizational coordination

The first challenge in organizational coordination was that the leaders did not pay enough attention to training. Interviews with project management personnel in Zhejiang and Ningxia provinces revealed that they could not mobilize adequate resources due to the low relative priority of training among leaders’ goals. On the contrary, the areas with higher levels of leaders’ attention such as Jilin reported higher level of training participation.

In our case, to be honest, we need to take care of everything. We are extremely busy…… For example, online training (asynchronous E-learning) is only a very small part of the China-Gates program, so we can't put lots of resources on it…… We can’t be anywhere; this is our situation. (Key informant interview of CDC leader in Zhejiang)

The second challenge in organizational coordination was how the CDCs and TB designated hospitals collaborate on organizing training, providing feedback, and solving problems. As we mentioned in Fig. 1, there was no effective coordination mechanism between CDCs and designated hospitals for TB CME. It hindered further expansion of the high-quality learning resources.

Barriers in internet technology

E-learning CME placed higher demands on hardware and software. According to the survey, there were three kinds of hardware equipment for synchronous E-learning: a few provincial hospitals had well-equipped special training centre; In most city-level hospitals, physicians used desktop computers and a projector to access to the E-learning platform as a group; However, in the most county hospitals, physicians could only use their personal computers in the office. In addition, technical issues arose when TB health workers received training, such as the slow processing speed of E-learning platform, network jams, system errors in learning progress, fuzzy sound and blurry videos.

Because we're not using a special-purpose device, sometimes the internet connection is not working. It happens sometimes that we could not join the (synchronous E-learning) classroom. (County-level TB health professional FGD in Jilin)

Another challenge came from the technology itself. Firstly, most medical staff feel that E-learning should not be a substitute for face-to-face training because of the barriers in teacher-student communication. Secondly, although they didn’t have to leave their work space for E-learning activities, the synchronous learning sessions still required all participants to join the virtual classroom at the same time. Many medical staff reflected that the training often conflicted with their work schedule, and they could not leave their office to participate in training during the working days.

Barriers in motivations

The lack of motivation was another source of barrier for training. The incentive mechanism in current TB health service delivery system in China has a sustained negative impact on TB health workers. Firstly, the medical institutions largely depend on services revenues to cover operational costs. However, according to the national TB control program, the TB departments are funded by the government to provide free medical services, they have limited capacity to generate revenues. In this way, TB health workers have lower income and lack of work motivation. Moreover, participation in training has no association with the promotion or payments, which makes medical personnel more reluctant to take part in training. Sometimes one single person in the department was asked to participate all the training, and signed up for every colleagues. Secondly, due to poor income, the TB workforce in the three provinces were generally older, with lower professional titles, and less education compared with their colleagues in other departments. Their learning capacities and digital literacy were relatively weak.

When you are not satisfied with your revenue, you don't have the motivation to work, you just deal with everything…… our clinical physicians, especially our heads of the clinical departments, they seem to feel that, managing patients, performance, and getting more money—he feels that that's the point, training is not important. (Key informant interview of project management personnel in Ningxia)

Since TB health personnel were more likely to be “passive learners” instead of “active learners”, the E-learning activities in many institutions still rely on external incentives such as CME credits. However, some project management personnel reflected that due to the stringent regulation of CME credits, it’s difficult for TB health workers to get credits from the China-Gates training project. They could only try other strategies such as sending repeated notifications to doctors or forcing them to participants, but neither of them works well.

Barriers in project sustainability

Many stakeholders expressed their concerns about the sustainability of the training. The capacity to mobilize resources of the training organizers was very limited: both national and provincial platforms are facing the tight constrains in terms of funding and human resources, which make it difficult to mobilize resources to continue the training after the program is completed. Additionally, government regulations stated that hospitals cannot charge patients for telemedicine consultations. Therefore, the platform has no existing model to raise funding to sustain itself.

Discussion

This study has aimed to analyse the barriers and facilitators to implementing E-learning in continuing medical education in three provinces of China. It found that the overall progress of the E-learning project of the China-Gates Foundation TB Control Program has been relatively smooth. According to the pre-set plan, 98 synchronous learning activities were conducted by the national and provincial platforms. The registration rate of the asynchronous online platform for TB professionals in the pilot areas has exceeded 60%. In general, the project has achieved the goal of transferring high-quality medical educational resources to grass-roots medical institutions. However, the project still faces a range of challenges in the implementation, such as unmet training needs for public health and primary care workers, inefficient management framework, insufficient hardware and software, the unsupported environment, and the lack of an incentive mechanism. The current training model has not satisfied the needs for long-term, systematic, and timely training of TB professional.

The China-Gates Foundation TB Control Program implemented a large-scale E-learning program in TB continuing medical education. Compared to the traditional face-to-face training, this E-learning program has several advantages in content and format of training: (i) This project invited national-level famous experts to give lectures, the content of lectures was revised by several waves of the experts’ consultation with academics across the country. (ii) TB medical personnel could stay at home to receive training from state-level experts and the high-quality educational resources has transferred to the grassroots level. Although it will not replace face to face training in the future, we did see great feasibility and necessity of E-learning, especially during the COVID-19 pandemic [26]. In addition, some blended learning program for TB researchers from low- and middle-income countries (LMICs), like the SORT IT course, has shown an encouraging and sustained research capacity improvement among participants [19, 20, 27, 28]. With the improved internet access in the future, it’s possible that E-learning will be components of the routine practices in CME.

Based on previous studies, E-learning has two contradictory effects: first, compared with traditional face-to-face learning, students in E-learning environment were more likely be inferred by online environment and lack of personal discipline, which has the negative effect on students’ performance. This phenomenon was well studied by many researches in high-level education where the students could acquire same teaching resources online and offline [5, 29,30,31,32,33]. Second, E-learning could expand access to education which is good for people from the underrepresented group. Only few studies realized the importance of accessibility improvement by E-learning [7], evidence shown that some teaching physicians from LMICs favoured the E-learning's unprecedented value in knowledge transmission and accessibility [34]. Since the high-quality tutorial was one of the main facilitators for China-Gates capacity building program, we argue that the second effect played a more important role in our study—many health workers could not take in-person classes from national experts. In further analyses, we did find some evidence that the program improved the knowledge of TB among primary care providers, probably due to the second phenomenon we mentioned (Wang et al. 2021, under review).

Our results highlight the lack of internal training incentives and low self-efficacy among TB personnel as one of the most important barriers. This phenomenon was caused by the policy environment and organizational culture, especially the incentive mechanism. In most study areas, the implementation of training programs still relied on external incentives, such as continuing medical education credits. Extrinsic motivations (continued medical education credit, attendance requirement) are important in the short terms, but the long-term success of training project depend on whether we can engage health workers as active participants in their learning. How to establish and strengthen the virtuous cycle of intrinsic motivation is worth further studying.

Different types of training have their own advantages, which are influenced by the type of course, the quality of the content, the communication between teachers and students, and the flexibility of time and space. Existing evidence showed that compared with face-to-face learning, E-learning has stronger negative effect for applied professional courses (e.g., business, law, and nursing), but little impact on theoretical courses such as philosophy and psychology [35]. It suggested that face-to-face training is more suitable for operational courses, which not only allows students to have hands-on opportunities, but also facilitates communication between teachers and students. However, for theoretical courses, quality educational resources are scarce, and medical personnel have to travel for training while they have work commitments. In this case, the advantages of E-learning became apparent. Therefore, E-learning and traditional face-to-face learning are complementary. The implementations of other pedagogies with a combination of face-to-face learning and E-learning in CMEs, such as blended learning [36], is a question for future studies.

Based on these findings, we proposed five policy recommendations. The first is to improve the organizational structure of training. All TB institutions should pay more attention to training and to create a learning culture of teams. Second, to promote multi-sectoral collaboration. In the context of “the trinity model” and as institutions with hierarchical administration, CDCs should assist the designated hospitals in this process. More management and coordination functions should be transferred from CDCs to designated hospitals, enhancing their abilities and willingness to train and serve hospitals at lower levels. Third, to strengthen the supervision and establishing effective incentive mechanisms. We suggest to add continuing medical education credits at the design stage, allowing primary care workers to obtain national credits from the basic courses, and to decentralize the authority of credits in advanced courses to the provincial and municipal levels. Fourth, to enhance the user experience through technical assistance. We would like to highlight several key elements: optimizing the operating speed of the training platform, adding resume breakpoint and the video replay function, setting up a forum to facilitate teacher-student communication and interaction, developing mobile apps, and conducting a survey on user behaviours in order to simplify the user interface and operation. Finally, the fifth recommendation is to create customized and diverse learning experiences. Designing blended learning modules to combine the online and offline courses needs to be considered. Doctors can selectively attend training sessions according to their own needs and schedules.

There are two main limitations in our studies: first, this study was performed within a specific group—health workers specialising in TB prevention and control, who were likely to have less than average interest due to the policy environment, and thus, the conclusion maybe inconsistent within other groups. In addition, we did not have the baseline information for synchronous E-learning participants, so we could not calculate the participation rate of synchronous E-learning activities as we did in the analyses of the asynchronous one.

Conclusions

It’s feasible to conduct large scale E-learning continuing medical education activities with TB health personnel in China. Training content and format seem to be key facilitators of the program implementation, while the matching the supply/demand of training, organizational coordination, internet technology, motivations, and sustainability are key barriers. Further research should consider the theory behind the sperate effects more carefully in order to set up priorities for E-learning activities in CME.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

apps:

Applications

CDC:

Centre for Disease Control and Prevention

CME:

Continuing medical education

COREQ:

Consolidated criteria for Reporting Qualitative research

COVID-19:

Coronavirus disease 2019

FGDs:

Focused group discussions

GDP:

Gross domestic product

MDR-TB:

Multidrug resistant tuberculosis

SD :

Standard deviation

SORT IT:

Structured Operational Research Training Initiative

TB:

Tuberculosis

References

  1. Clark RC, Mayer RE. E-learning and the science of instruction: proven guidelines for consumers and designers of multimedia learning. 4th ed. San Francisco: Wiley; 2016.

    Book  Google Scholar 

  2. Hayashi A, Chen C, Ryan T, Wu J. The role of social presence and moderating role of computer self efficacy in predicting the continuance usage of e-learning systems. J Inf Syst Educ. 2004;15(2):139–54.

    Google Scholar 

  3. Kaur P. E-learning pedagogy to promote ethics of global system education. RRIJM. 2019;4(5):527–9.

    Google Scholar 

  4. Education Management Information Centre, Ministry of Education of the People's Republic of China, Engineering Research Centre of the Ministry of Education for Digital Learning and Education Public Services, Baidu Education: 2017 China Online Education Industry White Paper. Beijing: Tsinghua University Press; 2018.

  5. Bettinger EP, Fox L, Loeb S, Taylor ES. Virtual classrooms: how online college courses affect student success. Am Econ Rev. 2017;107(9):2855–75.

    Article  Google Scholar 

  6. Deming DJ, Goldin C, Katz LF, Yuchtman N. Can online learning bend the higher education cost curve? Am Econ Rev. 2015;105(5):496–501.

    Article  Google Scholar 

  7. Goodman J, Melkers JE, Pallais A. Can online delivery increase access of education? Cesifo Working Paper. 2016.

  8. Xu D, Xu Y. The promises and limits of online higher education: understanding how distance education affects access, cost, and quality. Irvine: American Enterprise Institute; 2019.

    Google Scholar 

  9. Johnson HP, Mejia MC. Online learning and student outcomes in California’s community colleges. San Francisco: Public Policy Institute of California; 2014.

    Google Scholar 

  10. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

    Article  Google Scholar 

  11. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655.

    Article  Google Scholar 

  12. Eslaminejad T, Masood M, Ngah NA. Assessment of instructors’ readiness for implementing e-learning in continuing medical education in Iran. Med Teach. 2010;32(10):e407–12.

    Article  Google Scholar 

  13. Gagnon MP, Légaré F, Labrecque M, Frémont P, Cauchon M, Desmartis M. Perceived barriers to completing an e-learning program on evidence-based medicine. Inform Prim Care. 2007;15(2):83–91.

    PubMed  Google Scholar 

  14. Ruf D, Kriston L, Berner M, Härter M. General practitioners and online continuing medical education–which factors influence its use? Ger Med Sci. 2009;7:Doc08.

    PubMed  PubMed Central  Google Scholar 

  15. Wei X, Zou G, Yin J, Walley J, Sun Q. Comparing patient care seeking pathways in three models of hospital and TB programme collaboration in China. BMC Infect Dis. 2013;13(1):93.

    Article  Google Scholar 

  16. Wei X, Liang X, Walley J, Liu F, Dong B. Analysis of care-seeking pathways of tuberculosis patients in Guangxi, China, with and without decentralised tuberculosis services. Int J Tuberc Lung Dis. 2009;13(4):514–20.

    PubMed  Google Scholar 

  17. Tang S, Squire SB. What lessons can de drawn from tuberculosis (TB) Control in China in the 1990s?: an analysis from a health system perspective. Health Policy. 2005;72(1):93–104.

    Article  Google Scholar 

  18. Wang Z, Jiang W, Liu Y, Zhang L, Zhu A, Tang S, Liu X. Transforming tuberculosis (TB) service delivery model in China: issues and challenges for health workforce. Hum Resour Health. 2019;17(1):83.

    Article  Google Scholar 

  19. Guillerm N, Tayler-Smith K, Berger SD, Bissell K, Kumar A, Ramsay A, et al. What happens after participants complete a Union-MSF structured operational research training course? Public Health Action. 2014;4(2):89–95.

    Article  CAS  Google Scholar 

  20. Bissell K, Harries AD, Reid AJ, Edginton M, Hinderaker SG, Satyanarayana S, et al. Operational research training: the course and beyond. Public Health Action. 2012;2(3):92–7.

    Article  CAS  Google Scholar 

  21. Tang S, Wang L, Wang H, Chin DP. Access to and affordability of healthcare for TB patients in China: issues and challenges. Infect Dis Poverty. 2016;5(1):1–5.

    Article  Google Scholar 

  22. China-Gates TB Control Project Office of Clinical Centre for Tuberculosis Prevention, Chinese Centre for Disease Control and Prevention: Pilot Project for a Comprehensive New Model of Capacity Building for China-Gates TB Control Project (Phase III). Beijing; 2017.

  23. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349–57.

    Google Scholar 

  24. Robinson OC. Sampling in interview-based qualitative research: a theoretical and practical guide. Qual Res Psychol. 2014;11(1):25–41.

    Article  Google Scholar 

  25. Fereday J, Muir-Cochrane E. Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development. Int J Qual Methods. 2006;5(1):80–92.

    Article  Google Scholar 

  26. Ting DSW, Carin L, Dzau V, Wong TY. Digital technology and COVID-19. Nat Med. 2020;26(4):459–61.

    Article  CAS  Google Scholar 

  27. Guillerm N, Bissell K, Kumar A, Ramsay A, Reid A, Zachariah R, Harries A. Sustained research capacity after completing a Structured Operational Research and Training (SORT IT) course. Public Health Action. 2016;6(3):207–8.

    Article  CAS  Google Scholar 

  28. Guillerm N, Tayler-Smith K, Dar Berger S, Bissell K, Kumar A, Ramsay A, et al. Research output after participants complete a Structured Operational Research and Training (SORT IT) course. Public Health Action. 2015;5(4):266–8.

    Article  CAS  Google Scholar 

  29. Xu D, Jaggars SS. Performance gaps between online and face-to-face courses: Differences across types of students and academic subject areas. J Higher Educ. 2014;85(5):633–59.

    Article  Google Scholar 

  30. Xu D, Jaggars SS. The impact of online learning on students’ course outcomes: evidence from a large community and technical college system. Econ Educ Rev. 2013;37(C):46–57.

    Article  Google Scholar 

  31. Figlio D, Rush M, Yin L. Is it live or is it internet? Experimental estimates of the effects of online instruction on student learning. J Labor Econ. 2013;31(4):763–84.

    Article  Google Scholar 

  32. Brown BW, Liedholm CE. Can web courses replace the classroom in principles of microeconomics? Am Econ Rev. 2002;92(2):444–8.

    Article  Google Scholar 

  33. Bowen WG, Chingos MM, Lack KA, Nygren TI. Interactive learning online at public universities: evidence from a six-campus randomized trial. J Policy Anal Manage. 2013;33(1):94–111.

    Article  Google Scholar 

  34. Basu S, Marimuthu Y, Sharma N, Sharma P, Gangadharan N, Santra S. Attitude towards mobile learning among resident doctors involved in undergraduate medical education at a government medical college in Delhi, India. J Educ Health Promot. 2020;9(1):321.

    Article  Google Scholar 

  35. Oreopoulos P, Petronijevic U, Logel C, Beattie G. Improving non-academic student outcomes using online and text-message coaching. J Econ Behav Organ. 2020;171:342–60.

    Article  Google Scholar 

  36. Morton CE, Saleh SN, Smith SF, Hemani A, Ameen A, Bennie TD, Toro-Troconis M. Blended learning: how can we optimise undergraduate student engagement? BMC Med Educ. 2016;16:195.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank all the key informants who participated in this study. We also thank Dr. Xiao-Lin Wang from the Fourth People's Hospital of Ningxia, Dr. Xiao-Meng Wang from Zhejiang CDC, Dr. Yi-Ming Han from Hangzhou Red Cross Hospital, Dr. Yan-Li Yuan from Jilin provincial TB dispensary, and Dr. Yang Dong from Jilin Provincial Tuberculosis Hospital for their help during the field survey. We also thank Dr. Howard Bergman and Dr. Isabelle Vedel from McGill University for their insightful comments on the article and Mary Henein (McGill University) for editing the manuscript.

Funding

This study is part of the Duke University research project titled “Monitoring, Learning & Evaluation for the Implementation of the Comprehensive Model of Tuberculosis (TB) Care & Control in China” and received funding from the Bill & Melinda Gates Foundation (Approval number: OPP1149395). The funder of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report.

Author information

Authors and Affiliations

Authors

Contributions

XL and ST conceived and designed the study. ZW, LZ, and WJ participated the surveys and conducted data analysis. ZW prepared the tables and figures; ZW and XL wrote the first draft of the manuscript. ST, LZ and YL contributed to and critically revised the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Xiao-Yun Liu.

Ethics declarations

Ethics approval and consent to participate

The Duke University Institutional Review Board provided ethics approval of the survey (Approval number: 2017-0768). All respondents read a statement that explained the purpose of the study and gave their consent to continue.

Consent for publication

Not applicable.

Competing interests

The authors report no competing of interest. The authors alone are responsible for the content and writing of this article.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, ZY., Zhang, LJ., Liu, YH. et al. Process evaluation of E-learning in continuing medical education: evidence from the China-Gates Foundation Tuberculosis Control Program. Infect Dis Poverty 10, 23 (2021). https://doi.org/10.1186/s40249-021-00810-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40249-021-00810-x

Keywords