Barriers and Facilitators to Implement E-learning in Continuing Medical Education: Evidence From the China-gates Foundation Tuberculosis (TB) Control Program

[Objective] To evaluate the implementation of E-learning continuing medical education (CME) programs, analyse the barriers and facilitators during the implementation process, and provide policy recommendations based on the evidence from the China-Gates Foundation TB Control Program. [Methods] Routine monitoring data were collected through the project oce. In-depth interviews, focus group discussion with project management personnel, teachers, and trainees (N=78), and staff survey (baseline N=555, nal N=757) were conducted in selected pilot areas at the provincial, municipal, and county/district levels in the three project provinces (Zhejiang, Jilin, and Ningxia). [Results] By the end of June 2019, the national and provincial remote training platforms had organized 98 synchronous learning activities, with an average of 173.2 people (standard deviation, SD=49.8) per online training session, 163.3 people (SD=41.2) per online case discussion. In the pilot area, 64.5% of TB health workforce registered the asynchronous learning platform, and 50.1% obtained their professional certications. Participants agreed that E-learning CME was more economical, has better content as well as more exible work schedules. However, the project still faced challenges in terms of unmet learning needs, disorganized governance, insucient hardware and software, unsupported environment, and lack of incentive mechanisms. [Conclusions] Our results suggested that it’s feasible to conduct large scale E-learning continuing medical education activities in the three project provinces of China. Training content and format are key facilitators of the program implementation, while the matching the supply/demand of training, organizational coordination, internet technology, motivations, and sustainability are key barriers.


Introduction
Since the 1990s, with the great progress of information technology, the E-learning market has been quickly growing around the world. In 2002, the International Data Corporation (IDC) report estimated the world E-learning market was only $6.6 billion (Hayashi, Chen, Ryan, & Wu, 2020). However in 2018, the market size has grown to $190 billion and is projected to ourish into $423 billion by 2023 (Kaur, 2019).
In China, the number of online education users in 2018 reached 144 million, and they downloaded online education APPs more than 276 million times, which was an increase in both by a yearly rate of 20% E-learning CME for TB health personnel. We used both qualitative and quantitative approaches, to assess the participation to the E-learning CME program, to identify barriers and facilitators to implementing Elearning CME, and to provide related policy recommendations.

Method
Intervention design Three different types of data were used in our study. First, qualitative data (key informant interviews and focused group discussions) were collected to explore the barriers and facilitators of the implementation of project. 30-45-minute Key informant interviews were conducted with project management personnel at national and provincial levels, teachers who attended training sessions, and primary care sectors. In each designated hospital and CDC, 45-60-minute focused group discussions (FGDs) were conducted with 6-8 TB-related doctors, 2-3 public health physicians, or 1-3 primary care workers. The interview guide included issues on barriers and facilitators to the TB health workers' engagement in the E-learning, the subjective effectiveness of E-learning, and the feedback in the synchronous/asynchronous E-learning from the teachers and students, especially compared with the traditional face to face training. The participants were recruited by coordinators from provincial and county level CDC with a purposive sampling method (Robinson, 2014). The interviews and FGDs were conducted in a quiet and private meeting room or o ce room without any other irrelevant person. A senior researcher conducted the interviews and FGDs as the interviewer or facilitator, with a junior researcher as an observer and notetaker. All interviews were recorded the after the consent form were signed. In total, we interviewed 78 participants with 17 key informant interviews and 14 FGDs in July and August 2018 (Table 1).
In addition to qualitative data, the routinely collected data from the project management o ce monitored all the TB institutions in the polit area from January 2017 to June 2019, which provided quantitative data for the program implementation and participation. We also performed staff questionnaires in TB health workers from three provinces during the baseline and nal evaluation, which provided quantitative data about their training needs and the participants' opinion on the training program. In total, 555 TB-related health workers completed the baseline questionnaire and 757 completed the nal questionnaire.

Statistical analysis
For both quantitative and qualitative data, analysis was conducted for two dimensions. The rst dimension was the implementation of synchronous/asynchronous E-learning (descriptive statistical analyses including the average number of institutions and students to enrol in the synchronous training sessions, the number of students enrolled in the asynchronous learning platform and obtained the certi cates). The second dimension was the barriers and facilitators to implement the E-learning program. We transcribed the qualitative data and used a hybrid approach in thematic analysis (Fereday & Muir-Cochrane, 2006). The analytical framework was developed based on the topic guides and emerging issues from the interviews and FGDs. The quantitative data were analysed using Stata 14.0 (StataCorp, College Station, TX) and the qualitative data were analysed in MAXQDA 2018 (VERBI GmbH, Berlin, German).  (20) than Ningxia (13) and Zhejiang (12) ( Table 2).  (Table 3).  Key informants reported that different provinces have different patterns of training organization. In Zhejiang Province, asynchronous learning activities were mainly organized by the CDCs with the "province-city-county" three-tier administrative management mode, while synchronous learning activities were TB designated hospitals' responsibility. The organization of synchronous learning activities was led by provincial designated hospital. They faced many challenges: since the provincial TB designated hospital do not have a vertical administrative relationship with the city and county-level designated hospitals, like the CDCs do, they only maintain loose connections in the name of "technological guidance". At the beginning, they didn't have the information of contacts in other TB hospitals (the information was later provided by the local CDCs). The training remainders also need to be sent to the CDCs rst, and then forwarded to the local TB hospitals. In this way, the provincial hospitals could not be effective managers (Fig. 1). In Ningxia and Jilin, all institutions were noti ed of synchronous/asynchronous learning activities by provincial hospitals (including CDCs, hospitals, and TB dispensaries), while trainees at the township level were noti ed by county-level CDCs (Fig. 2).

Facilitators of E-learning CME
Facilitators of the training format In the interviews, most of the TB health workers agreed that the E-learning model is economical and timesaving compared to traditional face to face training model. In the past, only a few face-to-face training opportunities were available for TB health workers, especially for those from the city level institutions and below. The TB institutions usually send one or two doctors for training. They were asked to provide retraining for those who were not able to attend. In some cases, they even used meetings as a substitute for on-job training. A lack of trained teachers and inadequate learning materials made their learning quite di cult. E-learning have expanded access to high-quality continuing medical educational resources, and reduced its costs.
Compared with the traditional training model, the online and remote training are timesaving and more e cient. Traditional methods are not able to invite people come over frequently, and most hospitals can't afford the human and nancial costs. These two models, the E-learning models, have advantages in terms of convenience and economy. (Provincial TB health professional FGD in Zhejiang)

Facilitators of training content
In general, the training contents of the China-Gates Foundation Program were regarded as high-quality by the TB health workers. Teachers of the capacity building subproject are well-known experts in China, which provided great opportunities for city, county level and primary care health workers. The lower level health workers could receive training and technological guidance from the national experts, which is unlikely in traditional face-to-face training. Different health workers have different preferences: provincial and city-level TB health workers preferred the knowledge about disciplinary frontiers in synchronous Elearning activities, while county-level TB health workers favoured sections about routine clinical practice in asynchronous E-learning activities.
It's really nice to stay at home and have direct access to teachers from Beijing or across the country, teaching us the basic knowledge or cutting-edge ideas. It's brilliant. (Provincial TB health professional FGD in Jilin)

Barriers in the implementation of E-learning
Barriers in matching the supply and demand of training Although China-Gates Foundation Program provided TB staff with high-quality training contents, they were not evenly spread out among different types of TB health workers. In general, more courses were available to the clinical doctors: the total training loads including synchronous session once a week (or twice monthly) and nine modules of asynchronous sessions. However, the training loads for public health physicians and primary care workers was relatively small, with only seven and four modules received in three years respectively.
The uneven supply and demand were another key concern. Staff survey showed that clinical doctors thought they needed more training in the clinical diagnosis and treatment of TB (83.5% for multidrugresistant TB, MDR-TB, 78.4% for TB radiology and 70.1% for common TB), which perfectly matched with their training supplies (  The content of remote (synchronous) training is too deep to be used in clinical practice, which is beyond our competence, especially for county-level personnel. (County-level TB health professional FGD in Jilin)

Barriers in organizational coordination
The rst challenge in organizational coordination in training was the leaders did not pay enough attention to training. Interviews with project management personnel in Zhejiang and Ningxia provinces revealed that leaders paid relatively little attention to the capacity-building subprojects, so they could not mobilize adequate resources they needed. On the contrary, the areas where the project went smoothly with the higher level of participation, such as Jilin, tended to have higher levels of leaders' attention.
In our case, to be honest, we need to take care of everything. We are extremely busy. What do you think? For example, online training (asynchronous E-learning) is only a very small part of the China-Gates program, so we can't put lots of resources on it, the other works such as information technology have already taken up too much time, haven't they? And electronic medication monitoring (EMM), all of them are implementing right now. We can't be anywhere; this is our situation. (Key informant interview of CDC leader in Zhejiang) The remote (synchronous) training allowed us to hear lectures from national experts, not only in our hospital, but also at the city-level and below. The interaction between the Beijing Chest Hospital and our hospitals, including the tele-rounding, also worked very well, and everyone bene ted a great deal from it.

It's a great way to strengthen the capacity for our hospital and other TB institutions in Jilin. (Key informant interview of provincial TB designated hospital leader in Jilin)
The second challenge in organizational coordination was how the CDCs and TB designated hospitals collaborate on organizing training, providing feedback, and solving problems. Although China developed the new TB service delivery model -the CDCs, TB designated hospitals and primary care institutions work together in what is known as the "trinity model" -there was no coordinated working mechanism for TB continuing medical education, especially at the primary care level. It hindered further expansion of the high-quality learning resources by E-learning.

Barriers in internet technology
E-learning CME have placed higher demands on hardware and software. According to the survey, there were three kinds of hardware equipment for synchronous E-learning in hospitals of three provinces: the rst category was to have professional remote equipment, with a special remote centre as a training room, which existed only in a few provincial hospitals; the second category was accessing the remote platform by computer and duplicating the computer's display on the projector to share video with a group, which works for most city-level hospitals; the third category was using a personal computer in the participants o ce at home -most county hospitals fall into this category. The latter two forms of training space could hold only 3-5 people. In addition, a lot of technical issues arose when TB health workers received training, such as the slow processing speed of E-learning platform, network jams, system errors in learning progress, fuzzy sound and blurry videos. Technical failures have a signi cant negative impact on clinicians' participation in training.
Because we're not using a special-purpose device, sometimes the internet connection is not working. It happens sometimes that we could not join the (synchronous E-learning) classroom. (County-level TB health professional FGD in Jilin) "I've nished my study and the quizzes, but the system still showed that I haven't done my work. It will show you the answer to the quizzes and you can do it for the second time, but when I nished, it still not working. I didn't see any notice about this problem. The other problem is that the system could not distinguish the sections that you have learned and the sections you have not. You need to log in and nd out for yourself. So, I still don't get the certi cate now." (Provincial TB health professional FGD in Zhejiang) Another challenge came from the technology itself. Firstly, most medical staff feel that E-learning should not be a substitute for face-to-face training. One of the most obvious barriers is teacher-student communication, which is a great challenge in learning applied professions. Secondly, although they didn't have to leave their work space for E-learning activities, the synchronous learning sessions still required all participants to join the virtual classroom at the same time. Many medical staff re ected that the training schedule often con icted with their work schedule. For example, in many county-level hospitals, only one or two doctors were in charge of TB outpatient service, so they could not leave their o ce to participate in training during the working days.

Barriers in motivations
The lack of motivation was another source of barrier, which is deeply rooted in the health system. The incentive mechanism in current TB health service delivery system has a sustained negative impact on the enthusiasm for learning of TB health workers and the quality of the TB workforce. Firstly, the medical institutions in China largely depend on services revenues to cover operational costs. However, according to the national TB control program, the CDCs, TB dispensaries, and TB designated hospitals are funded by the government to provide free medical services, TB health workers have limited capacity to generate revenues. In this way, they have lower income and lack of work motivation. Moreover, participation in training has no association with the promotion of titles and bonuses, which makes medical personnel more reluctant to take part in training. There was a very common phenomenon called "training professional", which means one person in the department was asked to participate all the training, and signed up for every colleagues. Secondly, due to poor income, the TB workforce in the three provinces were generally older health workers with lower professional titles and less education compared with their colleagues in other departments. Their learning capacities and digital literacy were relatively weak. All these factors may lead to a low acceptance of E-learning among TB health workers, which has a negative effect on the implementation of China-Gates training project.
We are now implementing the new so-called "pay for performance" reform, we are redistributing the resources in our department, as known as the "secondary allocation". To our general section staff, we get even less income (than before). In this situation, your workload is not associated with your actual (revenue). When you are not satis ed with your revenue, you don't have the motivation to work, you just deal with everything, muddling along, that's all. ... our clinical physicians, especially our heads of the clinical departments, they seem to feel that, managing patients, performance, and getting more money -he feels that that's the point, training is not important. (Key informant interview of project management personnel in Ningxia) Since TB health personnel were more likely to be "passive learners" instead of "active learners", the Elearning activities in many institutions still rely on external incentives such as continuing medical education credits. Some project management personnel re ected that since the management of continuing medical education credits has recently become more stringent, it was more di cult for them to add some credits to TB health workers who attended the training. They could only try other strategies such as sending repeated noti cations to doctors and forcing them to participants. However, neither of them works well.

Barriers in project sustainability
Many stakeholders expressed their concerns about the sustainability of the training during the interviews. This argument was based on two main reasons: rst, the capacity to mobilize resources of the training organizers was very limited. Both national and provincial platforms are facing the tight constrains in terms of funding and human resources, which make it di cult to mobilize resources to continue the training after the program is completed. Second, due to the existing charging rules, the China-Gates program funds did not support equipment purchases or payments to teachers. According to the government regulations, hospitals cannot charge patients for telemedicine consultations either. The training organizers could only issue a small service fee to pay teachers through other funding. In this way, the platform has no existing model for funding to sustain itself.

Discussion
This study has aimed to analyse the barriers and facilitators to implementing E-learning in continuing medical education in three provinces of China. It found that the overall progress of capacity building subproject of the China-Gates Foundation TB Control Program has been relatively smooth. According to the pre-set plan, 98 synchronous learning activities were conducted by the national and provincial platforms. The registration rate of the asynchronous online platform for TB professionals in the pilot areas has exceeded 60%. In general, the project has achieved the goal of transferring high-quality medical educational resources to grass-roots medical institutions. However, the project still faces a range of challenges in the implementation, such as unmet training needs for public health and primary care workers, ine cient management framework, insu cient hardware and software, the unsupported environment, and the lack of an incentive mechanism. The current training model has not satis ed the needs for long-term, systematic, and timely training of TB professional.
To our knowledge, the capacity building sub-project of the China-Gates Foundation TB Control Program (Phase 3) is the rst time a large-scale E-learning program in TB continuing medical education has been implemented. Compared to the traditional face-to-face training, China-Gates compacity building project has several advantages in content and format of training: 1) This project invited national-level famous experts to give lectures, the content of lectures was revised by several waves of the experts' consultation with academics across the country. 2) TB medical personnel could stay at home to receive training from state-level experts and the high-quality educational resources has transferred to the grassroots level.
Although it will not replace face to face training in the future, we did see great feasibility and necessity of E-learning, especially during the COVID-19 pandemic (Ting, Carin, Dzau, & Wong, 2020). With the improved internet access in the future, it's possible that E-learning will be components of the routine practices in CME.
Based on previous studies, E-learning has two contradictory effects: rst, compared with traditional faceto-face learning, students in E-learning environment were more likely be inferred by online environment and lack of personal discipline, which has the negative effect on students' performance. This . Since the high-quality tutorial was one of the main facilitators for China-Gates capacity building program, we argue that the second effect played a more important role in our study -many health workers could not take in-person classes from national experts.
Our results highlight the lack of internal training incentives and low self-e cacy among TB personnel as one of the most important barriers. This phenomenon was caused by the policy environment and organizational culture, especially the incentive mechanism. In most study areas, the implementation of training programs still relied on external incentives, such as continuing medical education credits.
Extrinsic motivations (continued medical education credit, attendance requirement) are important in the short terms, but the long-term success of training project depend on whether we can engage health workers as active participants in their learning. How to establish and strengthen the virtuous cycle of intrinsic motivation is worth further studying.
Different types of training have their own advantages, which are in uenced by the type of course, the quality of the content, the communication between teachers and students, and the exibility of time and space. Existing evidence showed that compared with face-to-face learning, E-learning has stronger negative effect for applied professional courses (e.g. business, law, and nursing), but little impact on theoretical courses such as philosophy and psychology (Oreopoulos, Petronijevic, Logel, & Beattie, 2018).
It suggested that face-to-face training is more suitable for operational courses, which not only allows students to have hands-on opportunities, but also facilitates communication between teachers and students. However, for theoretical courses, quality educational resources are scarce, and medical personnel have to travel for training while they have work commitments. In this case, the advantages of E-learning became apparent. Therefore, E-learning and traditional face-to-face learning are complementary. The implementations of other pedagogies with a combination of face-to-face learning and E-learning in CMEs, such as blended learning (Morton et al., 2016), is a question for future studies.
Based on these ndings, we proposed ve policy recommendations. The rst is to improve the organizational structure of training. All TB institutions should pay more attention to training and to create a learning culture of teams. Second, to promote multi-sectoral collaboration. In the context of "the trinity model" and as institutions with hierarchical administration, CDCs should assist the designated hospitals in this process. More management and coordination functions should be transferred from CDCs to designated hospitals, enhancing their abilities and willingness to train and serve hospitals at lower levels. Third, to strengthen the supervision and establishing effective incentive mechanisms. We suggest to add continuing medical education credits at the design stage, allowing primary care workers to obtain national credits from the basic courses, and to decentralize the authority of credits in advanced courses to the provincial and municipal levels. Fourth, to enhance the user experience through technical assistance. We would like to highlight several key elements: optimizing the operating speed of the training platform, adding resume breakpoint and the video replay function, setting up a forum to facilitate teacher-student communication and interaction, developing mobile APPs, and conducting a survey on user behaviours in order to simplify the user interface and operation. Finally, the fth recommendation is to create customized and diverse learning experiences. Designing blended learning modules to combine the online and o ine courses needs to be considered. Doctors can selectively attend training sessions according to their own needs and schedules.
There are two main limitations in our studies: rst, this study was performed within a speci c grouphealth workers specialising in TB prevention and control, who were likely to have less than average interest due to the policy environment, and thus, the conclusion maybe inconsistent within other groups. In addition, we did not have the baseline information for synchronous E-learning participants, so we could not calculate the participation rate of synchronous E-learning activities as we did in the analyses of the asynchronous one.

Conclusion
In conclusion, our results indicated that it's feasible to conduct large scale E-learning continuing medical education activities with TB health personnel in China. Training content and format seem to be key facilitators of the program implementation, while the matching the supply/demand of training, organizational coordination, internet technology, motivations, and sustainability are key barriers. Further research should consider the theory behind the sperate effects more carefully in order to set up priorities for E-learning activities in CME.

Consent for publication
Not applicable.

Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Competing interests
The authors report no competing of interest. The authors alone are responsible for the content and Organizational structure of capacity building in Zhejiang