Framework of behavioral indicators evaluating TB health promotion outcomes: a modified Delphi study of TB policymakers and health workers
Infectious Diseases of Poverty volume 4, Article number: 56 (2015)
Although TB health promotion directed at policy makers and healthcare workers (HCWs) is considered important to tuberculosis (TB) control, no indicators currently assess the impact of such promotional activities. This article is the second in a series of papers that seek to establish a framework of behavioral indicators for outcome evaluation of TB health promotion, using the Delphi method. In the first article, we sought to establish a framework of behavioral indicators for outcome evaluation of TB health promotion among TB suspects and patients. The objective of this second article is to present an indicator framework that can be used to assess behavioral outcomes of TB health promotion directed at policy makers and HCWs.
A two-round, modified Delphi method was used to establish the indicators. Sixteen experts who were knowledgeable and experienced in the field of TB control were consulted in Delphi surveys. A questionnaire was developed following 4 steps, and involved ranking indicators on a five-point Likert scale. The consensus level was 70 %. Median, mode, and Coefficient of variation (CV) were used to describe expert responses. An authority coefficient (Cr) was used to assess the degree of each expert’s authority.
Consensus was achieved following the two survey rounds and several iterations among the experts. For TB health-promotion activities directed at policymakers, the experts reached consensus on 2 domains (“Resource inputs” and “Policymaking and monitoring behaviors”), 4 subdomains (“Human resources” among others), and 13 indicators (“Human resources per 100,000 person” among others). For TB health-promotion activities directed at HCWs, the experts reached consensus on 5 domains (“Self-protective behaviors” among others), 6 sub-domains (“Preventing infection” among others), and 15 indicators (“Average hours of daily workplace disinfection by ultraviolet radiation” among others).
This study identified a conceptual framework of core behavioral indicators to evaluate TB health-promotion activities directed at policymakers and HCWs involved in TB control. Validation in other parts of the world could lead to global consensus on behavioral indicators to evaluate TB health promotion targeted at policymakers and HCWs.
Please see Additional file 1 for translations of the abstract into the six official working languages of the United Nations.
Globally, the tuberculosis (TB) mortality rate has fallen by 41 % since 1990, and the world is on track to reach the global target of a 50 % reduction during 2015 . However, global TB control still faces many challenges, with an estimated 8.7 million incident cases in 2011 and 1.4 million deaths from TB since 2011. Progress in responding to multidrug-resistant TB (MDR-TB) remains slow , particularly in high-burden countries where the incidence of MDR-TB is unacceptably high [2, 3]. In addition, global economic crises and reduced investments in health services threaten national tuberculosis control programs [1, 4, 5]. TB control constitutes a global public good with benefits shared by everyone . Health policymakers and healthcare workers (HCWs) have important roles in efforts to control TB [7–10].
Given the population-wide benefits that are associated with investment in efforts to prevent the spread of TB, governments should play a leading role in TB control. Yet government neglect of TB control is one of the primary reasons for the worldwide persistence of the disease . The political will and commitment of a country to fight TB are essential for effective disease control [7, 8]. Lessons from TB control in China demonstrate the importance of political commitment: TB has historically been endemic in China, and progress in TB control was slow during the 1990s and early part of 21st century, mainly because of government neglect and limited healthcare resources [3, 5]. After 2003, however, China achieved great success in TB control due to significantly strengthened commitment and government leadership, increased funding, revised legislation related to TB case reporting, well-defined technical policies based on the Directly Observed Therapy-Short-Course (DOTS) strategy, and a modified free treatment policy . Currently, however, global financial crises and economic recessions have led to decreased funding for TB control, not only in China but also in other countries . Usually, in public health, it is through policy that technical expertise is translated into programs and interventions to promote the health and wellbeing of populations. Because policymakers work on behalf of governments to facilitate the development and implementation of programs and interventions, they play an important role in TB control. For these reasons, policymakers constitute a crucial target for TB health promotion, and it is crucial to identify a framework that can be used to evaluate the effectiveness of promotional activities targeted at them.
The performance of healthcare systems is closely related to the quantity, distribution, knowledge, skills, and motivation of their workforces, particularly those individuals delivering services . Constraints on these human resources have been reported as one of the main barriers to TB control . In 2003, national TB program (NTP) managers from 18 of the 22 TB high-burden countries ranked inadequate human resources (HR) as the first of the top five constraints against reaching global TB control targets set by the World Health Organization (WHO) . The WHO Global Plan to Stop TB 2006 – 2015 acknowledges that the main HR issues affecting tuberculosis control are insufficient quality, quantity, and distribution of HCWs . Awareness is increasing that HRs must be addressed in order to reach millennium development goals (MDGs) . In China, in addition to inadequate funding, a further difficulty in TB control is the shortage of trained HCWs. Many TB control facilities are staffed inadequately or by poorly trained and unmotivated HCWs . Strong evidence indicates that a lack of qualified HCWs is associated with diagnostic delay in China [14–20]. Thus, training HCWs adequately in working with TB patients and in advocating for new programs and policies is an important strategy in TB-endemic low and middle income countries. Although some reports have been published on assessing the capacity of HCWs, these have only assessed the quantity of healthcare workers in TB-control institutions, including number, educational levels, and professional titles [21–24]. There still remain a paucity of evidence-based indicator frameworks for evaluating health promotion interventions that target HCWs involved in TB control.
Health promotion focuses not just on individual knowledge and behavioral changes, but also on policy changes and capacity building . TB control needs multi-sector cooperation, including the concerted efforts of public health authorities, clinicians, policy makers, technical assistance agencies, laboratory specialists, and others . TB health programs should educate healthcare providers (both public and private), community members, public health officials, and policymakers on TB prevention and control . TB health-promotion programs should be comprehensive and include interventions for health service buyers (government/policymakers), health service providers (HCWs), health service users (TB suspects/patients), and the general public. To this end, in China, the 2008 Guidelines on Enforcement of Chinese Tuberculosis Control Program includes one chapter (Chapter 8) on comprehensive health promotion for health policymakers and HCWs in TB-control facilities, TB suspects and patients, contacts, students, migrants, and the public . Undoubtedly, indicators to evaluate the outcomes of any health intervention program are important  and should be identified during the planning stage . Although other major global diseases such as HIV/AIDS and non-communicable diseases have universally accepted and systematic indicator frameworks for assessing the outcomes of the health-promotion activities that address them [31, 32], global TB-control health promotion has no such indicators. In order to address this gap, our research group recently conducted a series of studies to identify frameworks of behavioral indicators to evaluate TB health-promotion outcomes. Evaluation indicators for each of the above target populations are complex and relatively independent. In our previous report, we sought to identify an indicator framework for assessing TB health-promotion activities targeted at TB suspects and TB patients . In this paper, we seek to address a gap related to the lack of indicators for assessing TB health promotion targeted at policymakers and HCWs. Although our earlier study  and the current study followed similar Delphi methodology involving the same study population, data were collected separately for TB patients and suspects, and for TB policymakers and healthcare workers. TB patients and suspects are a unique population; thus, it is important to report separately the behavioral indicators of TB promotional activities directed at them in order to inform policy and practice with a more focused understanding and effective use of data. Given the traditional lack of emphasis on evaluating policy efforts, it also helps to report separately on the findings for the two target groups. Such an approach also helps to ensure that the value of the findings for policymakers is not lost in the findings for TB patients and suspects.
This study was part of a larger project intended to explore a framework of behavioral indicators for evaluating TB health promotion targeted at various populations. We used a modified Delphi method described in a previous report  to establish an indicator framework to evaluate TB health promotion targeted at policymakers and HCWs. This method included two quantitative survey rounds that were completed from May to October 2012.
Selection of Delphi experts
Selection of experts for the Delphi survey was described in detail in a previous report . Here, we present a brief description of the selection process. Purposive sampling was used to select a panel of experts. Criteria included the following:
policymakers working in the field of TB control for at least 5 years;
TB HCWs with senior professional titles, significant knowledge of TB control, and extensive work experience in TB control in order to ensure expert authority; and
experts representing different geographic regions of China in terms of east, west, north, and south.
Sixteen experts were selected. Each received a phone call or an email requesting consent and describing the Delphi process and expectations regarding participation.
The instrument used for this modified Delphi method consisted of four parts, as reported previously : instructions for completing the survey, a questionnaire containing indicators for ranking, information used to evaluate expert authority in TB control, and general information and background about the experts.
Development of the Delphi survey instrument involved four steps (Fig. 1):
organizing TB experts to discuss and modify the questionnaire items, and developing the draft Delphi survey questionnaire with domains, subdomains, and indicators for pre-testing;
pre-testing the Delphi questionnaire with a convenience sample of 3 TB HCWs; and
modifying the Delphi questionnaire according to preliminary analysis of pre-testing, and forming the final Delphi questionnaire.
This process resulted in a group of 1 domain, 2 subdomains, and 4 indicators for health policymakers, and a group of 4 domains, 6 subdomains, and 13 indicators for TB HCWs, together with the operational definitions and data-collection sources for all potentially relevant indicators (Table 1). The experts were then asked to assess the importance and feasibility of each indicator using a five-point Likert scale .
Setting a consensus level
The Delphi method is based on panelists’ achieving consensus. However, experts can differ in interpretation and opinion, and it is typically difficult to gain 100 % agreement on all issues. In fact, no standard method or guidelines exist for determining appropriate consensus levels [44–51]. Therefore, the present study used a 70 % consensus level, as reported previously :
Consensus of inclusion: >70 % of participants scored the item ≥7;
Consensus of exclusion: > 70 % subjects scored the item ≤5; and
No consensus: item failed to meet either of the above criteria.
The procedure for administering the Delphi survey was reported previously . Participants were asked to rank the importance and feasibility of the indicators in the first round. These rankings were then analyzed, and results were sent back to the participants for review and ratification. Items achieving consensus for exclusion were directly excluded from the questionnaire for the second-round survey; items suggested for modification were revised and kept in the second-round survey; additional items suggested as indicators were added to the second-round survey. Participants were asked to re-rank the consensus results from the first round in the second survey. Responses in the second round that reached 70 % consensus were determined to be appropriate for creating the indicator framework. Finally, the framework was presented to the experts, who participated in the concluding discussion during which the items achieving final consensus were selected for inclusion in the ultimate framework.
Data analysis was undertaken by using the Statistical Package for the Social Sciences (SPSS) version 18.0. Median and mode were used to describe the central tendency of expert responses. Coefficient of variation (CV) was used to describe the variation in expert responses. The authority coefficient (Cr) was used to assess the degree of each expert’s authority, which was determined by judgment criteria for the indicator (Ca) and the expert’s familiarity with the indicators (Cs) .
The study was funded by the National Natural Science Foundation of China (Grant # 81001297). The project proposal was approved by the Institutional Review Board of Third Military Medical University, Chongqing, China. All experts who participated in Delphi survey signed an informed consent form to confirm their voluntary participation.
Characteristics of the experts
Two rounds of surveys were administered. During the first round, questionnaires were sent to seventeen experts, sixteen of whom responded. These sixteen content experts were from thirteen provinces/regions. All sixteen also completed the second-round survey. Descriptive information about the experts is available elsewhere .
Authority of experts (Cr)
The expert authority coefficient ranged from 0.91 to 0.92 with an average Cr of 0.92, which indicated that the experts had a high degree of authority on the indicators under evaluation .
Results of round-1 survey
The median score of the indicators’ importance and feasibility ranged from 7 to 9 for policymakers, except for a feasibility score of 5 on one indicator (“Number of policies TB control implemented”); modes ranged from 7 to 9; CVs were equal to, or less than 0.3, excepting modes of 0.4 and 0.5 for the feasibility of “Funds for TB control/person (RMB)” and “Number of policies TB control implemented,” respectively (Table 1). These results indicated that expert rankings showed a strong central tendency for all items in terms of importance and for most indicators in terms of feasibility, except for two indicators (“Funds for TB control” and “Number of policies TB control implemented”).
Both the medians and modes of the importance of two indicators for HCWs ranged from 7 to 9 among the experts; CVs were equal to, or less than 0.3, except for 0.4 for importance of one indicator (“Hand washing after appointment with each patient”) (Table 1). These results indicated that expert rankings showed a strong central tendency for all items in terms of feasibility and for almost all indicators in terms of importance, save for one indicator (“Hand washing after appointment with each patient”).
Indicator screening after first-round survey
Based on the consensus criteria listed earlier, indicators with a consensus score of >7 by >70 % of the experts were considered appropriate, and indicators with a consensus score of ≤5 by >70 % of experts participants were excluded. After the round-one survey, consensus on inclusion was achieved as follows:
1 domain (“Behaviors of policymakers”),
3 indicators (“Funds for TB control,” “Human resources for TB control,” and “Percentage of policies made for TB control”);
4 domains (“Self-protective behavior,” “Behaviors related to diagnosing TB patients,” “Behaviors related to treating TB patients” and “Behaviors related to health education targeted at TB patients”),
5 subdomains (“Preventing infection,” “Implementing clinic guidelines,” “Implementing standard treatment regimens,” “Regular patient visits,” and “Content of health education”);
11 indicators (“Hours/day of disinfecting workplace by ultraviolet radiation,” “Percentage of patients diagnosed according to clinic guidelines,” “Percentage of doctors ventilating workplace every day,” “Percentage of doctors wearing N-95 respirators,” “Percentage of patients using a standard treatment regimen,” “Percentage of patients covered by DOT,” “Percentage of patients receiving information on free treatment policy,” “Percentage of patients receiving information on importance of adherence to treatment,” “Percentage of patients receiving information on preventing transmission to others,” “Percentage of patients receiving information on regular follow-up sputum microscopy,” “Percentage of patients receiving information on side-effects of drugs”).
No indicators achieved consensus for exclusion (Table 1).
The results for policymakers indicated that all but one indicator (“Number of policies implemented”) in the first-round questionnaire were considered important and feasible. Experts also suggested that one indicator (“Number of policies implemented,” CV = 0.5 for feasibility) should be deleted, and one indicator (“Funds for TB control,” CV = 0.4 for feasibility), and one subdomain (“Input for TB control,” CV = 0.3 for feasibility) should be modified. Seven new items (1 domain, 1 subdomain, and 5 indicators) were recommended (Table 2). Ultimately, 2 domains, 4 subdomains, and 8 indicators were included for the second-round survey (Table 2).
As for the indicators for TB HCWs, experts suggested that one indicator (“Washing hands after appointment with each patient”) should be deleted from the second-round questionnaires, and 3 indicators should be modified. Experts also suggested that one domain (“Behaviors related to tracing TB suspects”), one subdomain (“Tracing TB suspects referred by non-TB control facilities”), and two new indicators (“Percentage of TB suspects successfully traced” and “Percentage of patients who received information on healthy lifestyle behaviors”) should be added (Table 2). Ultimately, 5 domains, 7 subdomains, and 14 indicators were included in the second-round survey questionnaire (Table 2).
Results of round-two survey
Results of the indicators in the round-two survey showed that items in the questionnaires for both policymakers and TB HCWs were both important and feasible (median or mode for importance and feasibility for all indicators ranged from 7 to 9; CVs for all indicators were equal to, or less than 0.3) (Table 2). The results indicated that the expert rankings had a strong central tendency for most items in terms of importance and feasibility.
Framework of behavioral indicators for assessing the impact of TB health promotion on health policymakers and TB experts
Based on our inclusion and exclusion criteria for the indicators for policymakers, consensus was achieved on the importance of all items and the feasibility of almost all items other than one indicator (“Percentage of meetings assigned to TB control in the workplace,” for which feasibility scored ≥7 by 62.5 % participants) (Table 2). No indicators achieved consensus for exclusion.
We organized a roundtable discussion with all the TB control experts to decide on final items for the framework. Following this discussion, “Percentage of meetings assigned to TB control in the workplace” was included in the framework, and 5 new indicators (“Percentage of bottom-up policymaking,” “Percentage of policies with measures to assess input, output, outcome, and impact,” “Percentage of policies with measures to promote cooperation of necessary sectors for policy implementation,” “Number of documents/policies for surveillance,” and “Percentage of evidence-based policymaking”) were added. Ultimately, 2 domains, 4 subdomains, and 13 indicators were included for policymakers (Table 3). For TB HCWs, inclusion consensus was achieved for all items in terms of both importance and feasibility based on consensus criteria (Table 2). Therefore, 5 domains, 7 subdomains, and 14 indicators were included for TB experts (Table 3).
Strong political commitment at various levels of government and significant resource development in health facilities in China have contributed to the success of the country’s TB program [3, 5, 9, 34]. The Chinese national Guidelines on Enforcement of the Chinese Tuberculosis Control Program (2008 version) emphasized health promotion targeted at both policymakers and HCWs . After 5 years of guidelines implementation, it is important to develop an evaluation framework to assess outcomes . Indicators are measures used to answer questions during the monitoring and evaluation of a health promotion intervention activity . This study sought to establish a framework of behavioral indicators for assessing the impact of TB health promotion activities directed at policymakers and HCWs. It is hoped that the findings are relevant to national TB control programs, especially in the current harsh economic climate, in which programs must ensure higher cost-effectiveness.
Sufficient financial and human resources are critical for TB control, and appropriate financial mechanisms should be developed to ensure that TB control projects, especially in resource-poor settings, are well supported . Political commitment and public policies on financial and human resource support at the national and local levels are among the primary support mechanisms for TB control [4–13]. Particularly for MDR-TB or extensively drug-resistant TB (XDR-TB) control, strong political commitment and adequate funding should underpin any strategy for addressing underlying societal and health-service determinants of MDR and XDR-TB . Policymakers in government roles can influence resource availability and use for TB control, including financial and human resources .
Overall, governments in every region of China play critical roles in supplying resources and organizing activities related to TB control. Once the central Chinese government has made political and financial commitments, it tends to honor them. However, this may not be the case with many local governments, especially at the county level in economically disadvantaged areas . The 2008 Guidelines on Enforcement of Chinese Tuberculosis Control Program stated that, in the domain of TB health promotion for policymakers, a primary objective is to improve their support of policy and finance . Therefore, the framework to evaluate policymakers’ behavior should address behaviors related to allocating resources to TB control. The framework proposed in this study included one domain on resource input, two related subdomains (“Human” and “Financial resources”), one indicator for human resources, and three indicators related to financial inputs.
Health promotion specialists increasingly recognize that, to be effective in improving citizens’ health and quality of life, they must incorporate policy advocacy interventions as integral strategies . Public policies at national and local levels are among the primary mechanisms for supporting TB control and can include financial support, human resource planning, health insurance, and even the reduction of TB stigma. In the past decade, TB control in China has benefited significantly from well-defined technical policies based on the DOTS strategy and from a modified free-treatment policy . However, policymakers still need to focus more on creating appropriate policy environments for TB control. For example, in order to address financial challenges, new policies can be considered, such as integrating the national TB control program into health insurance schemes . Other policy needs include using medical financing assistance for the poor to ensure access to TB control services for MDR-TB patients, using community health resources for primary health care in TB control , paying for TB care in public hospitals in order to discourage profiteering from service provision and illegal drug sales, and recognizing TB as an occupational disease to reduce job-related tuberculosis among HCWs . Therefore, indicators in this area should cover policymaking behaviors. This study proposed a framework with one such domain (of policymaking and monitoring,) and two subdomains (“Policymaking and monitoring of policy implementation”).
This study proposed one indicator to evaluate the number of policies made for TB control. Furthermore, while the quality of each policy is equally important, it is difficult to assess quality in policymaking and the impacts of policy interventions . We suggest satisfaction ratings as a way of assessing policy quality by both implementers and users. The question was posed as to whether an evidence-based or bottom-up approach employed in policymaking should be used to assess quality, because health research and policymaking in China usually operate in different environments without adequate communication . Although different policies might have different goals and objectives, policies should be proposed and enacted with related strategies for assessing input, output, outcomes, and impact . In addition, policy implementation involves multiple organizations, and within an organization, staff from different departments might be involved . All these actors have to make their own choices as to how to implement their respective parts of a policy. Therefore, policies should include strategies for encouraging cooperation among multiple sectors (such as allocating all resources necessary to implement a policy) . To this end, after final discussion with TB experts, four indicators (“Percentage of evidence-based policy making,” “Percentage of bottom-up policymaking,” “Percentage of policies with measures to assess input, output, outcome and impact,” and “Percentage of policies with measures to promote cooperation of necessary sectors for policy implementation”) were added to the framework for evaluating policymaking behaviors.
Monitoring is the continuous, systematic collection of data on specified indicators to provide indications of progress toward objectives and the achievement of intermediate results along the way. While effective monitoring is necessary for effective program management, it is not sufficient for assessing ultimate results . Therefore, this study proposed two indicators (“Percentage and number of times policy implementation was monitored” and “Number of documents/policies for surveillance”) under the subdomain of “Monitoring policy implementation.”
Doctors, nurses, and laboratory staff in TB-control health facilities must be trained properly and kept updated on the latest policy and healthcare developments related to TB . After training, it is important to evaluate the contribution of that training to improving both HCWs' productivity and the overall quality of tuberculosis control programs . However, little data is available on human resources for TB control, particularly data on the quality of these resources. One survey on human resources for TB control among 19 high TB-burden countries (HBCs) in 2003 assessed staff numbers, skills (training courses and coverage of training), performance (the estimated time needed to treat a new smear sputum positive patient), and estimated HR gap . This survey indicated that many HBC countries do not have accurate information on numbers, types, and distribution of staff involved in TB-control activities; staff attendance at training courses; or characteristics, duration, and intensity of training activities, particularly in China . In China, available studies on the assessment capacity of HCWs [21–24] only included number, education levels, and professional titles. No studies have assessed actual skills related to TB control. The aims of TB health promotion for HCWs in the 2008 Guidelines included correct and timely diagnosis of TB patients, treatment of TB patients with a standard regimen, carrying out TB health education to TB suspects or patients, and precautionary practices to prevent infection among HCWs . Thus, the 5 domains proposed in the indicator framework for HCWs (“Self-protective behaviors,” “Behaviors related to TB diagnosis,” “Behaviors related to tracing of TB patients,” “Behaviors related to TB treatment,” and “Behaviors related to health education”) are relevant and appropriate.
Measures effective in preventing HCWs from being infected with TB included updated periodic training to maintain awareness of potential risks and appropriate use of effective respiratory protection as well as active infection control procedures . There are three levels of infection control measures within health care facilities: administrative (managerial), environmental, and personal . Among those measures, ventilation, Ultraviolet germicidal irradiation (UVGI), and the use of respiratory protective equipment can be managed by HCWs. Thus, the framework proposed by this study included one subdomain (“Breaking the chains of transmission”) and 3 indicators.
Strengths and limitations
This study was the first to develop a framework of behavioral indicators for TB health promotion among policymakers and HCWs. In addition, after two sound surveys, consensus for inclusion was achieved on all items except one (62.5 % consensus for the feasibility of “arrangement of TB control in the health worker”). Yet, the study has a number of limitations:
Additional experts for the Delphi survey could have been contacted for better representation. The study only included policymakers from TB control institutes, but did not include experts in other sectors whose activities influence policy and resources for TB control, including policymakers in the Bureau or Ministry of Finance, or policymakers in the Ministry of Civil Affairs. In addition, HCWs at the county level were not included on the expert panel. It is necessary to tailor the framework of indicators to HCWs at different levels of the health system (such as county/district and primary care) because HCWs at different levels have different responsibilities in relation to TB control .
The study focused on constructing indicators to evaluate behavioral changes of frontline clinic HCWs only. It is equally important to consider indicators for HCWs in different departments within TB-control health facilities (nurses, laboratory staff, and health promoters) because their responsibilities vary.
As a mostly qualitative approach, Delphi surveys typically provide a means of structuring group discussions, raising issues for debate, and identifying questions for further empirical inquiries that enhance reliability and validity . Therefore this study organized roundtable discussions to decide on the final items included in the framework after the second-round survey. Though this study organized a “consensus conference” to discuss the validity of its Delphi results, the framework was not tested by a quasi-experimental design, a series of interviews, secondary documentary data collection, or focus-group discussions.
The title “framework” implies a comprehensive system or structure in TB care and control. One important item which our present study did not address, and which will benefit from future research attention, is the issue of TB-contact investigation.
Conclusion and implications
Evaluating health promotion programs is challenging, as health promotion takes place within fluctuating and complex settings . Consequently, implementation research should be carried out to tailor TB health-promotion frameworks to varying contexts. In addition, further studies are needed to construct indicators for evaluating TB health promotion at the individual level, including behavioral changes among HCWs in departments other than clinical settings (nurses, laboratory staff), and at environmental levels. The results of this study can be used as a basis for further research. However, before any performance indicator can be adopted, it needs to be defined clearly and tested for reliability, validity, and responsiveness (the ability to detect a significant change in performance). Therefore, more studies are desirable for testing the framework by a quasi-experimental design, a series of interviews, secondary documentary data collection, and focus-group discussions [53, 54].
World Health Organization. Global tuberculosis report 2013. http://apps.who.int/iris/bitstream/10665/91355/1/9789241564656_eng.pdf?ua=1. Access 2014 June 2
Zhao Y, Xu S, Wang L, Chin DP, Wang S, Jiang G, et al. National survey of drug-resistant tuberculosis in China. N Engl J Med. 2012;366:2161–70.
Wang Y. Report of the Fifth National Sampling Survey of TB Epidemiology. Beijing: Military Medical Science Press; 2011. p. 1–7.
Abubakar I, Zignol M, Falzon D, Raviglione M, Ditiu L, Masham S, et al. Drug-resistant tuberculosis: time for visionary political leadership. Lancet Infect Dis. 2013;13:529–39.
Jia Z, Cheng S, Wang L. Tuberculosis control in China: striving for sustainability. Lancet. 2012;379:2149.
Jack W. The public economics of tuberculosis control. Health Policy. 2001;57:79–96.
World Health Organization. An Expanded DOTS Framework for Effective Tuberculosis Control. WHO/CDS/TB/2002.297. 2002. http://apps.who.int/iris/bitstream/10665/67232/1/WHO_CDS_TB_2002.297.pdf. Access 2014 June 3.
World Health Organization. WHO tuberculosis programme: framework for effective tuberculosis control. WHO/TB/94.179. Geneva: World Health Organization; 1994.
Wang L, Liu J, Chin DP. Progress in tuberculosis control and the evolving public-health system in China. Lancet. 2007;369:691–6.
World Health Organization. WHO report 2004 Global tuberculosis control: surveillance, planning, financing. Geneva: WHO; 2004.
World Health Organization. The global plan to stop TB 2006–2015 Stop TB Partnership. Geneva: WHO; 2006.
Figueroa-Munoz J, Palmer K, Poz MR, Blanc L, Bergström K, et al. The health workforce crisis in TB control: a report from high-burden countries. Hum Resour Health. 2005;3:2.
Awofeso N, Schelokova I, Dalhatu A. Training of front-line health workers for tuberculosis control: lessons from Nigeria and Kyrgyzstan. Hum Resour Health. 2008;6:20.
Atun R, Weil DE, Eang MT, Mwakyusa D. Health-system strengthening and tuberculosis control. Lancet. 2010;375:2169–78.
Li Y, Ehiri J, Tang S, Li DK, Bian YQ, Lin H, et al. Factors associated with patient, and diagnostic delays in Chinese TB patients: a systematic review and meta-analysis. BMC Med. 2013;11:156.
Huan ST, Zhang B, Yan F, Liu XY, Duan MHJ, Zhao FZ, et al. Analysis on patient delay and reasons for pulmonary tuberculosis in poor rural. J Chinese Antituberculosis Assoc. 2007;29:70–3. 23.
Wu ZJ, Zeng RY, Shen LT, Zhang JF. Delay in identification of PTB in Shang Yang city. Pract Prev Med. 2008;15:448–9.
Li AX, Wang YL. Investigation of diagnostic delay of TB in Shang Qiu city. Henan J Preven Med. 2002;13:292.
Zhou YZ, Shen XB, Shi XQ, Lei PY, Li J, Wang SP, et al. Analysis on causes and countermeasures of delay diagnosis of pulmonary tuberculosis patients in North GuiZhou. Modern Prevent Med. 2009;36(4462–4464):4469.
Liu HM. Survey on 290 smears positive PTB. J Hengyang Med Coll. 2000;28:416–7.
Xie HB, Zhang H, Li X, Wang LX, Jiang SW. Analysis of the current human resource situation in the nationwide tuberculosis control and treatment organizations. Chinese Journal of Antituberculosis. 2011;33:12–5.
Jiang L, Zhong Q, Yin JJ, Li JW. Analysis on the human resources status in antituberculosis institutes of Guangdong province. Chin J Nat Med. 2008;10:106–9.
Chen X, Wang YL, He XX, Zhang HW, Zhang TH, Guo HJ, et al. Human resource survey and staffing assessment of Beijing district level tuberculosis institutions. Chinese Journal of Antituberculosis. 2012;V34:580–4.
Liu XJ, Zhang P, Cheng YF, Zhou P. Assessment of status of the human resources of county level TB dispensaries in YiChang city. Chinese Journal of Antituberculosis. 2013;V35:134–7.
Health promotion. The Ottawa Charter for Health Promotion. Ottawa: First International Conference on Health Promotion; 1986. http://www.who.int/healthpromotion/conferences/previous/ottawa/en/index1.html Access 2014 June 2.
World Health Organization. What is DOTS? A guide to understanding the WHO recommended TB control strategy known as DOTS. 1999. http://extranet.who.int/iris/restricted/bitstream/10665/65979/1/WHO_CDS_CPC_TB_99.270.pdf. Access 2014 June 2.
CDC. Tuberculosis elimination revisited: Obstacles, opportunities, and a renewed commitment. MMWR 48 (No. RR 9). 1999. http://dph.georgia.gov/sites/dph.georgia.gov/files/TB-Pub_Control_TBElimrevisited.pdf. Access 2014 June 2.
Department of Disease Control of Ministry of Health. Guideline on Enforcement of Chinese Tuberculosis Control Program. Beijing: Ministry of Health; 2008.
de Souza EM. Evaluation methods in health promotion programmes: the description of a triangulation in Brazil. Cien Saude Colet. 2010;15:2521–32.
Round R, Marshall B, Horton K. Planning for effective health promotion evaluation, Victorian Government Department of Human Services, Melbourne. 2005. http://docs.health.vic.gov.au/docs/doc/32F5DB093231F5D3CA257B27001E19D0/$FILE/planning_may05_2.pdf. Access 2014 June 2.
World Health Organization. National AIDS programmes: a guide to indicators for monitoring and evaluating national HIV/AIDS prevention programmes for young people. In: ISBN 92 4 159257 5. 2004. http://www.who.int/hiv/pub/epidemiology/napyoungpeople.pdf. Access 2014 June 2.
World Health Organization. Global Strategy on Diet, Physical Activity and Health: A framework to monitor and evaluate implementation. 2008. http://www.who.int/dietphysicalactivity/Indicators%20English.pdf. Access 2014 June 2.
Li Y, Ehiri J, Hu D, Zhang Y, Wang Q, Zhang S, et al. Framework of behavioral indicators for outcome evaluation of TB health promotion: a Delphi study of TB suspects and TB patients. BMC Infectious Diseases. 2014;14:268.
Tang S, Squire SB. What lessons can be drawn from tuberculosis (TB) control in China in the 1990s? An analysis from a health system perspective. Health Policy. 2005;72:93–104.
Schwartz R, Goodman R, Steckler A. Policy Advocacy Interventions for Health Promotion and Education: Advancing the State of Practice. Health Educ Behav. 1995;22:421.
Chai SJ, Mattingly DC, Varma JK. Protecting health care workers from tuberculosis in China: a review of policy and practice in China and the United States. Health Policy Plan. 2013;28:100–9.
Leeman J, Sommers J, Vu M, Jernigan J, Payne G, Thompson D, et al. An evaluation framework for obesity prevention policy interventions. Prev Chronic Dis. 2012;9:E120.
Jiang F, Zhang J, Shen X. Towards evidence-based public health policy in China. Lancet. 2013;381:1962–4.
Pressman JL, Aaron W. Implementation. How Great Expectations in Washington Are Dashed in Oakland. 3rd ed. Berkeley: University of California Press; 1984.
Department of Policy and Evaluation Millennium Challenge Corporation. Policy for Monitoring and Evaluation of Compacts and Threshold Programs. 2012. http://www.mcc.gov/documents/guidance/policy-050112-monitoring-and-evaluation.pdf. Access 2014 June 2.
Inter-professional ICN/IFRC/IHF/WMA seminar. Report National Training Course on Tuberculosis Infection Control and Health Care workers’ safety in China. 2011. http://www.wma.net/en/20activities/30publichealth/70tuberculosis/Report_of_National_Training_Course_on_Tuberculosis_Infection_Control_fianl.pdf. Access 2014 June 2.
Medical Center Occupational Health Section and Occupational and Environmental Lung Disorders Committee. Protecting health care workers from tuberculosis. J Occup Environ Med. 2008;50:852–5.
World Health organization. WHO Policy on TB Infection Control in health care facilities, congregate settings and households. WHO/HTM/TB/2009.419. 2009. http://apps.who.int/iris/bitstream/10665/44148/1/9789241598323_eng.pdf. Access 2014 June 2.
Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006;53:205–12.
Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32:1008–15.
Mitchell VW. The Delphi technique: An exposition and application. Technology Analysis & StPercentage of patientsgic Management. 1991;3:333–58.
Hsu CC, Sandford BA. The Delphi technique: making sense of consensus. Practical Assessment research &Evaluation. 2007;12:1–8.
Loughlin KG, Moore LF. Using Delphi to achieve congruent objectives and activities in a paediatrics department. Journal of Medical Education. 1979;54:101–6.
McIlrath C, Keeney S, McKenna H, McLaughlin D. Benchmarks for effective primary care-based nursing services for adults with depression: a Delphi study. J Adv Nurs. 2010;66:269–81.
Green B, Jones M, Hughes D, Williams A. Applying the Delphi technique in a study of GPs information requirements. Health and Social Care in the Community. 1999;7:198–205.
Green PJ. The content of a college-level outdoor leadership course. Spokane: Paper presented at the Conference of the Northwest District Association for the American Alliance for Health, Physical Education, Recreation, and Dance; 1982.
Rootman I, Goodstadt M, Hyndman B, McQueen D, Potvin L, Springett J, et al. Evaluation in health promotion, principles and perspectives. WHO Regional Publications, European Series; No.92. 2001. http://files.eric.ed.gov/fulltext/ED460081.pdf. Access 2014 June 2.
McKenna HP, Parahoo KA, Boore J. The evaluation of a nursing model for long-stay psychiatric patient care Part 2; presentation and discussion of findings. International Journal of Nursing Studies. 1995;31:95–113.
McKenna H, Hasson F, Smith M. A Delphi survey of midwives and midwifery students to identify non-midwifery duties. Midwifery. 2002;18:314–22.
This project was funded by the National Natural Science Foundation of China (Award # 81001297). The authors would like to thank participants on the expert panel for the discussion during the process of proposing the framework. The authors would like to acknowledge Professor Shenglan Tang of the Institute of Global Health, Duke University for his comments on the paper.
The authors declare that they have no competing interests.
YL designed the study. DH and YL designed the instrument for data collection and organized expert discussion; YL organized and analyzed data; DH and JC contacted the experts for Delphi survey. YL and JE drafted and edited the manuscript. EO gave suggestions on the manuscript draft and provided valuable edits. All authors interpreted the results, revised the report, and approved the final version.
About this article
Cite this article
Li, Y., Ehiri, J., Hu, D. et al. Framework of behavioral indicators evaluating TB health promotion outcomes: a modified Delphi study of TB policymakers and health workers. Infect Dis Poverty 4, 56 (2015). https://doi.org/10.1186/s40249-015-0087-4
- Health promotion
- TB healthcare workers
- TB HCWs
- Delphi method