Development and expert validation of a ‘Data-to-Action Continuum’ to measure and advance the data-use capabilities of national tuberculosis programs

The Global Accelerator to End Tuberculosis (hence referred to as the Accelerator) model of the United States Agency for International Development (USAID) aims to build local commitment and capacity to support countries in reaching global United Nations High-Level Meeting (UNHLM) targets of diagnosing and enrolling 40 million people on tuberculosis (TB) treatment and 30 million on TB preventive therapy (TPT) by 2022.¹ Key to achieving these ambitious goals and to monitoring implementation of the Accelerator is ensuring that countries have an effective system for data collection, analysis, and use. Many resources (tools, methods, frameworks) exist for improving data use for strategic and programmatic purposes; however, they are not always well adapted to local needs.^1,2 The selection of an appropriate approach or tool, adapted to the country context, is imperative for effective interventions to transform data into action.³ Each country and program are unique, making it difficult to choose the right interventions targeting decision-makers at various levels. Moreover, to monitor the implementation of the Accelerator and ensure that countries use data for decision-making, their data use capabilities must be measured, monitored, and strengthened in accordance with the Accelerator’s Performance-Based Monitoring and Evaluation Framework (PBMEF).⁴ The PBMEF is a key component of the Accelerator’s efforts to ensure effective accountability and monitoring of progress in reaching specific country and global TB milestones and targets.

Increased capability to analyze, visualize, and use TB data throughout the decision-making process would enable national TB programs (NTPs) to transform data to action. However, lack of a validated tool that identifies the strengths and weaknesses of necessary characteristics at each stage of the data-to-action process limited NTPs’ ability to identify and improve relevant capabilities. The NTP leadership and management team working to strengthen data systems and data use capabilities could benefit from a holistic maturity model⁵ with a multidimensional approach to measuring maturity—including items for strategic, organizational, technical, and people-related capacity for both current and desired maturity.⁶ To develop a holistic maturity model, following a systematic review, a set of 16 data-to-action measurement dimensions were identified and defined.⁷ An important next step was to describe the characteristics of each of the data-to-action dimensions across a continuum and facilitate expert validation to finalize the dimensions, continuum levels, and description of dimensions across all the levels with a focus on NTP users at the national, subnational, district, and facility levels (e.g., the monitoring and evaluation [M&E] director, NTP manager, facility managers, laboratory manager, etc.). To enable contextual decision-making, a data-to-action continuum (D2AC) scale was developed, following a user-centered and iterative design process, by a core group of the TB Data, Impact Assessment and Communications Hub (TB DIAH) project under the guidance of an advisory group consisting of international and country-level TB and data system experts. The D2AC targets key information user groups within the NTP (e.g., the national program manager, regional laboratory manager, etc.) while considering the perspective of TB landscape stakeholders such as donors and technical assistance partners (international and local nongovernmental organizations [NGOs], multinationals, etc.).

With appropriate and improved data-to-action capabilities, NTPs should be better able to collect, protect, manage, analyze, use, and disseminate TB and related data in a timely manner so that community health workers, healthcare providers, supervisors, managers, and policymakers can more effectively make decisions, track progress, and provide feedback on NTP performance to improve TB health outcomes across the care continuum. To strengthen the use of data for action, it is necessary for the NTP and stakeholders to know the state of their data-to-action capabilities on a maturity continuum which specifies people, processes, technology, and institutional characteristics at each level of the continuum, and to understand what is needed to enable decision making by users at different health system levels of the NTP. A list of illustrative questions is presented in Table 1.

Conceptual framework

Building on the maturity model research in the information system domain,⁸ data demand and use work under the USAID-funded MEASURE Evaluation project,⁹ and drawing from the Health Information System (HIS) Strengthening Model¹⁰ and the Performance of Routine Information System Management (PRISM) framework,^11,12 the D2AC conceptual framework (Figure 1) describes organizational, people, technology, and process-related factors affecting data-to-action capabilities embedded within the larger health system and HIS of a country. The framework highlights an interlinked and cyclical journey involving data collection, reporting, data analysis, and the use of related interventions which build on the leadership, governance, accountability, and capacity-building efforts of the NTP and the country’s information and communications technology (ICT) system and infrastructure.¹³ The framework shows that the interlinked interventions follow a continuous improvement approach to advance along continuum levels—often identified by a descriptor such as nascent, defined, established, institutionalized, and optimized—which are associated with an improvement of NTP performance in terms of using data for proactive and responsive clinical, programmatic, managerial, and policy decision making.

Figure 1.D2AC conceptual framework

Significance

The D2AC framework applies current thinking about situational maturity model design¹⁴—which, ideally, identifies a gap between actual and intended functioning that can then be closed by successful improvement activities—while also considering people’s capabilities to perform the suggested activities. The TB DIAH D2AC activity team developed the D2AC Toolkit, preserving the idea of the “data-to-action continuum” in the name given the dynamic interaction among health system components and multiple decision points across the TB care continuum. Despite growth in the number of maturity models, limited information is available on how these models were developed.^8,15 The Toolkit includes two tools; (1) a data collection tool that features an instrument with a measurement scale and (2) a data analysis tool that includes a visualization workbook with a list of priority actions derived from the continuum and an implementation plan template. Accompanying the two tools is a user guide to facilitate the self-administration of the tools.

The objective of this paper is to describe the D2AC Toolkit development process and present the toolkit validated by a group of international and country-level TB and HIS experts. The expert validation will be followed by field tests at the country level before finalizing and publishing the Toolkit and translating it for wider use.

METHODS

The methodological approach adapted Becker, et al.'s procedural model for the development of the D2AC.¹⁶ The procedural model recommends specific steps such as problem definition, maturity model review, model development strategy, iterative design, pilot, and use of maturity model. A follow-up paper will discuss the pilot and maturity model implementation steps.

Literature review

A systematic literature review identified and defined a set of 16 data-to-action measurement dimensions. These dimensions responded to the research question that aimed to identify appropriate dimensions required to improve NTP capabilities for transforming data to action.⁷ A seventeenth dimension was added later in the D2AC scale development process.

Iterative development phases

Building on the literature review findings, the Toolkit development process was led by a core team of the TB DIAH project and conducted in two phases. Each phase followed an iterative approach and had weekly development cycles focused on intermediate deliverables contributing to the final D2AC Toolkit. The deliverable from each weekly sprint was presented to and discussed with the project leadership team, comprised of the project director, deputy director, and a TB M&E and surveillance advisor, to ensure alignment with the D2AC requirements. The first phase was conducted from December 2020 to March 2021, followed by a validation workshop in April 2021. The March 2021 draft of the D2AC Toolkit included the following components: five continuum levels, a country profile template, the D2AC scale, user roles, and a glossary (described below). The D2AC Toolkit was updated in the second phase (May–September 2021) based on the workshop feedback and finalized for the second workshop, where the Toolkit would be pretested in October 2021. The second phase also applied an agile approach similar to that of the first phase. The second draft of the Toolkit included the following components: five continuum levels, a country profile template, the D2AC scale, user roles, a glossary, and the newly added data collection instrument and data analysis and visualization workbook.

Validation workshops

Workshop purpose and expert identification

As the next step in our method, the D2AC core team conducted a validation workshop in April 2021 and a pretest in October 2021. The purpose of the two workshops was to facilitate the review and validation of the Toolkit by engaging experienced or specialized participants in TB and/or data use.¹⁷ The feedback rubrics developed for the validation workshop and pretest are included in Appendices I and II, respectively. To capture diverse inputs, the workshop included three groups of participants,¹⁸ namely:

• Stakeholders: those who are involved and have a stake in the TB program;

• Experts: individuals with experience and expertise in the TB program and/or the use of data for decision-making in the context of HIS, health system planning, and M&E; and

• Facilitators: members of the D2AC core team who could answer questions, guide conversations, and synthesize data captured in the workshop.

Following a consultative process involving the TB DIAH and USAID teams, the D2AC core team identified a group of ten key informants to participate in the workshops and form a D2AC Advisory Group. The D2AC Advisory Group represented six projects or organizations, seven countries, and the global level (Table 2).

Workshop preparation

Each advisory group member was provided with a reviewer packet (Online Supplementary Document, Appendix III) developed by the D2AC team, which included background information on the D2AC project activity, the advisory group terms of reference, and a detailed explanation of the review process objectives, scope, meeting schedule, review timeline, feedback submission process, and person of contact.

Virtual workshops

The validation workshop and the review process were conducted virtually over a period of four weeks due to the COVID-19 pandemic.

In the phase I validation workshop, the team held introductory calls with ten reviewers and shared the D2AC overview presentation and feedback rubric after the calls. Ten reviewers provided feedback. Feedback was synthesized and discussed with the reviewers in a subsequent consensus-building meeting.

The phase II pretest workshop followed a similar approach. The introductory call was held on October 6, 2021. The PowerPoint slide deck presented during the introductory call, a recording of the presentation, and a feedback rubric (available as a Word document or a Google Form) were sent to all reviewers. Reviewers were given two weeks, until October 20, 2021, to review the D2AC Toolkit and submit feedback. Five feedback rubrics were received, and the D2AC team compiled all feedback received.

Data management and analysis

All feedback received, in each phase, was compiled and aggregated into a primary rubric by three members of the D2AC team, and a PowerPoint slide deck was developed presenting the main takeaways and detailed aggregate results for each question on the feedback rubric and element of the tool. These slides were presented to the internal D2AC team and then adapted for presentation to the workshop participants and USAID advisors.

The group introductory calls and the consensus meetings were recorded, and any feedback, questions, clarifications, or discussion points pertinent to updating the D2AC Data Collection Tool were transcribed. The meeting transcripts, notes from the one-on-one introductory calls, and the individual feedback rubrics were all carefully reviewed for two full-day workshops of the D2AC core team on April 23 and 30, 2021 and during the D2AC team’s weekly meetings following the October 20, 2021, receipt of the five feedback rubrics.

The objectives of the core team workshop were to conscientiously review the feedback collected, identify actions points (accepting or rejecting suggestions, with justification) and consensus areas, and update the Toolkit components that included the D2Ac conceptual framework, the country profile, the D2AC scale domains, subdomains, and capability statements, the user roles, the data collection instrument, the data analysis and visualization workbook, the glossary, and the implementation plan template.

RESULTS

This section describes the outputs of the D2AC Toolkit validation achieved by engaging relevant experts. The validation process did not focus on the Toolkit User Guide; hence, it is not presented in this paper.

D2AC: an overview

The continuum allows stakeholders to measure status, identify maturity pathways, and develop a roadmap to advance D2AC capabilities for achieving NTP goals. The goals are aligned with the user roles that vary from national to subnational, district to facility, and community levels. The expert-validated D2AC Toolkit includes:

• The MS Excel (Microsoft Inc, Seattle WA, USA) based D2AC Data Collection Tool (Online Supplementary Document, Appendix IV) for collecting individual responses with:

  • The D2AC conceptual framework

  • The five defined continuum levels

  • A country profile template to collect socioeconomic, demographic, and epidemiological indicators

  • A D2AC scale, with capability statements organized into five domains and 17 subdomains for each of the five continuum levels

  • Key user roles and decisions organized according to USAID’s TB objectives of reach, cure, prevent, and sustain

  • A glossary

  • A data collection instrument

  • An analysis matrix

  • An analysis dashboard

• An Excel-based D2AC Data Analysis Tool (Online Supplementary Document, Appendix V) that automatically aggregates responses from all completed data collection instruments and generates data visualizations and recommended priority actions. This enables decision-makers to interpret and apply the findings and develop an implementation plan using the template provided by D2AC.

D2AC Data Collection Tool

It should be noted the Data Collection Tool collects stakeholder opinions about whether elements of the TB HIS are present, functional, and performant. A performing TB HIS will facilitate data generation when data are compiled into the monthly report (i.e., indicator values for specific time periods. At the district, regional, and national levels, data generation would occur when district, regional, and national level totals are created for indicators and time periods.

D2AC continuum levels and their descriptions

The D2AC scale includes five levels with specific descriptions (Table 3). The five levels are named nascent, defined, established, institutionalized, and optimized. The “optimized” level and its description was changed from “autonomous/self-reliant” based on feedback received in the first review phase. The maturity descriptions guide the characterization of subdomains across the five continuum levels.

D2AC country profile

The country profile is intended to contextualize the D2AC maturity of a country in the context of its overall health system situation and avoid misinterpretation of the continuum level, especially for cross-country comparisons. The D2AC country profile includes information on demographics, geography, socioeconomic status, TB epidemiological burden and trend, NTP laboratory and workforce capacity, and TB health financing, as well as research and development indicators. Post validation, gross national income and gross domestic product rates were removed from the list of socioeconomic indicators. The feedback led to a revision of epidemiological burden and trend-related indicators with the inclusion of TB case notification rate, TB treatment coverage, TB treatment success rate, multidrug-resistant (MDR) and rifampicin-resistant (RR) TB incidence, MDR/RR-TB treatment enrollment rate, extremely drug-resistant TB incidence, HIV coinfection rate, and TPT coverage indicators. Furthermore, two of the World Health Organization (WHO) impact indicators—reduction in TB incidence rate (over time) and reduction in TB deaths (over time)—were removed. The laboratory center indicator was specified to capture the total number of laboratories conducting TB diagnosis and the workforce capacity indicator list was expanded to include NTP M&E staff supported by the government, and resources allocated towards M&E or TB M&E. The user fee indicator was removed from the health financing category and the health insurance indicator was expanded to capture specific information for the proportion of the population with TB who received social protection under the national health insurance scheme, the proportion of the health budget allocated to TB services, the proportion of the annual TB budget funded by donors, the proportion of domestic TB financing, and the proportion of cases that led to catastrophic costs due to TB. A new category was added to capture TB research and development data.

D2AC scale: domains and subdomains

The D2AC scale includes five domains and 17 associated subdomains (Table 4). Post-validation, the “data collection” domain was revised to become “data collection and reporting,” and a new associated subdomain of “reporting” was added. The “analysis and dissemination” domain became “data analysis and use.” The “data use policy” subdomain was updated to “data use guidance.” The capacity building subdomain of “data use capacity” was updated to “data interpretation,” the “health workforce and capacity development” subdomain was deleted, and a new subdomain of “skill and knowledge development” was added. The “system” domain was revised to “information and communications technology.” The definition of each of the subdomains is included in the tool. The D2AC scale describes the characteristics of each of the subdomains across each of the five continuum levels.

Key user groups and decision requirements

The D2AC user roles were grouped together based on the decision-making needs of different TB stakeholder groups and the User Roles tab in the tool was meant to enable respondents to assess how well the users’ information needs—organized according to USAID’s TB objectives of reach, cure, prevent, and sustain—are met by the current system. The user groups were healthcare provider, health facility/clinic manager, laboratory manager/technician, district TB coordinator/manager/health officer, regional TB coordinator/manager, regional laboratory manager, NTP manager/policymaker, national M&E director/manager, and national reference laboratory manager. Two new user groups—community health workers and TB advocates, civil society, and media—were added post-validation.

Data collection instrument

The purpose of the data collection instrument is to guide the evaluation of data use capabilities to routinely monitor and improve data use attributes associated with TB program management and service delivery at the facility, district, subnational, and national levels. The validated data collection instrument includes 44 questions (Online Supplementary Document, Appendix VI (Table S1)). The number of questions for each of the subdomains varies.

D2AC analysis matrix and dashboard

Each respondent can see the overall score for each domain, each associated subdomain (Figure 2), by level (see Figure 3), and for each question. The dashboard offers a visual summary to individual or group respondents. Figures 2 and 3 present sample dashboard visuals based on mock data to illustrate what the aggregate dashboards would look like to users in the context of implementation.

Figure 2.Overall aggregate score by subdomain

Figure 3.Aggregate subdomain scores within domain 3 for the regional level

D2AC Data Analysis Tool

An Excel-based D2AC Data Analysis Tool aggregates the responses from the data collection instruments completed individually or in groups during a D2AC assessment and includes tabs showing data visualizations for overall results, results by level of the TB program, and results by domain. The tool also provides customized priority actions for improvement by subdomain based on the average scores (and hence, continuum levels) achieved, an implementation plan template to be filled out during the assessment, and the D2AC scale and glossary for reference. This enables decision makers to make sense of and apply the findings of the D2AC assessment by developing an implementation plan using the template included in the D2AC tool.

DISCUSSION

While the D2AC offers to assess “as is” and “desired” levels of domains and subdomains and develop an improvement plan to advance capabilities required to transform data-to-action, it does not imply that an NTP can achieve goals only when it reaches the highest levels. The NTP goals should determine the desired continuum level for different domains and associated subdomains. For example, suppose the NTP’s goal is to extract a defined set of data elements from an electronic medical record and a lab information system to analyze the number of patients on treatment against the number of people who have positive test results. In that case, the organization can achieve its goal at level three where processes and systems are sufficiently defined. The D2AC Toolkit’s utility to guide decision makers critically depends on the quality, level of detail, and comprehensiveness of the Toolkit and respondent input coordination and buy-in from the country, as well as the ability to adapt the Toolkit to address contextual issues.¹⁹ While considerable effort is spent in the design of assessment toolkits, considerations of real-world implementation and contextual adaptations are often lacking.^15,20 These considerations include:

• Who will be the users of the Toolkit?

• Who will be responsible for the data collection, analysis, and the derivation of measures?

• Who will be involved or asked?

• How frequently will the assessments take place?

• What will be done with the results?

The D2AC country implementations will generate contextual data and help to provide guidance for addressing real-world implementation challenges and informing adaptation to country needs.

Limitations and challenges

The D2AC Toolkit has been designed to consider certain challenges commonly experienced by end users at the country level. Access and user-friendliness have been essential components to the Toolkit design—making the navigation throughout the different Toolkit’s components intuitive and for all the necessary resources (e.g., glossary, continuum level definitions, scale) to be accessible to users while they complete the data collection instrument.⁷ Furthermore, the initial design of the Tool on Microsoft Excel made the Toolkit available and able to complete offline, hence not requiring an uninterrupted working Internet connection to save responses, proceed with the questionnaire, and save or submit results. A digital version of the Toolkit is being developed in Fall 2022 and will be available on the TB DIAH Hub. The digital Toolkit will provide an alternative and even more seamless option for D2AC workshop facilitators, while the Excel-based Toolkit will remain available for any uses in a context with unstable Internet access.

Beyond access and user-friendliness, some challenges that may be faced when using the tool are related to country context customizations. These can be as minor as adjustments to the names used for different administrative units (regions vs provinces vs states) or for different roles and titles of functions or institutions involved in the fight against TB. Other instructions may need to be given to participants when considering how to answer some of the data collection instrument’s questions that may pertain more or less to integrated TB programs. Furthermore, workshops and implementations in non-predominantly English-speaking contexts will require translations of the Toolkit and workshop materials.

Although the Toolkit has attempted to build in as much provision for this as possible, one of the most important limitations is the appropriateness and timeliness of the actions taken following an assessment. The scope of the Toolkit only goes as far as to suggest specific and customized recommendations based on answers provided and scores achieved, and the D2AC workshop is designed to have participants co-create a recommended plan to implement priority actions they have identified. Beyond the workshop, initiative and funding to enact the outlined priorities will depend on buy-in from the government and other stakeholders.

Finally, COVID-19 (and possible new waves and lockdowns), as well as other unexpected circumstances (e.g., natural disasters, disease outbreaks, political instability, conflict), continue to raise a concern about unexpected turns of events, whether they may affect future D2AC implementations, the state of TB care and data in many countries, or access and reporting of quality TB data for proper use for evidence-based decision making. Other challenges made apparent during the 2022 field tests will be reflected upon, and any necessary changes made to the Toolkit and workshop protocol.

NEXT STEPS

The next steps include validating the tool through field tests in two countries (Ghana and Nigeria) under the leadership of their respective NTP. The experiences and lessons learned from the country implementation will inform the finalization of the D2AC Toolkit, including the Toolkit User Guide, and its publication in the public domain. Following the publication of the English language version of the Toolkit, further country implementations of D2AC (and translations of the Toolkit in accordance) will take place.

Another consideration for the next steps is to integrate questions specific to different participants who would be attending the workshop, who have different TB data needs, and for whom the Tool could include customized user role questions. This would allow for the analysis to also examine how well data needs are being met at different levels and for different roles in the TB system, and inform role-specific sets of recommendations or priority actions related to human resources and capacity building, including training, hiring, and supervision.

Other considerations for the D2AC is to create a generic or other disease-specific versions for the conceptual framework, design, and approach to be transferrable to uses in the context of other infectious diseases monitoring. The Toolkit, once published, will be a global good accessible to all public and private institutions wishing to assess their country’s performance on the D2AC scale. The primary objective of the D2AC implementation findings is to inform improvements, changes, and updates in country guidelines, protocols, standard operating procedures, and workflows to improve TB system performance and service delivery. The Toolkit may also be used to develop other, more specific tools, for data use training purposes, considered in the context of the development of M&E guidelines and standard operating procedures, or for routine assessments to track progress over time. Findings from the D2AC may be used for ongoing technical work or peer-reviewed research.

CONCLUSIONS

By integrating design science and a user-centered approach, the D2AC Toolkit can enable policymakers, program managers, and service providers to gauge current capabilities, identify future priorities, and develop an improvement plan to advance data use capabilities.

Acknowledgements

The authors wish to thank the following colleagues: Sevim Ahmedov, Zeleke Alebachew, and Joy Kamunyori of USAID, and Stephanie Mullen and Ezra Tessera of TB DIAH. Thanks to all members of the D2AC Advisory Group.

Disclaimers

The views expressed in the submitted article are of the authors and not necessarily those of author’s institution or USAID or the United States government.

Funding

This review was funded by the USAID-funded TB DIAH Associate Award No. 7200AA18LA00007.

Author Contributions

MK led the conceptualization and writing of this manuscript. JC contributed to the conceptualization of the manuscript and made a significant contribution to the writing of the manuscript. MS contributed to the conceptualization and review of the manuscript. DB and YGM reviewed the manuscript. All authors contributed to the work described in this manuscript.

Competing interests

The authors completed the Unified Competing Interest form at http://www.icmje.org/disclosure-of-interest/ (available upon request from the corresponding author), and declare no conflicts of interest.

Correspondence to:

Jeanne Chauffour, MS
John Snow, Inc. (JSI) - 2733 Crystal Drive, 4th floor, Arlington, VA 22202
[email protected]