Accurate documentation of health data is a critical component to quality of health care. However, the value of health records in providing patient information is highly dependent on their completeness and accuracy.1 Data capture tools used in health care should be easily accessible and information gathered should produce accurate and reliable records. Important clinical decisions both at individual patient level, as well as for programs and policy are pegged on synthesis based on these records.2–4 Sub-standard patient data documentation is prevalent in several less developed countries leading to inaccuracies, inconsistencies, and increased errors and missed opportunities in health care provision.5–7 These problems are occasioned by the fact that in these settings most records are managed manually by health care workers who are often overburdened due to staff shortages. Majority of these staff have no proper training in documentation and data collection. Most data errors include; lack of granularity, accuracy, completeness and missing information. These challenges are often compounded by difficulty in patients records retrieval due to deplorable chart storage spaces.2,3 Ensuring that patients’ data records are accurate and consistent across different documents and are easily retrievable is central to quality of health care. Unfortunately, these desirable attributes still remain as major gaps in the developing nations leading to poor patient care.8–11 Data quality issues impact key health indicators such as accurate diagnosis of prematurity. The problem is further compounded by the fact that gestational age estimates may also be largely inaccurate in rural communities and early pregnancy ultrasound is not available.12,13
Recognizing these challenges, the East Africa Preterm Birth Initiative (PTBi-EA) chose to use facility data sources and implemented data strengthening activities; as part of a package of interventions in a study designed to improve quality of intra-partum and immediate postnatal care in Migori County Kenya and the Busoga Region of Uganda.14 These data strengthening interventions included a set of proactive and engaging activities conducted by the study team together with the facility staff. There was an initial data strengthening training in all participating activities followed by a monthly ongoing facility-based mentorship on consistency and accuracy of documentation in the maternity registers. The teams were encouraged to use data for decision making and feedback. To monitor and further address gaps in data quality, we enhanced our data strengthening intervention with structured data quality assessments (DQAs) across all participating facilities. The findings would be fed in the whole process of improving the data strengthening intervention.
DQAs are standardized review of data quality that include verification and recounts of reported data and assessments of systems generating the data. Our assessment approach was modeled after DQAs that have been previously conducted in similar settings that focus on the rapid assessment of HIV patient monitoring systems, recreation of select indicators and validation of reports, and assessments on the quality and completeness of reports.15 This is in alignment with the Government of Kenya recommendation on the institutionalization of data quality improvement through the Kenya health sector quality assurance protocol,3 which has not been widely implemented to date.
The purpose of the DQAs was to identify areas for improvement in consistency of data across various data sources within maternity and also determine impact of PTBi-EA data strengthening activities over time. Through the DQAs, we also intended to assess the effect of infrastructure improvements at the health record departments on enhanced health information storage. We also aimed to determine accuracy of reports generated by facility-level integrated service delivery forms (MOH 711) against data elements items recorded in the maternity register. This paper documents the process and findings of the PTBi-DQA activities in the Kenyan study sites.
METHODS
DQAs were conducted in 17 selected health facilities participating in the PTBi-EA study, in Migori County, Kenya. The facilities were composed of one county referral hospital, 10 sub-county hospitals, four health centers, and two private mission hospitals. Description of these facilities and the parent study is reported elsewhere.14 Briefly, these facilities were larger volume facilities which included the county referral hospital as well as all the subcounty hospitals and a few higher volume health centers. The staff who handle records in these facilities include maternity ward staff, primarily nursing officers and midwives, and Health Records and Information Officers (HRIOs) During the study period, 12 health facilities had their records storage rooms renovated to improve safety and accessibility of records.
We conducted three DQAs over a period of three years. These involved retrospective review of maternity registers and patient charts to compare key indicators recorded in the two different data sources for completion and consistency. These was done by the PTBi-EA data team who identified and recorded documentation gaps and shared them with key maternity ward staff. Data strengthening reports were also provided to facilities on a quarterly basis highlighting gaps and progress over time. Data feedback sessions occurred during facility Continuous Medical Education (CME) with most of the staff being present. We also renovated 12 health facility records storage rooms. This was aimed at ensuring that patient records were stored safely and easily accessible. The indicators of interest included; Admission Date, Gestational Age (GA), Birth Weight (BW), Last Menstrual Period (LMP), APGAR score, delivery outcome, and final neonatal status at discharge.
We determined sample size for DQAs by obtaining average deliveries per month for each facility. We then calculated the probability of success (p) which was probability of getting documented gestational age in both patient file and maternity register (p=0.16). Gestational age was a key indicator in determining PTBi-EA study participants and therefore its documentation was used as probability of success. Using the Keish Leisle formula no= (Z2pq)/e2 (where; p = probability of finding a recorded gestational age in both register and maternity patient chart, q = 1-p, e is margin of error and Z2 = 1.96.). We got the sample size for each facility using three different confidence levels of 90%, 92% and 95% which gave us a minimum, moderate and maximum sample size respectively.
The minimum size was given priority A. The difference between minimum and moderate sample size gave us the number to sample if our priority A sample was not saturated. This was given priority B. If the priority B could not saturate the minimum sample size required, then the difference between the moderate and maximum gave us the number to fulfill the minimum sample size. This was given priority C. Using this procedure we retrieved minimum sample sizes for the selected DQA month for each facility (Table 1).
Simple random sampling technique was employed to access the patient charts from maternity admissions for the selected DQA month. We assigned random numbers to all the maternity admissions for that month. We sorted the admission in ascending order by the random number and assigned the first batch that represented the minimal sample size Priority A, the next batch that represented difference between minimum and moderate sample size was assigned priority B and the last batch that represented difference between maximum and moderate sample size was assigned priority C.
Retrospective data collection was conducted by PTBi-EA data team. All data collectors were trained on data collection tools. Data from maternity registers, patient charts and facility-level integrated service delivery forms (MOH 711) were entered into excel-based DQA data collection tools which included a cross validation tool and report comparison tool (Online Supplementary Document, Appendix 1 and 2).We used the cross validation tool to compare values in the maternity register and patient charts. The report comparison tool was used to compare data on integrated service delivery forms (MOH 711) versus data obtained from the maternity register.
We conducted the three DQAs in Kenya when all facilities were at optimal operation. Data were collected for the following time periods: DQA 1 (Sep 2016 to Feb 2017), DQA 2 (November 2017 to February 2018) and DQA 3 (Dec 2018 to Mar 2019). Data collection teams worked congruently with facility-based HRIOs to retrieve all sampled patient charts using the mother’s patient number. The charts were retrieved according to their priority-level assigned during the sampling process. If all three batches could not fulfill the saturation of minimum sample size then missing files were declared irretrievable.
Through the patient chart retrieval process, we assessed the proportion of maternity register entries that could successfully be traced to a patient chart, we reviewed all the maternity patient files/charts to compare if the number of charts reviewed was equal to the number of admissions documented in the maternity register for the specified DQA month. Results were recorded in a summary log.
To assess the completeness and consistency of data extracted from the maternity register, the PTBi-EA data team recorded key maternity indicators: baby’s birth weight, date of delivery and discharge status and gestational age from the patient file/chart and compared them to the corresponding line in the maternity register to determine if these two data sources have the same information documented. Indicator data from these data sources were recorded into a cross validation tool.
The PTBi-EA data team also recorded monthly counts of data from the MOH 711 forms for specific indicators namely: Fresh Still Birth (FSB), Macerated Still Birth (MSB), live births, preterm births, babies born with low birth weight and newborn deaths. Using the report comparison tool, these data were then compared to aggregate monthly counts of the same indicators in the maternity register, using official MOH indicator definitions.
Data analysis
We exported the data collected on the excel to STATA 13.0 for analysis. We analyzed the data using descriptive and inferential statistics. We conducted proportional test to check the differences in proportion at different time points by calculating z-test. All statistical tests were carried out at (0.05) level of significance. This analysis helped to assess whether there was any improvement in the file traceability and data consistency between various DQA time points and reduced variance between counts in maternity register and facility-level integrated service delivery forms (MOH 711).
Ethical considerations
The study was reviewed by KEMRI (Study ID# 0034/321) and UCSF (Study ID# 16-19162) institutional review boards and approved. All data that we extracted from routine data sources had no personal identifiers were stored on encrypted computers. The necessary permissions were sought and granted by the county health department.
As part of ethical consideration and for sustainability, we mentored the midwives who are the data generators in maternity departments to instil a culture of data quality data generation and its use during their day to day patient care. This was aimed to ensure that data quality become part of facility quality improvement. We also involved HRIOs during the data quality assessments and feedback to enable them to continue undertaking the assessment and giving feedback to maternity ward staff. Finally, we organized a workshop at the end of the project with key staff such as facility and maternity in-charges and HRIOs and trained them on the PTBi model of improving the quality of data documentation.
RESULTS
Table 2 shows the chart traceability at different time points of the data strengthening activities. The average chart traceability improved from 69% to 88% and to 90% from DQA1 through DQA 2 to DQA 3 respectively. The improvement on chart traceability was shown to be statistically significantly different when we compared traceability between DQA1 and DQA2 and DQA 1 and DQA 3 respectively (P<0.001).
Table 3 shows the output of analysis of the percentage matching for key indicators from two data sources across DQA 1, DQA2 through DQA 3. Marching of the key indicators was shown to improve across the three DQAs. The improvement in the matching of key indicators between patient chart and maternity register showed statistically significant improvement when we compared DQA1 and DQA2 as well as when we compared DQA 1 and 3 across all indicators (P <0.001) except for the birth weight (P >0.05).
Table 4 compares the variation of data recorded in MOH 711 from maternity register. This was achieved by comparing data recorded on MOH 711 to maternity register through highlighting the percent of maternity register that is reported on MOH 711 for selected indicators. Across the three DQAs, the under-reporting of the maternity register data into the MOH 711 was seen to steadily reduce for all indicators other than a small increase in live births between DQA 2 and 3 from 2% to 4%. For example, for FSBs counted, 58% of maternity register data is reported on MOH 711, indicating that there was a 42% under reporting of FSBs in MOH 711 in DQA 1 which reduced to 23% and 8% after DQA 2 and 3 respectively.
DISCUSSION
The results indicate significant improvement in the quality of maternity data, particularly in data match between maternity register and patient files. They also show an improvement in the concordance between key indicators data reported to MOH from the maternity register through the reporting tool, MOH 711 as well as an improvement in chart traceability overtime. This improvement had been observed in the initial PTBi data analysis in Uganda where we observed an increase in the completion on key fields. However the opposite was observed in Kenya which was attributable to a prolonged nurse’s strike.16 Additionally differential improvement in data quality, particularly related to outcome ascertainment such as stillbirth vs abortion vs immediate neonatal death as well as gestational age (GA) assessment and estimation between control and intervention facilities due to inherent effects of the package of PTBi-EA interventions had been observed.16
Improved chart traceability suggests that facilities reorganized the storage of patient records and that files could now be more easily retrieved. This ease of tracing of patients records has been shown to enhance the quality of patient care.9,10 Renovations carried out in some records storage rooms might have also contributed to significant changes in the chart traceability. This type of improvement in accessibility of patient’s records after renovations of the record storage rooms has been reported by a study by Tevie et. al.17 Overall improved concordance of data from the maternity register into the MOH reporting tool (MOH 711) suggests improvements in quality of reported information from the facilities to the central registry at the MOH meaning that the data reported was more complete, consistent and accurate over-time. Similar results on concordance of results have been observed by Wagennar et. al. in their study on the effect of data quality intervention in Mozambique.18 Regular data strengthening activities which focused on improving documentation within maternity was a core part of the PTBi-EA interventions which we believe must have contributed directly to these improvements; which agrees with other studies on regular data quality improvement in other areas such as HIV care.19
While it might be possible to attribute the improvement to PTBi- EA data strengthening and facility upgrade, limitations of this report could be attributable to: i) the presence of other partners working with the same data sources with possibility of inputting corrective measures, ii) factors, such as few staff in facilities, varied delivery volumes, and high staff turnover rates in the study sites, that could have limited effective data strengthening training and activities, iii) without a direct control group, we cannot eliminate the possibility that the changes were seasonal. In addition, improvements in data matching across data sources do not mean that data validity has improved and since not all indicators captured in maternity wards were reviewed, consequently our results may not be representative of all key maternity indicators needed for program planning or service provision. Howeve, this study adds to the body of knowledge process and outcome of conducting Data Quality Assessments of Maternity units in the less developed countries and may form the basis of further studies.
Conducting data quality improvement activities in resource poor settings can come with several challenges.20 Specifically, for this study we encountered and addressed several challenges. For example, due to the fear associated with audits, there was initial resistance to the assessments from maternity staff (e.g. midwives, nurse in-charge) and HRIOs but the resistance reduced with continued mentorship and feedback when the staff recognized the importance of data quality. Secondly, patient charts storage space was a major constraint due to the large amount of records generated when using manual records systems. We surmounted this challenge by renovating storage space in 12 facilities. In addition, initially there were frequent patient’s charts stock outs with risks of data loss. To address this challenge PTBi-EA sourced and supplied patient charts to the facilities and encouraged the county’s Department of Health to ensure a steady supply of the charts through direct purchasing and support from partners.
During the implementation of this study, several lessons were learnt which could benefit other resource constrained settings. For instance, due to the use of manual data collection tools, lack of proper data storage spaces and lack of training in data quality and use , data collected may be questionable and need additional data review and triangulation to ensure quality.21 In addition, lack of regular data quality audits and timely feedback, leads health care workers not appreciating the role of quality data in improving patient outcomes. Creating a culture of data quality assessments and feedback in maternity wards is key for generating reliable, complete and accurate health data.
CONCLUSIONS
Routine DQAs combined with targeted data strengthening activites which includes training health care workers can lead to overall quality improvement of maternity data that can encourage data use for program management and evaluation. We recommend that the County’s Department of Health put in place data quality assessments and feedback mechanism though the HRIOs to ensure that generated data is regularly audited and feedback is given to the maternity staff. They should also ensure training of maternity staff on data quality and its usefulness for patient care. The County’s Department of Health should also invest in a well-functioning health information system that ensures seamless generation, analysis and feedback of health data for patient care and that there should be leadership and governance to ensure that systems to support regular DQAs are in place as well as ensure continuous supply of data capture tools such as patient charts and improve the storage spaces.
For future research, we recommend a multi-county and multi-site data quality assessment and feedback study to assess the quality of data generated at different levels of health facilities using indicators from different departments and in addition to comparing manual and electronic health records.
Acknowledgments
We thank the Bill & Melinda Gates Foundation for their generous funding of the PTBI project in which this study was nested. We would also like to thank the health care providers of each study facility for participating in this study. We thank the Ministry of Health of Kenya and the Migori County Health authorities for their support and partnership in this work. We sincerely thank the PTBi-EA research teams in Kenya and UCSF for their diligence and sacrificial efforts in the work. Special appreciation goes to Ryan Keating who pioneered and laid a strong foundation for DQA for the PTBi-EA work in Kenya and Uganda and Elizabeth Butrick for her support during the work as well as reviewing and proof reading of the manuscript.
Funding
The study is supported by the East Africa Preterm Birth Initiative (PTBi-EA), a multi-year,
multi-country effort funded by the Bill & Melinda Gates Foundation.
Authorship contributions
All authors contributed to the writing of the manuscript.
Competing interests
The authors completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available upon request from the corresponding author), and declare no conflicts of interest.
Correspondence to:
Prof. Anthony Wanyoro
Kenyatta University
Department of Obstetrics & Gynecology
PO Box 43851-00100
Nairobi, Kenya.
[email protected]