Process evaluation (PE) studies are carried out to assist in interpreting the results of trials and to provide better understanding of their context, with the ultimate aim of informing policy and practice.2,3 Thus, PE studies examine, among other aspects, how an intervention is implemented, seek to understand participants’ perceptions of the intervention, and to explore contextual factors that influence the intervention and its outcome. In the case of an unsuccessful trial, PE is able to determine if the problem lay in the intervention design or its delivery (i.e. fidelity).
The Jom Mama intervention
The Jom Mama (translated as ‘Come on, mother’ in Bahasa Melayu) project was designed as a response to the rising public health challenge of overweight and obesity,4 focusing on young married Malaysian women and their spouses. The overall aim of the Jom Mama project was to determine whether a complex behavioural change intervention changed markers of health in the young Malaysian woman in the pre-conception period. Further details are given in an accompanying paper.1
The intervention, which was developed based on a number of preparatory studies,5–7 combined behavioural change communication (BCC) provided by community health promoters (CHPs), and access to a tailored mobile application (the Jom app) aimed at encouraging healthy lifestyle changes. A total of 48 CHPs were recruited from within the public health care system – 37 community health nurses, 10 staff nurses and one nursing sister. They were provided with four days training prior to the intervention, introducing them to the national guidelines for nutrition and physical activity, train them in BCC techniques, and the use of the Jom app. The course curriculum was developed by local communication experts based on a series of consultations with multiple stakeholders and assessment of trainees’ prior levels of knowledge and skills. The course participants were given a handbook explaining the course content, and the outcomes were evaluated using a pre-post design. The CHPs were provided with ‘peer-support’ during the intervention in order to sustain and further improve their communication skills. The intervention included a total of six contact points (CPs): three face-to-face (CPs 1, 2 and 5) with each woman, and her partner (if possible) at a site of their choice; and three via phone calls (CPs 3, 4 and 6) over a period of about 33 weeks.8
The primary outcome of the Jom Mama trial was change in waist circumference. In addition, a number of secondary outcomes were measured, including changes in BMI, HbA1c, fasting lipid profile, blood pressure, diet from a food frequency questionnaire (FFQ), physical activity and sedentary behaviour (International Physical Activity Questionnaire, IPAQ) and mental health assessed by the Depression Anxiety and Stress Scale 21 (DASS-21).8
This paper reports key findings of the Jom Mama PE study, which ran in parallel to the randomised controlled trial.1 The PE comprised six sub-studies out of which four are reported here: recruitment, attrition, BCC, and use of the Jom app (e-health).
METHODS
Design
The PE was guided by the realistic evaluation approach.9 A programme theory (Figure 1) was developed outlining the main components of the programme of intervention as perceived by the researchers.
Four of the domains (recruitment, attrition, BCC, e-health) that are of direct relevance to the Jom Mama trial1 are reported in this paper. A mixed methods approach was employed to assess the domains.
Data collection
Data collection (and analysis) was done in three cycles during the trial (Table 1). Each of the sub-studies applied a cluster of data collection methods, whereby triangulation was ensured, combining quantitative and qualitative performance indicators as relevant.
Recruitment and attrition
Throughout the trial, weekly quantitative data were collected on the number of people approached, those eligible and those agreeing to participate in the trial, as well as number of participants later withdrawing. Focus group discussions (FGDs) were conducted with a total of 16 recruiters in order to capture their perspectives on inhibiting and conducive factors for the recruitment process. Short phone-based, semi-structured interviews (SSIs) were conducted with 18 participants regarding the recruitment process and the information provided (cycles 1 and 2). A sample of 25 participants leaving the trial was also interviewed by phone in order to explore their reasons for leaving the study (cycles 2 and 3). Direct non-participatory observation of the recruitment process was carried out at various recruitment sites (cycles 1 and 2) based on an observation guide exploring psychosocial dynamics and environmental aspects of the recruitment process.
Behaviour change communication
During the trial, a total of 48 CHPs were part of the intervention. Out of these seven withdrew. Of the remaining 41, 36 participated in either the FGDs or audio recordings throughout the study. In the FGDs, 20 participated once, nine participated twice, six participated three times and one participated four times, and out of these 29 were community nurses, six were staff nurses and one was a nursing sister.
The FGDs were conducted with CHPs regarding the BCC sessions to explore inhibiting and conducive factors. In all SSIs a total of 40 participants explored their perceptions of the interaction with the CHPs, with particular focus on the two-way BCC methods used in contrast to the traditional instructive approach, and the pros and cons of face-to-face versus phone call CPs. A total of 16 FGDs and 45 SSIs were conducted.
Audio recordings were made (cycles 2 and 3) to capture the way in which the face-to-face CPs took place. A total of 39 audio recordings were made for face-to-face CPs, comprising 22 out of the 36 CHPs. Out of the recordings, 16 audios were from CP1, 16 from CP2, and seven from CP5. Out of the 22 CHPs with whom recordings were made, the average was more than one audio recording.
Jom app (e-health)
Quantitative data on the usage of the Jom app were captured by ‘Google analytics’ including frequency and time of checking in and the type of challenges chosen by the participants. In addition, 16 FGDs with CHPs and 30 SSIs with the participants were conducted to capture how they perceived the functionality of the app.
Data processing and analysis
The trial’s quantitative recruitment data were analysed descriptively using Microsoft Excel 2010 and Stata, version 13.1.10 All FGDs and IDIs were recorded and transcribed verbatim in the local language, Bahasa Melayu. The transcripts were coded manually using open codes in English. Coding and thematic analyses were done by cycles and emergent themes were incorporated into subsequent cycles. Coding was done independently by three researchers and differences were discussed until consensus was reached. Observation notes were triangulated with the FGD and SSI data. Each audio recording was assessed independently by two researchers based on a checklist developed by BCC trainers. The assessment consisted of BCC components in an interaction between a practitioner and a client, presence of male partner during the session, level of interaction and how the sessions were carried out. Subsequently, the two researchers compared notes and finalised the assessment. Quantitative e-health data extracted from Google analytics were analysed using Stata, version 13.1.10 The variables analysed were frequency of the app being launched, challenge selection, and visits to check-in, resources and flash challenge pages.
Research ethics
The project abided to the ethical codes of the Helsinki Declaration.11 This study was approved by the Malaysian Medical Research Ethics Committee, with reference (5) KKM/NIHSEC/P16-626. It is registered under the National Medical Research Registry (NMRR-16-387-29002). All participation was voluntary based on written informed consent prior to participation. Anonymity and confidentiality of participants in this study were assured.
RESULTS
Recruitment
Recruitment was conducted by research officers and clinic staff (henceforth collectively called recruiters). During recruitment, 5053 candidates were approached. Among them, 2075 candidates were eligible and only 548 participants attended baseline assessment (intervention 272 and control 276). A total of 53% (1592/2978) of the women were excluded due to residence outside the study district of Seremban, which indicates a highly mobile target population. The consort diagram of recruitment for the Jom Mama trial is shown in accompanying paper.1
The recruiters described the importance of having a designated room for recruitment, as distractions from other patients in the clinics and use of personal phones posed challenges. Having limited time and overlap with other tasks and lack of access to necessary forms inhibited the recruitment process. The recruiters also stressed the importance of receiving appropriate recruitment training as it meant a new way of communicating with clients. Tracking missing clients through ‘WhatsApp’, phone calls, and HIV screening forms were considered useful. The recruiters found that pamphlets that were developed later, facilitated the recruitment process better and suggested that even more simple support material should be provided. Females and younger clients were seen to show more interest; particularly time constraints were often mentioned as a significant obstacle for participating in the trial.
"They think that this trial is for those who want to get pregnant immediately"
(Recruiter, cycle 2)
Some of the participants confirmed that they thought that the purpose of Jom Mama was to facilitate conception, and in some cases the young couples lost interest when they realized that this was not the purpose of the project.
“Initially I thought it was to help in conception. However, when they showed me the video, then I understood.”
(Female participant, cycle 3)
Generally, the main points gathered by participants were ‘healthy lifestyle’, ‘attaining good health before conceiving’, and ‘avoiding diabetes among pregnant mothers’. The participants felt that the trial was attractive as a source of information, and they appreciated the flexible time-frame and the opportunity to get screening for diabetes. Many mentioned social media, friends, healthcare providers, employers and neighbours as important sources of information. The participants highlighted the clear and concise communication, sufficient information and use of designated recruitment counters, and they found the brochure and video helpful in understanding the purpose of the project. However, as noted by the recruiters, some participants found the recruitment sites crowded.
The observations conducted at the recruitment sites confirmed many of the findings above, particularly the conditions or environment where recruitment took place. As a consequence of changes due to these findings, observations during the second data collection cycle confirmed that these conditions had been improved. The registered shortfalls of the recruitment process during the first cycle, allowed the procedures to be improved, which was crucial for reaching the set sample size of the trial.
Attrition
Attrition was monitored concurrently to the trial (Figure 2). The commonest causes for leaving the trial were conceiving before completing all the CPs (60.6% in intervention arm and 72.4% in control arm) and moving outside the study area (6.3% in the intervention arm and 8.6% in the control arm).
Lack of time was mentioned as another reason for dropping out.
It is difficult as I work in shifts. Sometimes I work at night, sometimes during the day so if I have to go to the clinic, sometimes it is difficult.
(Female participant, cycle 2)
Among the participants who dropped out of the project, some were satisfied with the CHPs’ performance and the Jom app functions, whereas others had faced problems with the app being incompatible with their phone (and had to use their husband’s phone) or found the challenges, especially in relation to physical activity, difficult to accomplish. However, these issues were not explicitly mentioned as causes of dropping out. Most participants found the interval between CPs acceptable although some considered it too long, and they found the clinic to be a suitable location. Overall, there was a preference for face-to-face CPs though some found the 32 weeks duration too long and wanted it shortened to 24 weeks.
“Face-to-face approach is more effective if you take into consideration the interaction. I could ask my CHP any questions directly as she is in front of me. It is easier to talk face-to-face.”
(Female participant, cycle 3)
Behaviour change communication
The characteristics of the CHPs are displayed in Table 2.
The majority of the face-to-face contacts took place at the clinics at request of the couples. The new BCC method on which the CHPs were trained was different from the usual authoritarian communication approach. Generally, the CHPs had very basic understanding of BCC, understanding it as using open-ended questions and encouraging the clients to give more input in the interaction.
“I understand it as how we communicate with clients, our eye contact, language use, and the questions must not be closed-ended questions.”
(CHP, cycle 3)
Some of the CHPs appreciated that the use of open-ended questions and eye contact on which they had been trained yielded more input from the participants. However, some did not apply it fully as they found that the participants were unable to respond when they used the technique.
“I tried to apply the technique, but my clients were not able to comprehend what I was trying to do”
(CHP, cycle 2)
In some cases the CHPs’ uncertainty regarding the use of BCC technique and preference for traditional instructive approach hampered the application of BCC. Some CHPs also found that BCC technique was not suitable for participants from lower educational backgrounds.
"They [the trainers] want us to change our approach where they want information to come from the clients. It is true that if we were to give close-ended questions, we can only get minimal information"
(CHP, cycle 2)
From the participants’ perspective, the CPs provided a means to discuss their health problems. They said that the CHPs were committed as they were following up trial participants in addition to their usual core duties.
“Jom Mama is an addition to the nurse’s core work. She is committed to her patients, but she also commits herself to the trial participants like me.”
(Female participant, cycle 2)
However, some participants reported that their CHPs lacked skills in handling the tablet and were not convincing.
“The CHP was very slow as she was not familiar with the application.”
(Female participant, cycle 2)
Some participants perceived that the interaction with CHPs was similar to other communication on healthy lifestyle. The interaction was not found to be instructive. They acknowledged the use of some elements in BCC such as using scales to ascertain participants’ compliance with carrying out selected challenges.
“She did not give instructions. It was more like providing motivation. The way she spoke was OK.”
(Female participant, cycle 3)
Other participants regarded the interaction as somewhat formal. Although it was a two-way communication, the conversation was sometimes dominated by the CHPs in giving detailed explanation pertaining to healthy lifestyle. The CHPs would also give suggestions to participants to facilitate behaviour change, which was not in accordance with the training received.
“Sometimes she gives suggestions. She would ask: “Do you want to do this challenge? Why don’t you try? Try something new”.”
(Female participant, cycle 3)
The 39 audio recordings provided insights into the way the face-to-face CPs took place. Only 14 out of 39 recordings were at CPs with both the female participant and her partner being present, and in only eight out of these did spouses have moderate or active interaction. Baseline lifestyle habits were explored during CP1, and the CHPs conducted the session based on the questions in web interface of e-health. Generally, the CHPs reported being confident in giving information on nutrition, though some CHPs were not familiar with the mobile application. With regard to communication style, goal-setting and motivational discussions by CHPs showed room for improvement. When giving information, the CHPs appeared instructive and mainly used close-ended questions and most participants took a passive role, although some open-ended questions were also used. It was found that 85% of the recordings included the use of the rulers/scales for gauging the participants’ commitment in carrying out the challenges, and that 33% used open-ended questions.
Jom app (e-health) data
Data for 145 women who completed the trial intervention were included in the analysis. Access data between the participants and the Jom Mama app indicated that 137 (94.5%) of participants launched the app, and 131 (90.3%) visited in the check-in page. Only about half (81, 55.9%) of the participants accessed the app for flash challenges in relation to either nutrition or physical activity, and 51 (35.2%) accessed the resources page.
The CHPs appreciated the improvements made on the app during the project, and used the tablets for progress monitoring. Nevertheless, there were some practical challenges, as the app was in some cases incompatible with the participants’ phone interface, and its appearance on the phones and tablets of the CHPs and participants differed. It was suggested that more visual aids were introduced into the app, such as figures and exercise videos. In addition, some CHPs found the content to be repetitive, and they found that some participants chose the challenges that they found to be the easiest (but maybe not having the biggest health impact).
“They only want simple challenges because they do not have time to carry out the challenges”
(CHP, cycle 3)
In general, the participants found the Jom app to be user-friendly and interesting. More specifically, they appreciated the function of reminders to do the challenges chosen, and the way the resources section enabled them to gather information. Some also appreciated that the features foster competition with the spouse to complete the challenges. Some of the participants reported that they faced challenges of the app ‘hanging’ as well as login and installation issues. More specifically, they found that the challenges were easy to carry out, especially those involving home activities. However, some challenges were seen to be too demanding, and they confirmed the observation of the CHPs that they were inclined to take challenges that they were already doing frequently.
“Yes, I picked the easy challenges. I selected all the easy ones”
(Female participant, cycle 3)
DISCUSSION
This study aimed to understand the barriers and facilitators to implementing a novel, complex pre-conception intervention in young couples in Malaysia. The Jom Mama pre-conception intervention was not effective in preventing increase in waist circumference, and did not lead to practically significant changes in other secondary outcomes over the eight months prior to conception. However, there was a significantly smaller weight-gain in the intervention vs. the control group, predominantly in women with pre-existing obesity.1
We report the results from sub-studies on four domains of the extensive PE study that accompanied the Jom Mama trial. The realistic evaluation perspective9 emphasizes the distinction between implementation and theory challenges. Or put simply, if findings are negative is this due to inadequate implementation (lack of fidelity) of an otherwise well-designed intervention, or to poor design of the intervention? The discussion below is organised along these lines.
Implementation challenges
Results from the various data sources indicated that the CHPs only adopted the new BCC technique to some extent as possibly their training was either not sufficient or long enough given the novel concepts they were introduced to. In addition, three CHPs were recruited later to supplement CHP attrition, and they were given a less thorough training. Whereas some of the face-to-face CPs took place at the participants’ homes, the majority were conducted in the clinics at the request of the participants, which in the beginning often entailed lack of privacy and rather noisy environments.
The qualitative preparatory studies7 had indicated that the target group members themselves suggested the use of e-health interventions using smart phones. However, in reality, the project faced a number of practical problems in terms of weak or no connectivity and incompatibility of devices with the Jom app. In spite of a significant number of resources invested in developing the app, there were still a number of cases with login difficulties, ‘hanging’ and ‘bugs’ which led to the need for updates. In addition, there were indications that some of the CHPs had not achieved a sufficient competence level to support the participants’ e-health utilisation. However, based on our findings, these issues were to a large extent addressed along the way, as this was one of the purposes of the PE.
Early on, it became clear that recruitment was not providing a sufficient number of participants. Consequently, changes were introduced, which improved recruitment and ensured that the required sample size was achieved in the trial.
Theory challenges
Although the choice of the nurses as CHPs made sense because they were already part of the health care system, their former training made it challenging for them to adopt the new communication approach. In spite of four days training the evaluation showed that the CHPs’ competences were not quite sufficient on BCC or on e-health. Perhaps it was simply too challenging for them to unlearn more traditional ways of transmitting information to the participants. Thus, to use this group of health care workers for the key implementers of BCC may necessitate introducing the techniques as part of their basic training rather as a post-graduate addition. Moreover, some participants found it difficult to cope with the new CHP BCC communication style on which the CHPs had been trained. However, other Malaysian studies have documented positive behaviour changes based on motivational interviewing for persons with overweight, obesity or Type 2 Diabetes as well as physical activity in relation to smoking cessation, although neither studies included a process evaluation.12,13
Whereas the Jom app was tested among the target group prior to trial, this was not the case with the BCC component, and a process where the young couples had been more involved in the creation may have made it more compatible with the underlying cultural context, where individuals presenting to healthcare facilities often leave it to the healthcare providers to make decisions.
Furthermore, e-health may not be the best way to reach the participants in spite of findings in the pre-studies.7
Only about 15% of the total eligible participants completed the project,1 mainly by not enrolling for various reasons, but to a certain extent also through attrition. The time consumption of the intervention itself, including the six CPs, is estimated to be a total of three hours, and the baseline and endpoint measurement visits of the trial to require another four hours over an eight-month period. There is no evidence that this burden of participation was the main reason for the low uptake. However, it is probably safe to say that a more time intense intervention may decrease it further, whereas a decreased total duration will weaken the already sparse impact. In short, the findings showed challenges with implementation as well as theory.
Public health implications
A number of lessons emerged from the PE, which relate to the interpretation of the data from the Jom Mama trial and its implications for future research on public health interventions aimed at promoting health in the pre-conception period of the life-course. In spite of a significant effort to tailor the intervention to the target group and to conduct recruitment in various ways, it was only possible to recruit 26% of the eligible women, and only 15% completed the intervention. From a public health perspective, this represents a major challenge if a scaled-up, national pre-pregnancy programme were to be considered. Hence, although pre-conception interventions make sense from a theoretical point of view, implementing them in young married couples in this setting is very difficult. It seems that this group of young adults, in the process of starting families, are either not motivated sufficiently or simply not accessible, either due to busy work schedules or to other pressing priorities.
The intervention was based on a combination of BCC and the Jom app. In spite of four days of initial CHP training and continuous peer support, the PE showed that the CHPs still found it challenging to master the new and more interactive BCC. This may account for the lack of change in the primary outcome in the trial. Thus, an important lesson learnt is that it requires a more comprehensive and probably more long-term effort to change the approaches and culture of community health staff. The more open BCC style we aimed to induce may be somewhat challenging and would require a more substantial training.
Despite the problems of recruitment, we were able to recruit sufficient participants to the trial, after much effort. A further challenge resulted from 28% of the intervention and 30% of the control couples withdrawing from the trial. Despite some reservations noted above about the expectations of the trial and the contact with the CHPs discussed above, the couples withdrawing did so predominantly because they conceived. This makes it clear that this group of participants were indeed recruited in the pre-conception period of the life-course, but it raises the question of whether the intervention period of 33 weeks was too long, but possibly not long enough to affect change. A shorter trial period might require more intense engagement with the couples in the intervention arm. Given the lack of effect of the intervention on the primary outcome in those couples who did not withdraw from the Jom Mama trial, a shorter trial seems unlikely to be effective. Thus, alternative segments of the life-course should be considered, e.g. shifting the intervention time either to after the first pregnancy or earlier in the couple’s lives prior to marriage.
Furthermore, it should be borne in mind that the Jom Mama trial was based on the assumption that a tailored intervention addressed to individuals might produce significant healthy behaviour change to improve health markers such as waist circumference. This basic premise may be flawed, meaning that more structural interventions are needed, targeted at the deeper underlying social determinants of obesity and unhealthy lifestyle such as low income, socioeconomic position, education and the food and built environment.14
The lessons learned from the Jom Mama trial make a powerful argument for addressing these issues if the longer-term and transgenerational harm from unhealthy lifestyle leading to obesity in young adults is to be mitigated.
Strengths and limitation of the process evaluation study
A strength of the present study is that process evaluation sub-studies were thoroughly planned, based on the formulation of a programme theory. The sub-studies applied triangulation of different data collection methods on the same topic, and the resources allocated were substantial based on a longitudinal approach with three project cycles.
On the other hand, there were certain practical limits to data collection, e.g. the use of audio recordings to assess the face-to-face CPs, where direct observational methods would have been more ideal for assessing CHPs’ BCC skills including non-verbal communication. Furthermore, there was limited access to participants who declined to enrol or left the study, so the PE did not obtain a full picture of potential negative aspects as causes of attrition.
CONCLUSIONS
The present article reports an extensive PE study conducted in parallel to the Jom Mama trial in Malaysia, which explored the effectiveness of a pre-conception intervention with a combination of six CPs with specially trained CHPs and a tailored e-health platform. The four sub-studies in PE reported here comprise recruitment, attrition, BCC and use of the e-health platform.
The insights achieved from this research served two main purposes: 1) to facilitate the implementation of the trial (e.g. by improving the recruitment thereby allowing the sample size needed to be reached), and 2) to provide insights into implementation of the intervention (fidelity) allowing a deeper interpretation of the trial outcomes. They reveal that, despite considerable effort on recruitment, there was attrition over the trial due to conception before the trial was completed, and the training and engagement with the CHPs and the use of the dedicated app were sub-optimal. This has implications for future interventions aimed at inducing healthier behaviours in this section of the population.
ACKNOWLEDGEMENTS
Many thanks to the young couples, who engaged themselves in Jom Mama, and to the many clinic staff and research members that conducted the intervention and the evaluation. We would like to thank the Director-General of Health, Ministry of Health Malaysia for permission to publish this study. Prof. Mark Hanson is sponsored by the British Heart Foundation. This study was registered under the National Medical Research Register (NMRR-16-387-29002) and received approval from the Medical Research Ethics Committee ((5) KKM/NIHSEC/P16-626).
FUNDING
We are grateful for generous funding from the Malaysian Ministry of Health and Novo Nordisk. The other participating institutions have contributed substantially in kind.
AUTHORSHIP CONTRIBUTIONS
MAH, PM, JAH, SAN, RBB, JKHS and JCH were involved in all phases of the development of the Jom Mama trial. MZZ, JAH, ANMH, JKHS, DC, SMS and SFAZ were involved in the development of the process evaluation protocol whereas the fieldwork itself and data analysis were conducted by DC, SMS, ANMH, SFAZ and MZZ. The manuscript writing was driven by, JAH, SMS, DC, ANMH, MAH and JKHS. The final manuscript was approved by all the authors.
COMPETING INTERESTS
The study forms part of the Jom Mama project in Malaysia. The Jom Mama project is a public-private partnership with the Ministry of Health Malaysia, Novo Nordisk (Denmark), the University of Southampton (UK), the University of the Witwatersrand (South Africa), and the Steno Diabetes Center Copenhagen (Denmark) to address diabetes prevention in Malaysia. The development of the intervention package, project management and the costs of the Clinical Research Organization responsible for external quality control have been funded by an unrestricted grant from Novo Nordisk. The Ministry of Health Malaysia has provided the funding for human resources and trial implementation. JKHS and JCHC are employed by Novo Nordisk. MAH is supported by the British Heart Foundation.
Corresponding author:
Jens Aagaard-Hansen, Health Promotion Research, Steno Diabetes Center Copenhagen, Borgmester Ib Juuls Vej 83, 2730 Herlev, Denmark.
email: [email protected], phone: +45 3030 1258