Fidelity of Implementation of Train-the-Trainer Methodology for Delivery of a Preschool Nutrition and Physical Activity Curriculum

Train-the-Trainer (TTT) methodology uses an expert to train a non-expert on how to implement an intervention and is often used by Cooperative Extension personnel for delivery of programs in the natural setting. In these interventions using multiple educators and/or non-experts, a measure of implementation (fidelity of implementation) is necessary to determine if delivery of the instructional program is as it was designed to be. The objective of this paper is to report the fidelity of implementation (FOI) of iGrow Readers (a literacy-based curriculum that uses children’s books focusing on themes of healthy nutrition and physical activity behaviors) that was delivered by Extension personnel to preschool children in the natural setting of 14 childcare centers through TTT approach. Structural and instructional FOI were assessed by a trained Extension associate. Assessing the FOI of the curriculum delivered by TTT approach provides evidence that the TTT approach is a feasible method of delivery for programing provided by Extension personnel in the natural settings.


Background
The Cooperative Extension System, in partnership with the National Institute of Food and Agriculture, United States Department of Agriculture, was developed to assist with translating research into practice in the agriculture-related, health and human sciences. The Cooperative Extension is operated through the Land-Grant University system in the United States and has a nation-wide non-academic credit network that addresses public needs (Cooperative Extension System, 2020). In the area of health and human sciences, Extension has focused on public health priorities including lifestyle factors for obesity prevention. Obesity is a public health problem in the United States. The prevalence of obesity (body mass index equal or greater than 30) increased from 30.5% in 1999-2000 to 42.4% (Hales et al., 2020) in 2017-2018 in adults and in youth 2-19 years of age 13.9% in 199913.9% in -200013.9% in to 18.5% to 201513.9% in -201613.9% in (Hales et al., 2017. Lifestyle behavior interventions and programming are important in the prevention of obesity. Educational prevention programs including children aged 2-10 years in various settings such as the home, school and community have shown potential to reduce and prevent childhood obesity (Nigg et al., 2016). Programing delivered by Extension personnel can contribute to prevention of obesity particularly in childhood.
The Extension programing is often delivered by multiple educators in the natural setting (National Institute of Food and Agriculture). To accomplish the dissemination of education down to the user level, the Extension model often uses the Train-the-Trainer (TTT) model which uses an expert to train a non-expert on how to implement an intervention (Greif et al., 2015). In these interventions using multiple educators and/or non-experts to deliver training, particularly when collecting outcome data to determine the efficacy of the program, a measure of implementation (fidelity of implementation) is helpful in determining if instructional program or training was delivered as designed (O'Donnell, 2008). Additionally, fidelity of intervention is important to assess program effectiveness. For example, if the results of a study indicate that the intervention was not effective, the researchers may question if instructions and intervention materials were "used in the manner intended or that the instructor did not carry out the intended procedures" (O'Donnell, 2008). Fidelity measurements assist with determining the degree to which the program was implemented and used to inform the determination of efficacy of the program (O'Donnell, 2008). Together, TTT and fidelity measurement enhance the implementation and dissemination of nutrition education interventions and programing especially programming delivered by Extension personnel using multiple instructors in the natural setting. Additionally, little empirical research exists assessing issues with FOI within early childhood obesity prevention programming (Hamre et al., 2010). Even though, FOI in program evaluation dates back 30-35 years there is limited use with evaluation of preschool curriculums for nutrition and physical activity promotion (O'Donnell, 2008).
Fidelity measurements generally include five criteria: adherence, duration, quality of delivery, participant responsiveness, and program differentiation (O'Donnell, 2008). The Fidelity of Implementation (FOI) framework developed by Century et al., (2010) encompasses these five criteria and provides a common language for fidelity measurements. Century et al., (2010) categorize the criteria into two main components of structural and instructional fidelity. Structural fidelity components measure intentions of design and organization of the intervention. Instructional fidelity components measure educational and student engagement components which consist of actions, behaviors and interactions that the instructor is expected to perform in implementation.
Few studies exist with preschool children that involve delivery with a TTT approach and a thorough evaluation of fidelity of intervention implementation (Pfeiffer et al., 2013). There is a need to identify classroom-based interventions for preschoolers that have been successful and can be mass-disseminated by the TTT means (Vinci et al., 2016). iGrow Readers, a literacy-based curriculum that uses children's books focusing on themes of healthy nutrition and physical activity behaviors was implemented for obesity prevention programing by Extension personnel in the natural setting using TTT approach. This paper is reporting on the FOI of iGrow Readers that was delivered to preschool children by Extension personnel through TTT approach. By assessing the FOI of the curriculum delivered by TTT approach evidence can be provided to determine if the TTT approach by Extension personnel is a feasible method of delivery for obesity prevention education (nutrition and physical activity) to preschool children.

Methodology iGrow Readers curriculum
A full description of the iGrow Readers curriculum is described at the following site: http://igrow.org/healthyfamilies/health-and-wellness/igrow-readers/. Briefly, iGrow Readers curriculum uses developmentally appropriate children's books with the goal of reinforcing healthful behaviors in children (Fountas & Pinnell, 2017;Loes et al., 2015). Each lesson plan includes nutrition and physical activity objectives, the intent of the lesson, a preview, active reading and summarizing sections, along with two nutrition activities and two physical activity activities that can be implemented by preschool teachers.

Study design
For the iGrow Readers intervention study, 14 childcare centers in South Dakota, Minnesota, and Nebraska participated in the teacher training and implementation of curriculum delivery. Within centers, two classrooms were selected and randomly assigned into either control or intervention classroom. Centers were provided with $300 to purchase the consumables required by the lessons. Only teachers assigned to the intervention arm were evaluated for FOI and reported in this paper. Teachers assigned to intervention classrooms were asked to implement 15 selected books from the iGrow Readers curriculum over an eight-week period which equated to 2 books/lesson per week. Although each book/lesson has multiple nutrition and physical activities included, the teachers were instructed to implement one nutrition activity and one physical activity to go with each book. Intervention teachers were provided a curriculum that provided step-by-step instructions on implementation of the lessons. Lessons were implemented during structured large group classroom time with follow-up activities immediately thereafter. For the fidelity assessment, it was not possible to evaluate all lessons in all books, so 11 of the 15 books were selected for evaluation and 14 teachers were randomly selected to be evaluated so that there was even distribution across daycare centers. The particular lesson within a book that was evaluated was not specified so there is a range (1-6) of replicate evaluations of the same lesson by different teachers. The 14 teachers were evaluated multiple times on different lessons (range 2 to 10); the maximum possible being one nutrition and one physical activity lesson each from five different books. There were 55 nutrition and 37 physical activity lessons evaluated. Approval for this study was obtained from the Institutional Review Board at South Dakota State University.
Prior to implementation of the iGrow Readers curriculum, all preschool teachers participating in the intervention completed a training on delivery of the curriculum. An Extension professional (trainer), who was involved in the development of the iGrow Readers curriculum, traveled to each childcare center, trained the intervention preschool teachers, and provided the curriculum and respective material kits to implement the curriculum. The training began with a description of the organization of the curriculum binder and how to use the curriculum binder. Next, the trainer provided a detailed description of the nutrition and physical activity lessons. The trainer also provided information on lesson resources and reviewed the parent newsletters. Last, a demonstration was given on how to utilize the materials and supplies for implementation of the nutrition and physical activity lessons. Instructions were provided on how to tailor the curriculum to each classroom. The trainer allowed each teacher to review the curriculum binder and the materials kits. Teachers were given the opportunity to familiarize themselves with the curriculum and to ask questions of the trainer at the conclusion of the training. They were also provided with contact information from the trainer to help them trouble-shoot if they had questions.

Fidelity of implementation tool
The fidelity instrument was modeled after the framework for evaluation of fidelity posited by Century et al., (2010). The questions with respective responses were developed by the authors specifically to query the structural (intention of design and organization of the intervention) and instructional components (actions, behaviors, and interaction that the instructor is expected to perform to obtain the student engagement) of the iGrow Readers intervention. Structural fidelity was measured with four questions that queried for (Q1) meeting the objective; (Q2) introduction of the lesson; (Q3) adequacy of the materials for the lessons; and (Q4) how much the teacher referred to the leader guide/materials throughout the lessons (Table 1). Responses for the Q1, Q2, and Q3 were yes or no with yes=1, no=0. Three response selections for Q4 were often, rarely, and unobserved, and were dichotomized for analysis to often=1 versus rarely and unobserved=0. A summary structural fidelity score was calculated as the sum of the four questions (range 0-4) with higher scores indicating greater structural fidelity. Instructional fidelity was measured with six questions that queried for (Q5) effectiveness of teacher at facilitating group participation, (Q6) involvement of the children in the activity, (Q7) involvement of the teachers in the activity, (Q8) how interested were the teachers in the class, (Q9) how interested were the children in the class, and (Q10) was the teacher prepared for the lesson (Table 1). Response selections for Q5, Q6, and Q7 were ineffective, neutral, or very effective (coded 1,2, and 3, respectively). Response selections for Q8 and Q9 were little engagement in the lesson, somewhat engaged in the lesson, or actively engaged in the lesson (coded 1,2, and 3, respectively). Q10 responses selections were yes (=1) or no (=0). A summary instructional fidelity score was calculated as the sum of coded responses for the six questions (range 5-16) with higher scores indicating greater instructional fidelity. Complete data was required to be included in calculation of the summary score. For analysis of individual questions, responses to Q5, Q6, and Q7 were dichotomized to very effective =1 and neutral or ineffective =0, and responses to Q8 and Q9 were dichotomized to actively engaged = 1 and some engagement or little engagement = 0.

Description of fidelity assessment
Center directors could either elect to have fidelity assessed in-person or they could elect to record the lessons using an iPad. All fidelity assessments (in-person and recorded) were completed by the Extension associate who provided the training. To evaluate validity of fidelity assessment completed by the Extension associate, 25 percent of the lessons were evaluated by a second content expert using the video recorded sessions. The kappa statistic (Rosner, 2016) was calculated to assess inter-rater reliability for individual fidelity questions. For all but 5 questions the similarity in scoring between evaluators was so close there was insufficient variation to calculate kappa. For 4 of the 5 remaining fidelity questions, kappa was greater than 0.4; kappa was low (0.2) for structural Q3 fidelity question asking if the materials were adequate for the teachers. The agreement of the summary structural and summary instructional fidelity scores was assessed using a paired t-test. For the summary structural fidelity score, the mean difference between evaluators was 0.11 ± 0.17 (mean ± se, p = 0.5). For the summary instructional fidelity score, the mean difference between evaluators was 0.11 ± 0.33 (mean ± se, p = 0.7). Thus, for both summary scores there was no significant difference between the two evaluators FOI assessment of the lessons.

Data analysis
Descriptive statistics included percentages for response categories of individual structural and instructional fidelity questions; and mean and standard deviation for the summary structural and instructional fidelity scores. Regression models were used to evaluate the associations of individual structural and instructional fidelity questions, and the corresponding summary measures (dependent variables), with lesson type (nutrition or physical activity) and number of children in the classroom (predictor variables) (Rabe-Hesketh & Skrondal, 2012). The delivery of lessons was nested within teachers and teachers were randomly assigned to specific books for evaluation. The regression models incorporated this nesting of evaluations within teachers by treating teachers as a random effect. For the individual questions, generalized linear mixed models were used with the logit of the response as the dependent variable and lesson type (nutrition versus physical activity) and number of children in a classroom as predictor variables. Individual questions with more than two responses were dichotomized (as described in previous section that describes the tool) for analysis. For some questions, it was not possible to evaluate the associations of teacher or lesson with the individual question responses because the outcome was perfectly predicted. For the summary fidelity measures, linear mixed models were used with the summary measure as the dependent variable and lesson type (nutrition versus physical activity) and number of children in a classroom as predictor variables. The intraclass correlation coefficient (ICC) was used to estimate the degree to which measures within teachers were correlated. For the logistic mixed models, the ICC was calculated on the logit scale as the variance for the teacher random effect (random intercept) divided by the sum of variance for the teacher random effect and variance for a standard log-normal distribution (Rodriguez & Elo, 2003). For linear mixed models, the ICC was calculated as the variance for the teacher random effect (random intercept) divided by the sum of variance for the teacher random effect and error variance. A higher ICC was taken as stronger evidence for a teacher effect.

Findings / Results
Fidelity assessments were completed on 92 lessons (22% of all lessons; 55 nutrition lessons, 37 physical activity lessons) delivered by 14 teachers. Of the 92 assessments completed there were 79 structural fidelity assessments and 77 instructional assessments with complete data. Eighty-four percent of the lessons evaluated for structural summary fidelity scored 3 or greater (maximum of 4) and 27% of the total had the maximum score of 4 ( Figure 1). Sixty-seven percent of the teachers scored 13 or greater on the instructional fidelity score (maximum of 16) and 60% of the total had the maximum score of 16.

Figure 1. Frequency of lessons receiving respective structural and instructional summary scores
The associations of lesson type and number of children with individual structural and instructional questions are shown in Table 1. There was not a significant association of the lesson type or the number of children for any of the structural and instructional questions queried in FOI. The delivery of the lessons was independent of the lesson type (nutrition or physical activity). For some of the structural and instructional questions (Table 1) the ICC was 0.4 or greater, which indicates significant correlation for scores within teachers and is consistent with a stronger influence of teachers for these aspects of lesson delivery. For example, the ICC for "Referral to leader guide/materials throughout lesson" was .59; and indicates this aspect of delivery was more variable for lessons given by different teachers than for lessons given by the same teacher. The associations of lesson type and number of children with summary structural and instructional measures are shown in Table 2. For the summary structural fidelity measure, there was no significant association of lesson type or number of children. The ICC was 0.30 indicating significant correlation of measures within teacher. For the summary instructional fidelity measure, there was no significant association with number of children, and there was weak evidence for an association with lesson type (p = 0.08). The weak association suggests that mean instructional fidelity for physical activity lessons was higher than nutrition, which was consistent with all the OR's for instructional questions in Table 1 being greater than 1.0. The ICC was 0.25 indicating significant correlation of measures within teacher.

Discussion
The focus of this paper is to report on the FOI of iGrow Readers that was delivered to preschool children through TTT approach. iGrow Readers, a literacy-based curriculum that uses children's books focusing on themes of healthy nutrition and physical activity behaviors was implemented by Extension personnel in the natural setting using TTT approach. In interventions such as iGrow Readers that use multiple educators and/or non-experts, FOI is necessary to determine if delivery of the instructional program is as it was designed to be and is important for valid assessment of program effectiveness (O'Donnell, 2008). This is particularly important in field intervention trials in which nonresearchers deliver the intervention. Furthermore, assessing the FOI of the curriculum delivered by TTT approach provides evidence that the TTT approach is a feasible method of delivery for programing provided by Extension personnel particularly if the FOI structural and instructional summary scores are high as reported in the iGrow Readers intervention reported in this paper. Durlak and DuPre (2008) have reported that programs with seemingly better implementation versus poorer implementation have resulted in mean effect sizes that are two to three times higher.
Both the structural and instructional fidelity scores for the iGrow teachers are high compared to Durlak's interpretation of what constitutes a good score. Positive results for implementation have been obtained with levels around 60% and few studies have reached levels greater than 80% (Durlak & DuPre, 2008).
However, even with the high structural and instructional implementation, there was variation in responses to the individual FOI structural and instructional questions that was attributed to the teacher delivering the lesson. When evaluating the FOI scores between the nutrition and physical activity lessons, the odds of delivering the nutrition lessons as directed were less than the delivery of the physical activity lessons as directed. This may indicate that preschool teachers are more comfortable with including physical activity into their curriculum versus that of the factbased nutrition education activities. Research has shown that teachers are more likely to teach a subject if they have a background in the subject. For example, if a person has a physical activity background or instruction on how to incorporate movement into the classroom, they are more likely to teach that subject compared to others (Brener et al., 2013).
There was little variation among the teachers in the structural fidelity questions that assessed if the teachers met the objective of the lesson and introduced the lesson. This may indicate that those in the profession of leading and preparing lessons for preschool have been introduced to curriculum delivery as part of their profession (Lavin, 1993). There was more variability in structural assessment of adequacy of materials and referring to the leader guide. These two questions are assessing the teacher's preparedness for the lesson. Thus, indicating that more emphasis on preparing for the teaching of a curriculum may be needed when providing training for trainers who deliver curriculums to preschoolers, particularly in states that have less stringent oversite and requirements for the pre-kindergarten education (NIEER, 2016;Putman et al., 2016).
Also, with the instructional fidelity questions, there was little variation among the teachers for effectiveness of teachers at facilitating group participation in an activity and involvement of the children in the activity. Like the structural questions, these two questions may be assessing the core competency required of early childhood educators in their profession of managing children in daily activities (Theobald et al., 2011). There was more variability in those questions that queried for the teacher specific characteristics of involvement of teacher in the activity, interest of the teacher in the class, and interest of the children in the class. Those teachers who had greater interest in the material may have shown greater interest and involvement in the material. Others have documented this phenomenon that a greater interest in the material by the teacher may lead to children having a greater interest in the material (Brener et al., 2013;Lavin, 1993). It also has been documented that teachers with greater job satisfaction (Dinc & Kocyigit, 2017) and greater academic achievement (Uyar et al., 2018) have a greater commitment to teaching. Although, job satisfaction was not evaluated in the iGrow readers FOI, the instructional component of FOI was measured. The instructional component of the FOI did evaluate the teachers and the child's involvement and interest in the activities, as well as, the teacher effectiveness in facilitating the group and preparation for the class.

Conclusion
The FOI scores indicate that curriculum in this intervention was implemented as intended; thus, supporting that the TTT approach used by Extension personnel is a feasible method of delivery for nutrition and physical activity curriculum in the natural settings such as preschools.

Suggestions
The results suggest that TTT is an effective method to deliver nutrition and physical activity curriculum to preschoolers in field intervention studies in natural settings. Future studies are needed to assess the extent to which variation among teachers may be reduced by ensuring teachers understand their role as the implementer in teaching the children, either through increased training, more specific instructions on intervention delivery, or both. Additional training may take more time and resources which often are limited when disseminating community programing. The planners will need to balance the time allocated for training-the-trainer with the expected outcome from the curriculum.

Limitations
However, there are some limitations that need to be considered when interpreting the results. To accommodate the number of FOI measurements that needed to be done, not all the observations could be done in person. Those that could not be assessed in person were assessed using video recordings. There may be differences between in-person FOI and recorded FOI evaluations. A strength of the study is that the same evaluator conducted the in-person and recorded evaluations. Furthermore, the evaluators' consistency was validated by having a second content expert evaluate 25 percent of the recorded lessons. An additional limitation is that this fidelity assessment was not based on a balanced design in that the field implementation of the study hindered control over the particular books and lessons used for fidelity assessment. Nonetheless, the large number of lessons evaluated should provide reliable assessments of relevant potential influences (e.g., teacher and lesson types) on FOI.