Analysis

Each curriculum was analyzed separately due to the independence of the research teams, the nonrandom assignment of curricula to research teams and sites, and the differences in control conditions. Because students were nested in classrooms or programs and repeatedly assessed with multiple measures, multi-level models containing a series of student, teacher, and classroom-level covariates were used to address the cross-level correlated errors, allowing for a mixture of random and fixed effects (see appendix B for details). For each curriculum, these models were used to estimate differences between treatment and control group means for each of the 27 outcome measures. The type of model used to analyze each outcome measure depended on the number of time points it was observed.

Two types of models for repeated measures (spline and simple) were used for outcome measures with comparable data from two or three time points. Analysis of covariance (ANCOVA) was conducted for outcome measures observed at one time point. The more observations of a measure from different time points included in a model, the better able the model is to identify the parameters of interest, in this case the treatment and control group means of the measures. For this reason, the spline repeated measures model is the preferred model followed by the simple repeated measures model, and then the ANCOVA. The analysis of each measure uses the most preferred model that can be used given the number of time points the measure was observed. Table D lists the model used with each measure.

For the eight student-level outcome measures with observations at three time points, a repeated measures spline model was used to compare the treatment and control group means for the spring pre-kindergarten and spring kindergarten observations. In addition, the model was used to check for differences in group mean measures at the baseline observation, check for such differences at the start of treatment if there was a lag between curriculum implementation and the baseline data collection, and compare the mean rates of growth for the treatment and control groups in pre-kindergarten and in kindergarten (the statistical techniques used are discussed in appendix B and the results from these three analyses are provided in appendix A). For the four student-level outcome measures and five classroom-level outcome measures with observations at two time points, a simple repeated measures model was used to compare the treatment and control group means at spring pre-kindergarten. Similarly, it was used to check on group mean differences at the baseline and start of treatment, and compare the rates of growth in pre-kindergarten.

ANCOVA models were used to estimate the difference in mean outcome measures between the treatment and control group in the spring of pre-kindergarten or kindergarten when only one observation was available. The availability of only one observation of a measure occurred in two situations. First, four of the kindergarten student measures (the CTOPP, SSRS Social Skills, SSRS Problem Behaviors, and LBS) were not on the same scales as the pre-kindergarten measures. The ANCOVA model for these kindergarten measures included students’ scores on the respective pre-kindergarten scale as a covariate to address any differences in the groups that occurred, despite randomization. Second, six pre-kindergarten classroom instruction measures were based on the TBRS that was given only in the spring of pre-kindergarten. Group mean differences for these were estimated using an ANCOVA without a similar baseline covariate. These models may be biased by any initial differences in instruction that may have existed despite randomization, as there is no baseline measure.

Top