Exhibit III.a.1. Scenario III, Missing Data Dependent on Post-test: Data Missing for 5% of Students
| Imact Estimate | Standard Error of Imact Est. | 90% Cl | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Data | Pre-test Data Available? | Estimate | True Imact | Bias | Bias Level | Standard Estimate | Unbiased Estimate | Bias | Bias Level | % of Samples in which 90% Cl Contains .20 |
| No Missing Data | No | 0.203 | 0.200 | 0.003 | Low Bias | 0.085 | 0.088 | -0.002 | Low Bias 0.892 0.892 | |
| Yes | 0.203 | 0.200 | 0.003 | Low Bias | 0.062 | 0.062 | 0.000 | Low Bias | 0.886 | |
| A. Pre-Test (X) Data Missing | ||||||||||
| Case Deletion | No | 0.195 | 0.200 | -0.005 | Low Bias | 0.062 | 0.088 | -0.002 | Low Bias | 0.898 |
| Yes | ||||||||||
| Dummy Variable Method | No | 0.204 | 0.200 | 0.004 | Low Bias | 0.062 | 0.062 | 0.000 | Low Bias | 0.904 |
| Yes | ||||||||||
| Mean Value Imputation | No | 0.210 | 0.200 | 0.010 | Low Bias | 0.062 | 0.060 | 0.002 | Low Bias | 0.900 |
| Yes | ||||||||||
| Single, Non-stochastic | No | 0.202 | 0.200 | 0.002 | Low Bias | 0.062 | 0.060 | 0.002 | Low Bias | 0.899 |
| Regression Imputation | Yes | |||||||||
| Single, Stochastic | No | 0.202 | 0.200 | 0.002 | Low Bias | 0.062 | 0.060 | 0.002 | Low Bias | 0.909 |
| Regression Imputation | Yes | |||||||||
| Multiple Stochastic | No | 0.202 | 0.200 | 0.002 | Low Bias | 0.062 | 0.060 | 0.002 | Low Bias | 0.902 |
| Regrssion Imputation (n=5) | Yes | |||||||||
| EM Algorithm with Multiple Imputation (n=5) | No | 0.202 | 0.200 | 0.002 | Low Bias | 0.062 | 0.060 | 0.002 | Low Bias | 0.904 |
| Yes | ||||||||||
| B. Post-Test (Y) Data Missing | ||||||||||
| Case Deletion | No | 0.187 | 0.200 | -0.013 | Low Bias | 0.086 | 0.084 | 0.001 | Low Bias | 0.895 |
| Yes | 0.195 | 0.200 | -0.005 | Low Bias | 0.062 | 0.060 | 0.002 | Low Bias | 0.898 | |
| Mean Value Imputation | No | 0.185 | 0.200 | -0.015 | Low Bias | 0.081 | 0.084 | -0.003 | Low Bias | 0.876 |
| Yes | 0.186 | 0.200 | -0.014 | Low Bias | 0.059 | 0.060 | -0.001 | Low Bias | 0.868 | |
| Single, Non-stocastic | No | 0.187 | 0.200 | -0.013 | Low Bias | 0.086 | 0.084 | 0.001 | Low Bias | 0.089 |
| Regression Imputation | Yes | 0.196 | 0.200 | -0.004 | Low Bias | 0.062 | 0.060 | 0.002 | Low Bias | 0.898 |
| Single, Stochastic | No | 0.187 | 0.200 | -0.013 | Low Bias | 0.086 | 0.085 | 0.001 | Low Bias | 0.895 |
| Regression Imputation | Yes | 0.195 | 0.200 | -0.005 | Low Bias | 0.062 | 0.061 | 0.002 | Low Bias | 0.895 |
| Multiple, Stochastic | No | 0.188 | 0.200 | -0.012 | Low Bias | 0.086 | 0.084 | 0.002 | Low Bias | 0.898 |
| Regression Imputation (n=5) | Yes | 0.196 | 0.200 | -0.004 | Low Bias | 0.062 | 0.060 | 0.002 | Low Bias | 0.904 |
| EM Algorithm with Multiple Imputation (n=5) | No | 0.188 | 0.200 | -0.012 | Low Bias | 0.086 | 0.084 | 0.002 | Low Bias | 0.902 |
| Yes | 0.195 | 0.200 | -0.005 | Low Bias | 0.062 | 0.060 | 0.002 | Low Bias | 0.900 | |
| Weighting - Simple | No | 0.187 | 0.200 | -0.013 | Low Bias | 0.086 | 0.084 | 0.001 | Low Bias | 0.894 |
| Yes | 0.195 | 0.200 | -0.005 | Low Bias | 0.062 | 0.060 | 0.002 | Low Bias | 0.898 | |
| Weighting - Sophisticated | No | 0.187 | 0.200 | -0.013 | Low Bias | 0.086 | 0.084 | 0.001 | Low Bias | 0.894 |
| Yes | 0.195 | 0.200 | -0.005 | Low Bias | 0.062 | 0.060 | 0.002 | Low Bias | 0.898 | |
| Fully Specified Regression Models | No | 0.195 | 0.200 | -0.005 | Low Bias | 0.062 | 0.060 | 0.002 | Low Bias | 0.898 |
| w/ Treatment-Covariate Interactions | Yes | |||||||||
| Note: When pre-test scores are available, they are used as a covariate in the analysis model. In addition, we used pre-test scores to impute values and create weights. Bias estimates were computed as described in Chapter 4 and repeated at the beginning of this appendix. The level of the bias is characterized as "High Bias" or "Low Bias" based on the criteria established in Chapter 4. 90% CI refers to the 90-percent confidence interval around the impact estimate. For more details on the simulations, see Chapter 4 and Appendix C. | ||||||||||