Impact Evaluation of Academic Instruction For After-School Programs

Contractors: MDRC, Public/Private Ventures, Bloom Associates, and Survey Research Management

Background/Research Questions:

The 21st Century Community Learning Centers Program, established in 1999, is a state-administered grant program to provide opportunities for academic enrichment to meet state and local student academic achievement standards in core academic subjects, such as reading and mathematics. This Title IV, Part B program of the Elementary and Secondary Education Act is funded at $1 billion in FY08 and targets high poverty schools with priority given to schools in need of improvement. Thus, an evaluation of Harcourt's Mathletics and Success For All's Adventure Island—two academic programs designed for the afterschool setting—provides appropriate information for the program.

Building off of a 2005 national evaluation of the 21st Century Community Learning Centers Program in which grantees were found to offer predominantly homework help that did not produce academic impacts, this study tests the efficacy of two structured and intensive academic programs designed for the after-school setting.

• What is the effectiveness of offering one year of the academic program (reading or math)? Does program implementation and the one-year impact improve in a second year of operation?
• What is the effectiveness of offering two years of the academic program (reading or math)?
• What is the association between receiving the academic program (reading or math) for two consecutive years and math achievement?

Design:

The evaluation consists of two parallel studies at the elementary school level (25 programs for each study, one testing the math program—Harcourt Math—and one testing the reading program—Success For All). The evaluation includes initially 1000 students who were randomly assigned to receive treatment and 1000 to receive the typical support (e.g., general help in a variety of subjects). A subset of the sites continued the program for a second year in which students were randomly assigned a second time to receive one year of treatment in the first year of program operation; to receive one year of treatment in the second year of program operation; to two years of services; and to zero years of the academic program.

Duration: 5 years (October 1, 2003–September 30, 2008)

Current Status: The first report was released in June 2008 and the second report was released in September 2009 (see http://ies.ed.gov/ncee/pubs/20084021/).

Key Findings:

• There was a positive impact on math achievement for students offered one year of the math program. Two years of the math program resulted in no difference between those offered and those not offered the math program.
• There was no impact on reading comprehension achievement for students offered one year of the reading program. Two years of the reading program resulted in less gains in reading achievement for those offered relative to those not offered the reading program.