Glossary of Terms: What Works Clearinghouse

The What Works Clearinghouse (WWC) seeks to provide users with the information they need to be effective consumers of education research. Please review our glossary to better understand the terms used by the WWC.

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Top

A

analytic sample: The sample on which the analysis was based.
attrition: The loss of original sample members for the analysis. High levels of attrition can call into question the validity of results for a randomized controlled trial (RCT). An RCT with high attrition cannot receive the highest rating of Meets Evidence Standards, but can receive a rating of Meets Evidence Standards with Reservations if it establishes baseline equivalence of the analytic sample.
author query: A request sent to study authors for additional or clarifying information needed to rate a study. Reasonable attempts are made to contact the first or subsequent authors and provide them time to respond to the query. Information garnered from author queries is public information and noted in the report. Studies in which there was no response to an author query, or in which the author could not provide needed information, are rated based on the information presented in the study.

Top

B

baseline: A point in time before the analysis was conducted. When studies are required to demonstrate equivalence, it must be on characteristics of the analytic sample at baseline.

Top

C

causal: Evidence that an observed effect is a consequence of the intervention.
certified reviewer: A reviewer who has attended WWC training and passed the certification process. WWC certified reviewers are responsible for assessing studies against WWC standards and assigning a study rating. There are separate training and certification processes for (a) randomized controlled trials and quasi-experimental designs, (b) regression discontinuity designs, and (c) single-case designs.
clustering correction: An adjustment to the statistical significance when the units of assignment and analysis differ. For example, a study may have assigned entire classrooms (the unit of assignment) to the intervention and comparison conditions, but then analyzed data at the individual student level. If not accounted for when estimating impacts, this type of mismatch can lead to inflated estimates of statistical significance. Therefore, the WWC uses a t-test applied to the effect size that incorporates the number of clusters and intra-class correlation.
comparison group: The analysis group that does not receive the intervention.
comparison group design: A study design in which outcomes for a group receiving the intervention are compared to those for a group not receiving the intervention. Comparison group designs eligible for WWC review are randomized controlled trials, quasi-experimental designs, and regression discontinuity designs.
confounding factor: A component of a study that is completely aligned with one of the study conditions. For example, a study may have one intervention school and a different comparison school. In this case, it is impossible to separate how much of the observed effect was due to the intervention and how much was due to the particular school in which the intervention was used. Therefore, a study with a confounding factor cannot meet standards, as the impact cannot be attributed solely to the intervention.
content expert: A well-established researcher with substantive expertise in a topic. Each topic has a content expert, who serves as a consultant to assist the principal investigator with content-specific questions that arise in reviews. Content experts are also used in the development of quick reviews. The principal investigator may consult with a content expert to determine key outcomes or subgroups on which the quick review should focus.
counterfactual: The impact of an intervention on a group is always relative to a comparison group or counterfactual. The counterfactual may not be receiving any services or intervention, or may be receiving a different intervention. For group design studies, an analysis comparing groups receiving different dosage levels or different versions of a single intervention is not an acceptable counterfactual.

Top

D

delivery method: The way in which an intervention was implemented: individual, small group, whole class, or whole school. “Individual” means that students work independently or one-on-one with the teacher, paraprofessional, parent, or other service provider. “Small group” means that the intervention is delivered with students working in small groups; groups can be any size, as long as there are at least two students per group. For the “whole class” method, the intervention is delivered with students working with the teacher as an entire class a majority of the time. The “whole school” method means that the intervention is provided school-wide, not as an individual classroom approach.
developer/distributor: The person or company from whom the intervention can be purchased. The WWC solicits feedback from the distributor and/or developer of a product or practice when beginning a review of an intervention. The WWC notifies the distributor/developer of the WWC review and provides a list of all WWC-identified citations related to the intervention, inquires if the list is complete, invites comment on the intervention description slated for use in the report, and requests that the distributor/developer sign an agreement not to disclose any information about the review prior to its release. If the initial contact is the distributor of the intervention, but not the original developer, the distributor is asked whether they wish to name an individual to also receive the materials to review and comment.
difference-in-differences adjustment: An adjustment to the effect size to account for pre-intervention differences. For unadjusted outcomes that meet WWC standards (with or without reservations), the WWC will recalculate the intervention comparison difference or effect size, taking into account pre-intervention differences when available. Specifically, we calculate the impact as the difference between the pre- and post-intervention mean difference for the intervention group and the pre- and post-intervention mean difference for the comparison group.
differential attrition: The difference in attrition rates between the intervention and comparison groups. High levels of differential attrition in a randomized controlled trial suggest that the intervention group (as analyzed) may differ from the control group (as analyzed) in ways other than the treatment. Differential and overall attrition rates are considered in determining whether the attrition level is high enough to warrant requiring a demonstration of baseline equivalence of the analytic sample.
does not meet eligibility screens: A study that is out of the scope of the review protocol. Studies outside of the scope can include those without causal designs or studies with causal designs that are ineligible for review (for example, age or grade range outside the range specified in the protocol).
does not meet evidence standards: A study with a low level of causal evidence. This is the rating given to studies with causal research designs that were not implemented rigorously enough to conclude with confidence that the intervention caused the observed changes in outcomes.
Doing What Works: A companion website to help educators make use of WWC practice guides.
domain: A group of closely-related outcomes. It is the organizing construct for a set of related outcomes through which intervention studies claim effectiveness (for example, fluency in Literacy). In intervention reports and practice guides, the WWC assesses the rigor of evidence on the effectiveness of interventions within each domain identified in the protocol. The intervention rating, improvement index, and extent of evidence are determined at the domain level.

Top

E

effect size: A standardized measure of the effect of an intervention on student outcomes. The effect size represents the change (measured in standard deviations) in an average student’s outcome that can be expected if that student is given the intervention. Because effect sizes are standardized, they can be compared across outcomes and studies.
effective: An intervention is effective if it improves outcomes relative to what would have been seen without the intervention.
effectiveness: An intervention demonstrates effectiveness if the research has shown that the program caused an improvement in outcomes.
effectiveness rating: The WWC summary of findings from research on an intervention. In intervention reports, the WWC rates the effects of an intervention in each outcome domain as one of the following: positive effects, potentially positive effects, mixed effects, potentially negative effects, negative effects, and no discernible effects. The rating of effectiveness takes into account four factors: the quality of the research on the intervention, the statistical significance of the research findings, the size of the differences between participants in the intervention and comparison groups, and the consistency in findings across studies.

Effectiveness Rating Key
Positive Effects: strong evidence of a positive effect with no overriding contrary evidence. Potentially Negative Effects: evidence of a negative effect with no overriding contrary evidence.

Potentially Positive Effects: evidence of a positive effect with no overriding contrary evidence. Negative Effects: strong evidence of a negative effect with no overriding contrary evidence.

Mixed Effects: evidence of inconsistent effects. No Discernible Effects: no affirmative evidence of effects.
eligibility: A study is eligible for review if it falls within the scope of a review protocol and uses a causal design.
emotional disturbance: Social, emotional or behavioral functioning that departs from the generally accepted, age appropriate ethnic or cultural norms and adversely affects the academic progress, social relationships, and personal adjustment of a child.

Emotional disturbance is defined in the Federal Register as a “condition exhibiting one or more of the following characteristics over a long period of time and to a marked degree that adversely affects a child’s educational performance:
An inability to learn that cannot be explained by intellectual, sensory, or health factors.

An inability to build or maintain satisfactory interpersonal relationships with peers and teachers.

Inappropriate types of behavior or feelings under normal circumstances.

A general pervasive mood of unhappiness or depression.

A tendency to develop physical symptoms or fears associated with personal or school problems.
Emotional disturbance includes schizophrenia, but does not apply to children who are socially maladjusted, unless it is determined that they have an emotional disturbance” (Assistance to States for the Education of Children with Disabilities and Preschool Grants for Children with Disabilities, 2006).
equivalence: A demonstration of the similarity of the analysis groups at baseline. Randomized controlled trials with high attrition and quasi-experimental designs must establish that the intervention and comparison groups used in the analysis were equivalent on observable characteristics at baseline. Characteristics for which equivalence must be established are outlined in the protocol for a review.
evidence standards: The set of standards used by the WWC to review studies. These standards are used to evaluate the strength of the evidence of the effectiveness of educational interventions. Studies are given a rating of Meets Evidence Standards, Meets Evidence Standards with Reservations, and Does Not Meet Evidence Standards.
extent of evidence: An indicator of how much evidence was used to determine the rating of effectiveness of an intervention and how broadly findings may be applied to different settings. The extent of evidence categorization for intervention reports focuses on the number and sizes of studies of the intervention. This scheme has two categories: small and medium to large. Domains with a small extent of evidence include only one study, or one school, or findings based on a total sample size of less than 350 students and 14 classrooms (assuming 25 students in a class). Domains with a medium to large extent of evidence include more than one study, more than one school, and findings based on a total sample of at least 350 students or 14 classrooms.

Effectiveness Rating Key
	Positive Effects: strong evidence of a positive effect with no overriding contrary evidence.		Potentially Negative Effects: evidence of a negative effect with no overriding contrary evidence.
	Potentially Positive Effects: evidence of a positive effect with no overriding contrary evidence.		Negative Effects: strong evidence of a negative effect with no overriding contrary evidence.
	Mixed Effects: evidence of inconsistent effects.		No Discernible Effects: no affirmative evidence of effects.

Top

F

forcing variable: The variable used to assign units to groups in a regression discontinuity design.

Top

I

improvement index: Along a percentile distribution of students, the improvement index represents the change in an average student’s percentile rank that can be expected due to the intervention. Given that the average student starts at the 50th percentile, the improvement index values range from -50 to +50 with positive numbers denoting results favorable to the treatment group. Unlike the evidence rating, the improvement index is based entirely on the size of the effect, regardless of the statistical significance of the effect, the study design, or the analyses.
indeterminate effect: An effect that is neither statistically significant nor substantively important.
Institute of Education Sciences (IES): The research arm of the U.S. Department of Education. Established by the Education Sciences Reform Act of 2002, IES’s mission is to expand knowledge and provide information on the condition of education, practices that improve academic achievement, and the effectiveness of federal and other education programs.
intervention: An educational program, product, practice, or policy aimed at improving student outcomes.
intervention group: The group in an analysis that receives the intervention.
intervention rating: The WWC summary of findings from research on an intervention. In intervention reports, the WWC rates the effects of an intervention in each outcome domain as one of the following: positive effects, potentially positive effects, mixed effects, potentially negative effects, negative effects, and no discernible effects. The rating of effectiveness takes into account four factors: the quality of the research on the intervention, the statistical significance of the research findings, the size of the differences between participants in the intervention and comparison groups, and the consistency in findings across studies.

Effectiveness Rating Key
Positive Effects: strong evidence of a positive effect with no overriding contrary evidence. Potentially Negative Effects: evidence of a negative effect with no overriding contrary evidence.

Potentially Positive Effects: evidence of a positive effect with no overriding contrary evidence. Negative Effects: strong evidence of a negative effect with no overriding contrary evidence.

Mixed Effects: evidence of inconsistent effects. No Discernible Effects: no affirmative evidence of effects.
intervention report: A summary of findings of the most reliable research on a given program, practice, or policy in education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, and summarizes the findings of those that meet standards.
intra-class correlation: A measure of the relatedness of outcome measures. Based on empirical literature in the field of education, the WWC has adopted a default value for clustering corrections of 0.20 for academic outcomes and 0.10 for behavioral outcomes.

Effectiveness Rating Key
	Positive Effects: strong evidence of a positive effect with no overriding contrary evidence.		Potentially Negative Effects: evidence of a negative effect with no overriding contrary evidence.
	Potentially Positive Effects: evidence of a positive effect with no overriding contrary evidence.		Negative Effects: strong evidence of a negative effect with no overriding contrary evidence.
	Mixed Effects: evidence of inconsistent effects.		No Discernible Effects: no affirmative evidence of effects.

Top

L

level of evidence rating: The amount of evidence associated with a practice guide recommendation. The ratings takes the following factors into account: the number of studies, the quality of the studies, whether the studies represent the range of participants and settings on which the recommendation is focused, whether the findings from the studies can be attributed to the recommended practices, and whether the findings in the studies are consistently positive.

Top

M

meets evidence standards: The highest rating for a study reviewed by the WWC. Studies receiving this rating provide the highest degree of confidence that an observed effect was caused by the intervention. Only well-implemented randomized controlled trials (RCTs) and regression discontinuity designs (RDs) that do not have problems with attrition may receive this highest rating.
meets evidence standards with reservations: The middle rating for a study reviewed by the WWC. Studies receiving this rating provide a lower degree of confidence that an observed effect was caused by the intervention. Randomized controlled trials (RCTs) and regression discontinuity designs (RDs) that are not as well-implemented or have problems with attrition, along with strong quasi-experimental designs, may receive this rating.
methodological lead: An expert with strong skills in the theoretical analysis of methods used in research.
minimal evidence: No causal evidence for the recommendation exists. However, the recommendation is still considered as important as recommendations with strong and moderate levels of evidence (this rating was formerly low evidence). The panel cannot point to a body of research that demonstrates the practice’s positive effect on student achievement. In some cases, this simply means that the recommended practices would be difficult to study in a rigorous, experimental fashion; in other cases, it means that researchers have not yet studied this practice, or that there is weak or conflicting evidence of effectiveness. A minimal evidence rating does not indicate that the recommendation is any less important than other recommendations with a strong evidence or moderate evidence rating.
mixed effects: An effectiveness rating that indicates evidence of inconsistent effects for an intervention. This is demonstrated through either of the following: a) at least one study shows a statistically significant or substantively important positive effect and at least one study shows a statistically significant or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect; or b) at least one study shows a statistically significant or substantively important effect and more studies show an indeterminate effect than show a statistically significant or substantively important effect.
moderate evidence: Evidence exists for the recommended strategy, but the evidence may not be generalizable to a wide student population or show a clear causal relationship. This rating is assigned when the panel found high quality causal research that links a practice with positive results, but the research may not adequately rule out other causes of the positive results, or the school and classrooms are not similar to those targeted by the guide.
multiple comparison adjustment: An adjustment to the statistical significance of results to account for multiple comparisons. The WWC uses the Benjamini-Hochberg correction to adjust the statistical significance of results within a domain when study authors perform multiple hypothesis tests simultaneously without adjusting the p-value.

Top

N

negative effects: Strong evidence of a negative effect for an intervention. This occurs when two or more studies show statistically significant negative effects, at least one of which met WWC standards for a strong design, and no studies show statistically significant or substantively important positive effects.
no discernible effects: No affirmative evidence of effects for an intervention. This occurs when none of the studies shows a statistically significant or substantively important effect, either positive or negative.

Top

O

outcome: Knowledge, skills, attitudes, and other desired benefits that are attained as a result of an activity. To examine the effectiveness of an intervention for the WWC, eligible research must compare the outcome for a group receiving the intervention to the outcome for a group not receiving the intervention. An outcome measure is an instrument, device, or method that provides data on the outcome. An outcome domain is a group of closely related outcome measures, believed to provide information on the same underlying skill or ability.
outcome domain: A group of closely-related outcomes. It is the organizing construct for a set of related outcomes through which intervention studies claim effectiveness (for example, fluency in Literacy). In intervention reports and practice guides, the WWC assesses the rigor of evidence on the effectiveness of interventions within each domain identified in the protocol. The intervention rating, improvement index, and extent of evidence are determined at the domain level.
overall attrition: The attrition rate of the combined intervention and comparison groups. Differential and overall attrition rates are considered in determining whether the attrition level is high enough to warrant requiring a demonstration of baseline equivalence of the analytic sample.

Top

P

panel chair: A nationally recognized researcher with expertise in a practice guide topic. The panel chair assists in identifying and recruiting panel members, developing the review protocol and literature search parameters, and overseeing the development of the guide.
peer review: A review process to gather external feedback on WWC products before releasing them to the public.
population: The subset of students that is the focus of an intervention. Some topics require students to have certain characteristics, such as English language learners or special education students.
positive effects: Strong evidence of a positive effect for an intervention. This occurs when two or more studies show statistically significant positive effects, at least one of which met WWC evidence standards for a strong design and no studies show statistically significant or substantively important negative effects.
potentially negative effects: Evidence of a negative effect for an intervention. This occurs when a) one study shows a statistically significant or substantively important negative effect and no studies show a statistically significant or substantively important positive effect; or b) two or more studies show statistically significant or substantively important negative effects, at least one study shows a statistically significant or substantively important positive effect, and more studies show statistically significant or substantively important negative effects than show statistically significant or substantively important positive effects.
potentially positive effects: Evidence of a positive effect for an intervention. This occurs when at least one study shows a statistically significant or substantively important positive effect, no studies show a statistically significant or substantively important negative effect, and fewer or the same number of studies show indeterminate effects than show statistically significant or substantively important positive effects.
practice guide: Recommendations for educators to address challenges in their classrooms and schools. These are based on reviews of research, the experiences of practitioners, and the expert opinions of a panel of nationally recognized experts.
principal investigator: An expert on a topic, with or without strong methodological skills. Principal investigators are responsible for leading reviews in topic areas and quick reviews.
prioritization: The process by which interventions are selected for review.
program type: The way in which an intervention was used: curriculum, supplement, or practice. A curriculum defines what topics must be understood, and at what level, to achieve a particular grade or standard. A curriculum is typically something a school or district would purchase to implement in its schools and/or classrooms. A supplement is delivered in addition to the base curriculum. A practice is a lesson or repeated instruction intended to introduce, develop, or improve a skill. A practice is not a branded intervention or product, but rather a method, approach, or strategy for teaching.
properly aligned: The units of assignment and analysis are the same.
protocol: A formal document developed to determine the scope of a review. Protocols include key definitions, express the types of research studies and interventions reviewed, set reviewed parameters, and explain methodology.

Top

Q

quality assurance: An internal review process for WWC products.
quality review team: An independent team that responds to concerns about WWC reports.
quasi-experimental design: A design in which groups are not created randomly. For a quasi-experimental design to be rigorous, the intervention and comparison groups must be similar, demonstrating baseline equivalence on observed characteristics, before the intervention is started. Strong quasi-experimental designs will, at best, be rated as Meets Evidence Standards with Reservations.
quick review: A timely and objective assessment of recent research. To be eligible for the quick review process, a study must: a) examine the effectiveness of an education or school-based program in improving student outcomes, b) be recently released to the public and reported in a major national news source, and c) use a design for which the WWC has developed evidence standards.

Top

R

randomized controlled trial: A design in which group assignment is random. Carried out correctly, random assignment results in groups that are similar on average in both observable and unobservable characteristics, and any differences in outcomes between the groups are due to the intervention alone. Strong randomized controlled trials can receive the highest rating of Meets Evidence Standards.
recommendation: Practice guides provide practical, evidence-based guidance to teachers and administrators on how to address current challenges in education through a series of recommendations. Practice guide recommendations reflect the findings of research on the effectiveness of educational practices, as well as the experience and judgment of the practice guide panel. Each recommendation is a distinct principle that the panel suggests schools or districts implement to address and educational challenge. The panel assigns each recommendation a level of evidence rating that reflects the evidence base that supports the recommendation.
registries: Online databases of researchers and randomized controlled trials. The registry of evaluation research is an online database of researchers who conduct evaluations of the effectiveness of educational interventions. This resource is designed to help schools, districts, and educational program developers identify potential researchers to conduct studies of effectiveness of educational interventions. The registry of randomized controlled trials is an online database of completed and in-progress randomized controlled trials in education. This resource is designed to help schools, districts, and educational program developers identify research regarding the effectiveness of educational interventions. The information for both registries is supplied by the evaluators. The WWC does not verify the accuracy of the information submitted, nor does it assess their qualifications or endorse any individuals or organizations listed.
regression discontinuity design: A study using a continuous scoring rule for group assignment. For example, students may be assigned to a summer school program if they score below a preset point on a standardized test, or schools may be awarded a grant based on their score on an application. A regression line or curve is estimated for the intervention group and similarly for the comparison group, and an effect occurs if there is a discontinuity in the two regression lines at the cutoff.
research standards: The set of standards used by the WWC to review studies. These standards, also known as evidence standards, are used to evaluate the strength of the evidence of the effectiveness of educational interventions. Studies are given a rating of Meets Evidence Standards, Meets Evidence Standards with Reservations, and Does Not Meet Evidence Standards.

Top

S

single-case design: An approach in which a single case provides its own control for comparison. This design involves repeated, systematic measurement of a dependent variable before, during, and after the active manipulation of an independent variable. It is identified by the following features: an individual case is the unit of intervention administration and data analysis, the case provides its own control for purposes of comparison, and the outcome variable is measured repeatedly within and across different conditions or levels of the independent variable.
standard deviation: A measure of the variation in outcomes. A measure with a small standard deviation would indicate that participants had more similar outcomes than a measure with a large standard deviation.
statistical adjustment: Including baseline measures in a statistical model at the level of the unit of analysis. A number of different techniques can satisfy the statistical adjustment requirement, including regression adjustment, ANCOVA, and hierarchical linear modeling. Randomized controlled trials with high attrition and quasi-experimental designs require statistical adjustment to control for differences in baseline characteristics if the effect size of the difference in baseline characteristics is between 0.05 and 0.25 standard deviations.
statistical significance: The likelihood that a finding is due to a real difference rather than chance. The WWC labels a finding statistically significant if the likelihood that the difference is due to chance is less than five percent.
statistical, technical, and analysis team: Highly experienced researchers who revise and develop the WWC standards.
strong design: A design that receives the highest rating of Meets Evidence Standards.
strong evidence: Consistent evidence exists that the recommended strategy improves student outcomes for a wide population of students. This rating is assigned when the panel found high quality causal research that links a practice with positive results, ruling out other causes of the positive results.
study: An examination of the effect of an intervention on a particular sample and set of outcomes. In most cases, a study is a single article or manuscript that examines the effect of an intervention. A study may also unfold over multiple manuscripts, such as an examination of a beginning reading program that analyzes both immediate and long-term effects of an intervention on a common sample that is reported on in manuscripts produced over several years. In contrast, a single manuscript may contain multiple studies, such as an examination of a dropout prevention program analyzed separately in three different cities.
study design: The method by which treatment and comparison groups are determined. Designs eligible for WWC review are randomized controlled trials, quasi-experimental designs, regression discontinuity designs, and single-case designs.
study rating: The result of the WWC assessment of a study. The rating is based on the strength of the evidence of the effectiveness of the educational intervention. Studies are given a rating of Meets Evidence Standards, Meets Evidence Standards with Reservations, and Does Not Meet Evidence Standards.
study review guide: A spreadsheet used as a tool for certified reviewers to assess a study.
substantively important: An effect size of 0.25 or greater, regardless of statistical significance.
systematic review: Explicit methods used to identify, select, appraise, and extract data from research. A WWC systematic review consists of five steps: 1) developing a review protocol; 2) searching the literature; 3) reviewing studies, which includes screening studies for eligibility, reviewing the methodological quality of each study, and reporting on high quality studies and their findings; 4) combining findings within and across studies; and 5) summarizing the review.

Top

T

topic: An outcome, population, or type of intervention studied by the WWC. Topics are defined by intended outcome (such as improving reading skills), intended population (such as English language learners), and types of interventions (such as education technology) that may improve outcomes for students.

Top

U

unit of analysis: The level at which an analysis is conducted.
unit of assignment: The level at which group assignment is conducted.

Top

W

webinar: A service that allows conferencing events to be shared with remote locations.