Glossary of Terms: What Works Clearinghouse

The What Works Clearinghouse (WWC) seeks to provide users with the information they need to be effective consumers of education research. Please review our glossary to better understand the terms used by the WWC.

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

Top

A

analytic sample: The sample on which the analysis was based.
attrition: When an outcome variable is not available for all participants initially assigned to a study sample due to the loss of original sample members. If a randomized controlled trial (RCT) has high levels of attrition the validity of the study results can be called into question. An RCT with high attrition cannot receive the highest rating of Meets WWC Evidence Standards without Reservations, but can receive a rating of Meets WWC Evidence Standards with Reservations if it establishes baseline equivalence of the analytic sample.
author query: A request sent to study authors for additional or clarifying information needed to rate a study. Reasonable attempts are made to contact the first or subsequent authors and provide them time to respond to the query. Information garnered from author queries is public information and noted in the report. Studies for which there was no response to an author query or the author could not provide needed information are rated based solely on the information presented in the study. Any information obtained from the authors is considered to be public information, and the source will be noted if it is used in a report.

Top

B

baseline: A point in time before the intervention was implemented. When studies are required to demonstrate equivalence, it must be done with characteristics of the analytic sample at baseline.

Top

C

causal: Evidence that an observed effect is a consequence of the intervention.
certified reviewer: A reviewer who has attended training on WWC standards and passed the certification process. WWC certified reviewers are responsible for assessing studies against WWC standards and assigning a study rating. There are separate training and certification processes for (a) randomized controlled trials and quasi-experimental designs; (b) regression discontinuity designs; and (c) single-case designs.
clustering adjustment: An adjustment to the statistical significance of a study’s results when the units of assignment and analysis differ. A mismatch problem occurs when random assignment is carried out at the cluster level and the analysis is conducted at the student level, but the correlation among students within the same clusters is ingored in the computing the standard errors of the impact estimates. Althought the point estimates of the intervention’s effects based on such mismatched analyses are not affect as a result of ignoring this feature of the study sample, the standard errors of the impact estimates generally will be underestimated, thereby leading to overestimates of statistical significance. To fairly assess an intervention’s effects, in cases where study authors have not corrected for the clustering, the WWC computes clustering-corrected statistical significance estimates.
comparison group: The analysis group that does not receive the intervention being tested.
comparison group design: A study design in which outcomes for a group receiving an intervention are compared to those for a group not receiving the intervention. Comparison group designs eligible for WWC review are randomized controlled trials, quasi-experimental designs, and regression discontinuity designs.
confounding factor: A component of a study that is completely aligned with one of the study conditions. For example, a study may have one intervention school and a different comparison school. In this case, it is impossible to separate how much of the observed effect was due to the intervention and how much was due to the particular school in which the intervention was used. Therefore, a study with a confounding factor cannot meet standards, as the impact cannot be attributed solely to the intervention.
content expert: A well-established researcher with substantive expertise in a topic. Each WWC review has a content expert, who serves as part of the leadership team to assist with the development of the review protocol. The content expert may also handle content-specific questions that arise in reviews, along with determining key outcomes or subgroups on which the review should focus.
counterfactual: The comparison condition. The counterfactual may be receiving a different intervention or not receiving any services or intervention. For group design studies, groups receiving different dosage levels or different versions of a single intervention are not acceptable counterfactuals.

Top

D

delivery method: The method by which an intervention was delivered: individual, small group, whole class, or whole school. An individual method means that students work independently or one-on-one with the teacher, paraprofessional, parent, or other service provider. Small group means that the intervention is delivered with students working in small groups that can be any size, with at least two students per group. For the whole class method, the intervention is delivered to an entire class a majority of the time. With a whole school method the intervention is provided school-wide, not as an individual classroom approach.
design: The method by which intervention and comparison groups are determined. Designs eligible for WWC review are randomized controlled trials, quasi-experimental designs, regression discontinuity designs, and single-case designs.
developer/distributor: The creator of the intervention or the person or company from whom the intervention can be purchased. The WWC solicits feedback from the developer/distributor of a product or practice when beginning a review of an intervention. The WWC notifies the developer/distributor of the WWC review and provides a list of all WWC-identified citations related to the intervention, inquires if the list is complete, invites comment on the intervention description slated for use in the report, and requests that the developer/distributor sign an agreement not to disclose any information about the review prior to its release. If the initial contact is the distributor of the intervention, but not the original developer, the distributor is asked whether they wish to name an individual to also receive the materials to review and comment.
difference-in-differences adjustment: An adjustment to the effect size to account for pre-intervention differences. For unadjusted outcomes that meet WWC evidence standards (with or without reservations), the WWC will recalculate the effect size difference between the intervention and comparison groups, taking into account pre-intervention differences when available. Specifically, the WWC calculates the impact as the difference between the pre- and post-intervention mean difference for the intervention group and the pre- and post-intervention mean difference for the comparison group.
differential attrition: The difference in attrition rates between the intervention and comparison groups. High levels of differential attrition in a randomized controlled trial suggest that the intervention group (as analyzed) may differ from the comparison group (as analyzed) in ways other than due to the intervention. Differential and overall attrition rates are considered in determining whether the attrition level is high enough to warrant requiring a demonstration of baseline equivalence of the analytic sample.
does not meet WWC eligibility screens: A study that is out of the scope of the review protocol. Studies outside of the scope can include those without causal designs or studies with causal designs that are ineligible for review (for example, age or grade range outside the range specified in the protocol).
does not meet WWC evidence standards: A study with a low level of causal evidence. This is the rating given to studies with causal research designs that were not implemented rigorously enough to conclude with confidence that the intervention caused the observed changes in outcomes.
Doing What Works: A companion website (http://dww.ed.gov) that helps educators make use of WWC practice guides and integrate evidence-based practices into classroom instruction.
domain: A group of closely related outcomes. A domain is the organizing construct for a set of related outcomes through which studies claim effectiveness. For example, the WWC alphabetics domain within the Literacy topic includes the following outcomes: phonemic and phonological awareness, letter identification, print awareness, and phonics. In intervention reports and practice guides, the WWC assesses the rigor of evidence on the effectiveness of interventions within each domain identified in the review protocol. The intervention rating, improvement index, and extent of evidence are determined at the domain level.

Top

E

effect size: A standardized measure of the magnitude of an effect. The effect size represents the change (measured in standard deviations) in an average student’s outcome that can be expected if that student is given the intervention. Because effect sizes are standardized, they can be compared across outcomes and studies.
effective: An intervention is effective if it improves outcomes relative to what would have been seen without the intervention.
effectiveness: An intervention demonstrates effectiveness if the research has shown that it caused an improvement in outcomes.
eligibility: A determination of whether a study falls within the scope of a review protocol and uses a causal design.
emotional disturbance: Social, emotional or behavioral functioning that departs from the generally accepted, age appropriate ethnic or cultural norms and adversely affects the academic progress, social relationships, and personal adjustment of a child.

Emotional disturbance is defined in the Federal Register as a “condition exhibiting one or more of the following characteristics over a long period of time and to a marked degree that adversely affects a childs educational performance:
An inability to learn that cannot be explained by intellectual, sensory, or health factors.

An inability to build or maintain satisfactory interpersonal relationships with peers and teachers.

Inappropriate types of behavior or feelings under normal circumstances.

A general pervasive mood of unhappiness or depression.

A tendency to develop physical symptoms or fears associated with personal or school problems.
Emotional disturbance includes schizophrenia, but does not apply to children who are socially maladjusted, unless it is determined that they have an emotional disturbance” (Assistance to States for the Education of Children with Disabilities and Preschool Grants for Children with Disabilities, 2006).
equivalence: A demonstration of the similarity of the analysis groups at baseline. Randomized controlled trials with high attrition and quasi-experimental designs must establish that the intervention and comparison groups used in the analysis were equivalent on observable characteristics at baseline. Characteristics for which equivalence must be established are outlined in the protocol for a review.
evidence standards: The set of standards used by the WWC to review studies. These standards are used to evaluate the strength of the evidence of the effectiveness of educational interventions. Studies are given a rating of Meets WWC Evidence Standards without Reservations, Meets WWC Evidence Standards with Reservations, or Does Not Meet WWC Evidence Standards.
extent of evidence: An indicator of how much evidence supports the findings in an intervention report.

The extent of evidence categorization for intervention reports focuses on the number and sizes of studies of the intervention in order to give an indication of how broadly findings may be applied to different settings. There are two extent of evidence categories: small and medium to large.
small: includes only one study, or one school, or findings based on a total sample size of less than 350 students and 14 classrooms (assuming 25 students in a class)
medium to large: includes more than one study, more than one school, and findings based on a total sample of at least 350 students or 14 classrooms

Top

F

forcing variable: The variable used to assign units to groups in a regression discontinuity design.

Top

I

improvement index: The expected change in percentile rank for an average comparison group student if the student had received the intervention. It is measured as the percentile difference between the intervention group mean the comparison group mean using the comparison group distribution. In this example, the estimated average impact of the intervention is a 0.4 standard deviation improvement in reading scores. An average comparison group student (at the 50th percentile) would be expected to have scored 0.4 standard deviations above the mean if he or she had received the intervention, or at the 66th percentile of students. The resulting improvement index is +16, corresponding to moving performance for the average student from the 50th to the 66th percentile of the comparison group distribution.
indeterminate effect: An effect that is neither statistically significant nor substantively important.
Institute of Education Sciences: The research arm of the U.S. Department of Education. Established by the Education Sciences Reform Act of 2002, IES’s mission is to expand knowledge and provide information on the condition of education, practices that improve academic achievement, and the effectiveness of federal and other education programs.
intervention: An educational program, product, practice, or policy aimed at improving student outcomes.
intervention group: The group in an analysis that receives the intervention being tested.
intervention report: A summary of findings of the highest-quality research on a given program, practice, or policy in education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, and summarizes the findings of those that meet standards.
intra-class correlation: A measure of the relatedness of outcome measures in a cluster. A mismatch problem occurs when random assignment is carried out at the cluster level and the analysis is conducted at the student level, but the correlation among students within the same clusters is ignored in the computing the standard errors of the impact estimates. Although the point estimates of the intervention’s effects based on such mismatched analyses are not affect as a result of ignoring this feature of the study sample, the standard errors of the impact estimates generally will be underestimated, thereby leading to overestimates of statistical significance. To fairly assess an intervention’s effects, in cases where study authors have not corrected for the clustering, the WWC computes clustering-corrected statistical significance estimates. Based on empirical literature in the field of education, the WWC has adopted a default value for the intra-class correlation of 0.20 for academic outcomes and 0.10 for behavioral outcomes.

Top

L

level of evidence rating: The amount of evidence associated with a practice guide recommendation. The rating takes the following factors into account: the number of studies, the quality of the studies, whether the studies represent the range of participants and settings on which the recommendation is focused, whether the study findings can be attributed to the recommended practices, and whether the study findings are consistently positive. The level of evidence for a recommendation may be strong, moderate, or minimal.

Top

M

meets WWC evidence standards with reservations: The middle possible rating for a study reviewed by the WWC. Studies receiving this rating provide a lower degree of confidence that an observed effect was caused by the intervention. Randomized controlled trials and regression discontinuity designs that are not as well implemented or have problems with attrition, along with strong quasi-experimental designs, may receive this rating.
meets WWC evidence standards without reservations: The highest possible rating for a study reviewed by the WWC. Studies receiving this rating provide the highest degree of confidence that an observed effect was caused by the intervention. Only well-implemented randomized controlled trials and regression discontinuity designs that do not have problems with attrition may receive this highest rating.
methodological lead: An expert with strong skills in the empirical methods used in research.
minimal evidence: No causal evidence for a practice guide recommendation. The panel cannot point to a body of research that demonstrates the practices positive effect on student achievement. In some cases, this simply means that the recommended practices would be difficult to study in a rigorous, experimental fashion. In other cases, it indicates that researchers have not yet studied this practice, or that there is weak or conflicting evidence of effectiveness. A minimal evidence rating does not indicate that the recommendation is any less important than other recommendations with a strong evidence or moderate evidence rating.
mixed effects: Mixed: evidence that intervention’s effect on outcomes is inconsistent.

This is demonstrated through either of the following: a) at least one study shows a statistically significant or substantively important positive effect and at least one study shows a statistically significant or substantively important negative effect, but no more such studies than the number showing a statistically significant or substantively important positive effect; or b) at least one study shows a statistically significant or substantively important effect and more studies show an indeterminate effect than show a statistically significant or substantively important effect.
moderate evidence: Limited evidence for a practice guide recommendation. This rating is assigned when the panel finds high quality causal research that links a practice with positive results, but the research may not adequately rule out other causes of the positive results, or the school and classrooms are not similar to those targeted by the guide.
multiple comparison adjustment: An adjustment to the statistical significance of results to account for multiple comparisons. The WWC uses the Benjamini-Hochberg (BH) correction to adjust the statistical significance of results within a domain when study authors perform multiple hypothesis tests simultaneously without adjusting the p-value. The BH correction is used in three types of situations: studies that tested multiple outcome measures in the same outcome domain with a single comparison group, studies that tested a given outcome measure with multiple comparison groups, and studies that tested multiple outcome measures in the same outcome domain with multiple comparison groups. Because repeated tests of highly correlated constructs will lead to a greater likelihood of mistakenly concluding that the impact was different from zero, in all three situations, the WWC uses the BH correction to reduce the possibility of making this error.

Top

N

negative effects: Negative: strong evidence that intervention had a negative effect on outcomes.

This occurs when two or more studies show statistically significant negative effects, at least one of which met WWC standards for a strong design, and no studies show statistically significant or substantively important positive effects.
no discernible effects: No Discernible: no evidence that intervention had an effect on outcomes.
This occurs when none of the studies shows a statistically significant or substantively important effect, either positive or negative.

Top

O

outcome: Knowledge, skills, attitudes, and other desired benefits that are attained as a result of an activity. To examine the effectiveness of an intervention for the WWC, eligible research must compare the outcome for a group receiving the intervention to the outcome for a group not receiving the intervention. An outcome measure is an instrument, device, or method that provides data on the outcome. An outcome domain is a group of closely related outcome measures, believed to provide information on the same underlying skill or ability.
outcome domain: A group of closely-related outcomes. A domain is the organizing construct for a set of related outcomes through which studies claim effectiveness. For example, the WWC alphabetics domain within the Literacy topic includes the following outcomes: phonemic and phonological awareness, letter identification, print awareness, and phonics. In intervention reports and practice guides, the WWC assesses the rigor of evidence on the effectiveness of interventions within each domain identified in the review protocol. The intervention rating, improvement index, and extent of evidence are determined at the domain level.
overall attrition: The attrition rate of the combined intervention and comparison groups. High levels of overall attrition in a randomized controlled trial suggest that the intervention group (as analyzed) may differ from the comparison group (as analyzed) in ways other than due to the intervention. Differential and overall attrition rates are considered in determining whether the attrition level is high enough to warrant requiring a demonstration of baseline equivalence of the analytic sample.

Top

P

panel chair: A nationally recognized researcher with expertise in a practice guide topic. The panel chair assists in identifying and recruiting panel members, developing the review protocol and literature search parameters, and overseeing the development of the guide.
peer review: A review process to gather external feedback from independent researchers on WWC reports before releasing them to the public.
population: The subset of students that is the focus of an intervention. Some topic reviews require students to have certain characteristics, such as English language learners or special education students.
positive effects: Positive: strong evidence that intervention had a positive effect on outcomes.

This occurs when two or more studies show statistically significant positive effects, at least one of which met WWC evidence standards for a strong design and no studies show statistically significant or substantively important negative effects.
potentially negative effects: Potentially Negative: evidence that intervention had a negative effect on outcomes with no overriding contrary evidence.

This occurs when a) one study shows a statistically significant or substantively important negative effect and no studies show a statistically significant or substantively important positive effect; or b) two or more studies show statistically significant or substantively important negative effects, at least one study shows a statistically significant or substantively important positive effect, and more studies show statistically significant or substantively important negative effects than show statistically significant or substantively important positive effects.
potentially positive effects: Potentially Positive: evidence that intervention had a positive effect on outcomes with no overriding contrary evidence.

This occurs when at least one study shows a statistically significant or substantively important positive effect, no studies show a statistically significant or substantively important negative effect, and fewer or the same number of studies show indeterminate effects than show statistically significant or substantively important positive effects.
practice guide: A WWC publication that presents recommendations for educators to address challenges in their classrooms and schools. They are based on reviews of research, the experiences of practitioners, and the expert opinions of a panel of nationally recognized experts.
principal investigator: An expert on a topic, with or without strong methodological skills. Principal investigators are responsible for leading reviews in topic areas, single study reviews, and quick reviews.
prioritization: The process by which interventions are selected for review.
program type: The way in which an intervention is used, as a curriculum, supplement, or practice. A curriculum defines what topics must be understood, and at what level, to achieve a particular grade or standard. A curriculum is typically something a school or district would purchase to implement in its schools and/or classrooms. A supplement is delivered in addition to the base curriculum. A practice is a lesson or repeated instruction intended to introduce, develop, or improve a skill. A practice is not a branded intervention or product, but rather a method, approach, or strategy for teaching.
properly aligned: Analysis is conducted at the same level as assignment. A mismatch problem occurs when random assignment is carried out at the cluster level and the analysis is conducted at the student level, but the correlation among students within the same clusters is ignored in the computing the standard errors of the impact estimates. See clustering adjustment for more details.

Top

Q

quality assurance: An internal review process for WWC products.
quality review team: An independent team that responds to concerns about WWC reports.
quasi-experimental design: A design in which groups are created through a process that is not random. For a quasi-experimental design to be rigorous, the intervention and comparison groups must be similar, demonstrating baseline equivalence on observed characteristics, before the intervention is started. Strong quasi-experimental designs will, at best, be rated as Meets WWC Evidence Standards with Reservations.
quick review: A timely and objective assessment of recent research. To be eligible for the quick review process, a study must: a) examine the effectiveness of an education or school-based program in improving student outcomes, b) be recently released to the public and reported in a major national news source, and c) use a design for which the WWC has developed evidence standards.

Top

R

randomized controlled trial: A design in which groups are created through a process that is random. Carried out correctly, random assignment results in groups that are similar on average in both observable and unobservable characteristics, and any differences in outcomes between the groups are due to the intervention alone. Strong randomized controlled trials can receive the highest rating of Meets WWC Evidence Standards without Reservations.
rating of effectiveness: In intervention reports, the WWC rates the effects of an intervention in each outcome domain. The rating of effectiveness takes into account four factors:
the quality of the research on the intervention
the statistical significance of the research findings
the size of the differences between participants in the intervention and comparison groups,
the consistency in findings across studies
Effectiveness Rating Key
Positive: strong evidence that intervention had a positive effect on outcomes.
Potentially Positive: evidence that intervention had a positive effect on outcomes with no overriding contrary evidence.
Mixed: evidence that intervention’s effect on outcomes is inconsistent.
No Discernible: no evidence that intervention had an effect on outcomes.
Potentially Negative: evidence that intervention had a negative effect on outcomes with no overriding contrary evidence.
Negative: strong evidence that intervention had a negative effect on outcomes.
recommendation: Practical, evidence-based guidance for teachers and administrators on how to address a current challenge in education. Practice guide recommendations reflect the findings of research on the effectiveness of educational practices, as well as the experience and judgment of the practice guide panel. Each recommendation is a distinct strategy that the panel suggests schools or districts implement to address an educational challenge. The panel assigns each recommendation a level of evidence rating that reflects the evidence base that supports the recommendation.
registries: Online databases of researchers and randomized controlled trials (RCTs). The registry of evaluation research is an online database of researchers who conduct evaluations of the effectiveness of educational interventions. This resource is designed to help schools, districts, and educational program developers identify potential researchers to conduct studies of effectiveness of educational interventions. The RCT registry is an online database of completed and in-progress RCTs in education. This resource is designed to help schools, districts, and educational program developers identify research regarding the effectiveness of educational interventions. The information for both registries is supplied by the evaluators. The WWC does not verify the accuracy of the information submitted, nor does it assess their qualifications or endorse any individuals or organizations listed.
regression discontinuity design: A design in which groups are created using a continuous scoring rule. For example, students may be assigned to a summer school program if they score below a preset point on a standardized test, or schools may be awarded a grant based on their score on an application. A regression line or curve is estimated for the intervention group and similarly for the comparison group, and an effect occurs if there is a discontinuity in the two regression lines at the cutoff.
review protocol: A formal document developed to determine the scope of a review. Protocols include key definitions, express the types of research studies and interventions reviewed, set reviewed parameters, and explain methodology.

Effectiveness Rating Key
	Positive: strong evidence that intervention had a positive effect on outcomes.
	Potentially Positive: evidence that intervention had a positive effect on outcomes with no overriding contrary evidence.
	Mixed: evidence that intervention’s effect on outcomes is inconsistent.
	No Discernible: no evidence that intervention had an effect on outcomes.
	Potentially Negative: evidence that intervention had a negative effect on outcomes with no overriding contrary evidence.
	Negative: strong evidence that intervention had a negative effect on outcomes.

Top

S

single study review: A review of an individual study that includes the WWC’s assessment of the quality of the research design and technical details about the study’s design and findings.
single-case design: A design in which an outcome variable is measured repeatedly within and across different conditions that are defined by the presence or absence of an intervention. This design involves repeated, systematic measurement of a dependent variable before, during, and after the active manipulation of an independent variable. It is identified by the following features: an individual case is the unit of intervention administration and data analysis, the case provides its own control for purposes of comparison, and the outcome variable is measured repeatedly within and across different conditions or levels of the independent variable.
standard deviation: A measure of variation across observations in a sample. A low standard deviation indicates that the observations in the sample tend to be very close to the mean. A high standard deviation indicates that the observations in the sample tend to be spread out over a large range of values.
statistical adjustment: Including baseline measures in a statistical model at the level of the unit of analysis. A number of different techniques can satisfy the statistical adjustment requirement, including regression adjustment, ANCOVA, and hierarchical linear modeling. Randomized controlled trials with high attrition and quasi-experimental designs require statistical adjustment to control for differences in baseline characteristics if the effect size of the difference in baseline characteristics is between 0.05 and 0.25 standard deviations.
statistical significance: The likelihood that a finding is due to chance rather than a real difference. The WWC labels a finding statistically significant if the likelihood that the difference is due to chance is less than five percent (p = 0.05).
statistical, technical, and analysis team: Highly experienced researchers who revise and develop the WWC standards.
strong design: A design that receives the highest rating of Meets WWC Evidence Standards without Reservations.
strong evidence: Consistent evidence supporting a practice guide recommendation. This rating is assigned when the panel finds high quality causal research that links a practice with positive results, ruling out other causes of the positive results.
study: An examination of the effect of an intervention on a particular sample and set of outcomes. In most cases, the results of a study are presented in a single article or manuscript. A study may also present results over multiple manuscripts, such as an examination of a beginning reading program that analyzes both immediate and long-term effects of an intervention on a common sample that is reported on in manuscripts produced over several years. In contrast, a single manuscript may contain multiple studies, such as an examination of a dropout prevention program analyzed separately in three different cities.
study rating: The result of the WWC assessment of a study. The rating is based on the strength of the evidence of the effectiveness of the educational intervention. Studies are given a rating of Meets Evidence Standards without Reservations, Meets Evidence Standards with Reservations, and Does Not Meet Evidence Standards.
study review guide: A spreadsheet used as a tool for certified reviewers to assess a study.
substantively important: An effect size of 0.25 or greater, regardless of statistical significance.
systematic review process: Explicit methods used to identify, select, appraise, and extract data from research. A WWC systematic review has five steps: 1) developing a review protocol; 2) searching the literature; 3) reviewing studies, including screening studies for eligibility, reviewing the methodological quality of each study, and reporting on high quality studies and their findings; 4) combining findings within and across studies; and 5) summarizing the review.

Top

T

topic: An outcome, population, or type of intervention studied by the WWC. Topics are defined by intended outcome (such as improving reading skills), intended population (such as English language learners), and types of interventions (such as education technology) that may improve outcomes for students.

Top

U

unit of analysis: The level at which an analysis is conducted. For example, a study that looks at student outcomes will likely conduct the analysis using student-level data.
unit of assignment: The level at which group assignment is conducted. For example, a study may have random assignment conducted at the school level, though analysis may be at the school or student level.

Top

W

webinar: A presentation that may be attended remotely via the web.