By Line Renate Hanssen, Johanne Fyhn, and Sanni N. Breining
Impact assessments are all the rage these days, seeking to answer questions about which interventions are most effective, e.g. in improving academic performance among bilingual children or generating employment among vulnerable adults.
However, identifying the impact can be a difficult task which requires a lot of experience and theoretical knowledge. Overall, two approaches can be considered: an experimental approach or a quasi-experimental approach.
The experimental approach in social science is often done using Randomised Control Trials (RTC) which is known for bringing solid evidence on the impact. But sometimes randomization is not possible (e.g. because of costs) or desirable (perhaps because of ethics). Further, if the intervention already has been implemented and you need to detect results ex post an RCT is infeasible. Luckily, in these cases, quasi-experimental approaches can be pursued.
In this article, we will discuss which options are available to use when determining impact. We will further present a decision tree which should be helpful in deciding which option to pursue.
What is an experiment vis-á-vis a quasi-experiment?
In impact evaluation we are interested in establishing a causal relation between certain factors. A causal relation is when one factor explains or contribute to explain variation in another factor. When trying to estimate the impact of intervention, the crucial question is this; does the intervention explain changes in the outcome, e.g. academic level.
To be able to estimate the causal effect or impact of a program on outcomes, a so-called counterfactual must be developed. A counterfactual is what the outcome would have been for program participants, had they not participated in the program. The challenge in real-life evaluations is that you can never directly observe the counterfactual situation! Hence, one must infer the counterfactual situation, which in practice means making some assumptions. We will briefly touch upon some of the assumptions later in this article. For now, we will illustrate the difference between experimental and quasi-experimental methods.
The golden standard: Random Controlled Trials
In a true experiment, the participants are randomly divided into two groups; one group which continue as usual or receive a placebo-intervention (this is most common in medical experiments). The other group is exposed to the intervention, from which one expects to see an impact.
Constructing an experiment enables evaluators to compare the effectiveness of a new policy or programme with what would have happened if nothing had been changed. The random assignment into the groups eliminates (if the groups are large enough) observed and non-observed biases between the groups.
This type of experiment is known as a Randomized Controlled Trial and will results in a high level of evidence. Hence, one will often start to consider, if it is possible to apply an experimental approach. If this is not possible, a quasi-experimental approach can be applied, in which case one will inevitably have to compare and judge its merit to the experimental approach.
Click to enlarge
The second best: Quasi-experiments
As mentioned a quasi-experimental design is typically considered when an experiment is not desirable or doable, e.g. in the following situations:
A quasi-experiment is, as the name implies, a way to mimic a true experiment without being able to actually perform an experiment as an RCT. In a quasi-experiment one tries to estimate the causal effect of an intervention but does so without randomisation.
Instead of randomising the intervention to the target population one tries to assign the intervention using some criterion or statistical procedure. In a quasi-experiment it might be necessary to construct a comparison group from observable characteristic or to check for non-random assignments in statistical ways.
A quasi-experimental design can be made in two different time-perspectives:
- It is unethical or impracticable to keep a comparison group from receiving a treatment or intervention.
- The character/elements of the intervention changes content during the experiment period.
- The experimental conditions of the intervention differ significantly from how the intervention would be under "normal conditions“.
- An RCT is assessed to be either too time consuming or too costly.
- Threats like attrition or spill over are too difficult to control, thus impairing the experiment’s validity.
- The sample size (number of participants) is too small to identify any effect.
The big difference between the relevance of an RCT and a quasi-experiment, is that while an RCT only can be relevant in the case where the programme is implemented with a random assignment to treatment, then a quasi-experiment is also relevant in the case of non-random assignment to the programme or initiative. Examples include self-selection into the initiatives, assignment on convenience, administrator assignment etc. A quasi experiment is also relevant in the case of no direct control group. In a quasi-experiment a comparison group can be constructed with use of background data on the population.
A quasi-experiment often demands a high level of available data, as the evaluation only estimates the actual effect if all characteristics that might affect the impact of the intervention are accounted for. In particular, there needs to be data for eliminating differences between the intervention and control group in order to establish the counter-factual situation. As a result, a quasi-experiment is not relevant if there is no background data on the population. Background data is e.g. gender, age, socio-demographic variables as income, educational level, health, crime, etc. A quasi-experiment also requires a project-team with competences in statistical programmes and econometrical method of estimating effects.
Sometimes RCT’s are combined with quasi-experimental designs. This occurs in cases where there is doubt about the validity of the randomisation or if there are reasons to believe that the RCT is not successful (e.g. contamination of control groups). Lastly, RCT and quasi-experiments can be combined to qualify the results and rule out any administrative problems in the RCT.
- Ex ante design: Designing an experiment in the future as is also normally done in an RCT.
- Ex post design: Detecting an experiment in existing data.
Choosing the right impact assessment approach
Choosing the optimal method or mix of methods is sometimes straight forward and is at times very complex. When deciding which approach to apply there are a number of relevant questions you need to consider upfront. Most importantly:
The decision tree below is made to help you find the optimal method by answering the questions on the left. Please note that this model is intentionally reducing complexity meaning nuances are lost. However, the model still should provide input on important considerations to make when finding the right approach.
- Can a comparison group the established?
- Can you randomise to a treatment and non-treatment group before introducing the intervention?
- Can baseline data (a before measure or background data) be collected before the intervention?
Click to enlarge
Can a comparison group be established?
When conducting an impact assessment, you really want to have a comparison group which you can compare against when evaluating the interventions effectiveness (to estimate the counterfactual situation).
If this is not possible you will be left with approaches which are neither experimental nor quasi-experimental.
If you do not have a baseline measurement, you can apply an after approach where you simply measure the relevant outcomes after the intervention period and merely investigate this metric. This means that there is no comparison with prior result (before measure) or with the results of a comparison group. Without a before measure you cannot know if there has been a progression in the outcome of interest.
If you do have a baseline, you can choose to make a before-after comparison of the group receiving the intervention. In a before-after design you supplement the after measure with a before measure and investigate the average difference in results before and after the intervention - i.e. you look for a progression. However, because there is no comparison group, it is not possible to assess, if a given development would have occurred even in the absence of the intervention. To be able to make a valid assessment of impact you must be able to assume that there are no other factors but the intervention influencing a given result.
Can you randomise to a treatment and non-treatment group before introducing the intervention?
If you cannot randomize to a treatment and non-treatment group before introducing the intervention an RCT is ruled out.
If you can include a comparison group but do not have a baseline measure you can use the simple difference approach. In the simple difference approach a comparison group is included, but it is not necessarily coincidental, who receives an intervention. Hence, you cannot know whether there were initial differences between the groups, which have caused possible differences in the results.
If you can construct a comparison group and have background information you can choose between different quasi-experimental approaches instead which are based on different assumptions and have different advantages and disadvantages (see below).
Click to enlarge
If you are considering applying the quasi-experimental designs, the following considerations can be helpful to narrow in on the optimal method:
- Matching: Mostly used in ex-post evaluation. Has advantages when the intervention group is smaller than the comparison group. Preferred in combination with diff-in-diff or regression analyses.
Remember that statistical methods of quasi-experimental designs can often be combined with each other or with non-experimental methods.
- Regression: Can be used in ex-ante and ex-post evaluations. Requires extensive background data.
- Fixed effect panel data:Only ex-post evaluations. Requires specific data-design: Either time-series data over a time period or cross-sections data, which is data with repeated observations.
- Diff-in-diff regression: Is a commonly used method with a very high level of evidence. It can be used in ex-ante and ex-post evaluations. Requires measurements of comparison- and treatment group before and after and preferable a high level of background data as well.
- Regression discontinuity: Is mainly used in ex-post evaluations. Requires an exogenous threshold in the data as well as background data.
Wrapping up, we are able to conclude: An RCT is not always the right choice or even a possible choice when setting out to determine the impact of a given intervention or policy. However, a number of other approaches are very feasible alternatives if you are aware of their assumptions and preconditions.
Quasi-experiments can – as opposed to an RCT - be made ex ante as well as ex post. Further, a quasi-experiment – opposed to RCT - is also relevant in the case of non-random assignment to the programme or initiative and equally so in the case of no control group. Instead, a comparison group can be constructed with use of background data on the population.
The above solely focuses on the aspect of identifying causal effects. In Ramboll Management Consulting, we seldom focus on effects only but often combine this approach with more theory-based evaluation where we investigate how and why effects arise. This often means including qualitative data collection and analysis such as in-depth case studies and surveys.
If you want to learn more
If you want to read more on the specific approaches we recommend the following resources from the World Bank which also can be found online:
Further, Better Evaluation has a very nice introduction to the topic.
In Ramboll, we have extensive experience with both experimental and quasi experimental impact assessments depending on the demands of the clients and data availability and timing.
Please contact us if you are interested in learning more about experimental or quasi experimental impact evaluations. You'll find contact infomation in the right column.