Determining impacts of interventions

Urban Life 23 February 2018 Line Renate Hanssen Johanne Fyhn Sanni N. Breining

Impact assessments are all the rage these days, seeking to know which interventions are most effective. Overall, two approaches can be considered. This expert article unfolds the methodological pros and cons.

Expert columns
16 mins

Written in collaboration with Johanne Fyhn, and Sanni N. Breining

Impact assessments are all the rage these days, seeking to answer questions about which interventions are most effective, e.g. in improving academic performance among bilingual children or generating employment among vulnerable adults.

However, identifying the impact can be a difficult task which requires a lot of experience and theoretical knowledge. Overall, two approaches can be considered: an experimental approach or a quasi-experimental approach. 

The experimental approach in social science is often done using Randomised Control Trials (RTC) which is known for bringing solid evidence on the impact. But sometimes randomization is not possible (e.g. because of costs) or desirable (perhaps because of ethics). Further, if the intervention already has been implemented and you need to detect results ex post an RCT is infeasible. Luckily, in these cases, quasi-experimental approaches can be pursued.

In this article, we will discuss which options are available to use when determining impact. We will further present a decision tree which should be helpful in deciding which option to pursue.  

What is an experiment vis-á-vis a quasi-experiment?

In impact evaluation we are interested in establishing a causal relation between certain factors. A causal relation is when one factor explains or contribute to explain variation in another factor. When trying to estimate the impact of intervention, the crucial question is this; does the intervention explain changes in the outcome, e.g. academic level.

To be able to estimate the causal effect or impact of a program on outcomes, a so-called counterfactual must be developed. A counterfactual is what the outcome would have been for program participants, had they not participated in the program. The challenge in real-life evaluations is that you can never directly observe the counterfactual situation! Hence, one must infer the counterfactual situation, which in practice means making some assumptions. We will briefly touch upon some of the assumptions later in this article. For now, we will illustrate the difference between experimental and quasi-experimental methods. 

The golden standard: Random Controlled Trials

In a true experiment, the participants are randomly divided into two groups; one group which continue as usual or receive a placebo-intervention (this is most common in medical experiments). The other group is exposed to the intervention, from which one expects to see an impact.

Constructing an experiment enables evaluators to compare the effectiveness of a new policy or programme with what would have happened if nothing had been changed. The random assignment into the groups eliminates (if the groups are large enough) observed and non-observed biases between the groups.

This type of experiment is known as a Randomized Controlled Trial and will results in a high level of evidence. Hence, one will often start to consider, if it is possible to apply an experimental approach. If this is not possible, a quasi-experimental approach can be applied, in which case one will inevitably have to compare and judge its merit to the experimental approach.

The second best: Quasi-experiments

As mentioned a quasi-experimental design is typically considered when an experiment is not desirable or doable, e.g. in the following situations: 

  • It is unethical or impracticable to keep a comparison group from receiving a treatment or intervention.
  • The character/elements of the intervention changes content during the experiment period.
  • The experimental conditions of the intervention differ significantly from how the intervention would be under "normal conditions“.
  • An RCT is assessed to be either too time consuming or too costly.
  • Threats like attrition or spill over are too difficult to control, thus impairing the experiment’s validity.
  • The sample size (number of participants) is too small to identify any effect.

A quasi-experiment is, as the name implies, a way to mimic a true experiment without being able to actually perform an experiment as an RCT. In a quasi-experiment one tries to estimate the causal effect of an intervention but does so without randomisation.

Instead of randomising the intervention to the target population one tries to assign the intervention using some criterion or statistical procedure. In a quasi-experiment it might be necessary to construct a comparison group from observable characteristic or to check for non-random assignments in statistical ways.

A quasi-experimental design can be made in two different time-perspectives: 

  • Ex ante design: Designing an experiment in the future as is also normally done in an RCT.
  • Ex post design: Detecting an experiment in existing data.

The big difference between the relevance of an RCT and a quasi-experiment, is that while an RCT only can be relevant in the case where the programme is implemented with a random assignment to treatment, then a quasi-experiment is also relevant in the case of non-random assignment to the programme or initiative. Examples include self-selection into the initiatives, assignment on convenience, administrator assignment etc. A quasi experiment is also relevant in the case of no direct control group. In a quasi-experiment a comparison group can be constructed with use of background data on the population.

A quasi-experiment often demands a high level of available data, as the evaluation only estimates the actual effect if all characteristics that might affect the impact of the intervention are accounted for. In particular, there needs to be data for eliminating differences between the intervention and control group in order to establish the counter-factual situation. As a result, a quasi-experiment is not relevant if there is no background data on the population. Background data is e.g. gender, age, socio-demographic variables as income, educational level, health, crime, etc. A quasi-experiment also requires a project-team with competences in statistical programmes and econometrical method of estimating effects.

Sometimes RCT’s are combined with quasi-experimental designs. This occurs in cases where there is doubt about the validity of the randomisation or if there are reasons to believe that the RCT is not successful (e.g. contamination of control groups). Lastly, RCT and quasi-experiments can be combined to qualify the results and rule out any administrative problems in the RCT.

Choosing the right impact assessment approach 

Choosing the optimal method or mix of methods is sometimes straight forward and is at times very complex. When deciding which approach to apply there are a number of relevant questions you need to consider upfront. Most importantly: 

  1. Can a comparison group the established?
  2. Can you randomise to a treatment and non-treatment group before introducing the intervention?
  3. Can baseline data (a before measure or background data) be collected before the intervention?

The decision tree below is made to help you find the optimal method by answering the questions on the left. Please note that this model is intentionally reducing complexity meaning nuances are lost. However, the model still should provide input on important considerations to make when finding the right approach. 

Can a comparison group be established? 

When conducting an impact assessment, you really want to have a comparison group which you can compare against when evaluating the interventions effectiveness (to estimate the counterfactual situation). 

If this is not possible you will be left with approaches which are neither experimental nor quasi-experimental.   

If you do not have a baseline measurement, you can apply an after approach where you simply measure the relevant outcomes after the intervention period and merely investigate this metric. This means that there is no comparison with prior result (before measure) or with the results of a comparison group. Without a before measure you cannot know if there has been a progression in the outcome of interest.  

If you do have a baseline, you can choose to make a before-after comparison of the group receiving the intervention.  In a before-after design you supplement the after measure with a before measure and investigate the average difference in results before and after the intervention - i.e. you look for a progression. However, because there is no comparison group, it is not possible to assess, if a given development would have occurred even in the absence of the intervention. To be able to make a valid assessment of impact you must be able to assume that there are no other factors but the intervention influencing a given result. 

Can you randomise to a treatment and non-treatment group before introducing the intervention? 

If you cannot randomize to a treatment and non-treatment group before introducing the intervention an RCT is ruled out. 

If you can include a comparison group but do not have a baseline measure you can use the simple difference approach. In the simple difference approach a comparison group is included, but it is not necessarily coincidental, who receives an intervention. Hence, you cannot know whether there were initial differences between the groups, which have caused possible differences in the results. 

If you can construct a comparison group and have background information you can choose between different quasi-experimental approaches instead which are based on different assumptions and have different advantages and disadvantages.