Customize Consent Preferences

We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below.

The cookies that are categorized as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... 

Always Active

Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

No cookies to display.

Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

No cookies to display.

Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

No cookies to display.

Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.

No cookies to display.

Advertisement cookies are used to provide visitors with customized advertisements based on the pages you visited previously and to analyze the effectiveness of the ad campaigns.

No cookies to display.

Technopolis Group

The term “impact evaluation” means different things to different people, but a helpful definition is the systematic analysis and estimation of causal effects of policy interventions and other events. In short, thoroughly trying to understand whether an intervention works or not. The fundamental problem with causal inference however, is missing data – for each person, district, municipality, region, country or subset of entrepreneurs/researchers/etc. we can only observe at most one potential outcome. Not only does that pose issues for understanding the effect size of any intervention, but more importantly whether the outcome would perhaps have been better without the intervention.

The answer to this dilemma has been experimental research methods, which have become mainstream across many disciplines in the social and behavioural sciences after their widespread and longstanding use in medicne. A randomized controlled trial (or randomized control trial, RCT) is a type of scientific experiment or intervention study (as opposed to observational study) that aims to reduce certain sources of bias when testing the effectiveness of new treatments. This is accomplished by randomly allocating subjects to two or more groups, treating them differently, and then comparing them with respect to a measured response.

Fervent fans of experimental methods in social sciences, dubbed ‘randomistas’, were given a boost in 2019 after three pioneers of randomised experiments in international development – Esther Duflo, Abhijit Banerjee and Michael Kremer – won the Nobel Memorial Prize in Economics. Their organisation, the Abdul Latif Jameel Poverty Action Lab (or JPAL), has run over 1,000 randomised controlled trials to understand how to reduce poverty and has championed the use of the method internationally.

However, with some exceptions, other areas of empirical social science lag behind in terms of acceptance and adoption of these experimental methods. Some of this lag is to do with a misunderstanding of experimental methods and their application. At the same time, as with any new ‘standard’ in scientific methodology, with its growth in use and authority so grew its criticism. Below, a few pro’s and con’s are listed regarding the use of RCTs.

Three benefits:

  1. RCTs are currently the optimal and least-biased method for estimating, on average, whether something works, when done well. Those last three words are important. Just because a study is an RCT, does not automatically mean it is ‘gold standard’. In actual fact, there is a continuum from the truly outstanding to the totally rubbish.
  2. When combined with information on cost and implementation, RCTs provide very powerful information for decision makers (in our case this often includes programme managers in the educational sector and government officials) on the ‘best bets’ for spending their budgets. However, without implementation and theory, RCTs are a ‘black box’, meaning it can be difficult to interpret and decide how to act on the result.
  3. Crucially, experimentation means being transparent also about what does not work. There is no avoiding the challenges this brings; and it may mean taking a more honest approach: recognising in public that you do not have all the answers and need to test out your ideas. Though this work in progress, with a trend towards more open science, the publication and sharing of negative results is further encouraged by the set-up of RCTs.

Three challenges:

  1. RCTs are not suited to answering all kinds of questions, and there are some things to which subjects are just not willing to be randomised. Questions interesting to answer through RCTs but with low success odds include mixed attainment grouping v setting in schools (too ideological), financial incentives for teachers (too controversial), and changing the secondary school start times to later in the day to accommodate sleepy teenagers (too impractical). For these kinds of questions, we need alternative designs.
  2. Sometimes the answer depends. RCTs tell you what works on average, but how is one school supposed to know how applicable an estimate is to them? I have learned that we need hundreds of schools even to estimate what works on average, so imagine how many we would need to be able to tell what works for different types of schools and pupils? But data archiving and linkage has great potential for understanding variation in outcomes..
  3. Decision makers want answers now, but RCTs take time to plan and deliver well. Without excellent planning and communication throughout, RCTs just won’t work. There is no easy solution to this. However, it does mean that it is essential to persuade and demonstrate to decision-makers of the value of RCTs, and to make sure the RCTs we design now are still relevant when the results come out.

There are some measures that can be taken to be mitigate the above listed challenges. A few ideas are to make sure to communicate the benefits well to participants, so they do not drop out, collect high-quality data on cost and implementation and think about context and timing carefully, to ensure relevance at the end.

At the same time the field is not stagnant, the use of experiments to guide effective social action is being championed, and reinvented constantly. It’s not only the classic social scientific experiment that has entered the toolbox of the policymaker. Prototyping is an approach traditionally used by engineers, designers and web developers, and is now part of the landscape of policy and service design and decision making. Another set of approaches central to learning about policy are quasi-experimental designs (QEDs), sometimes called ‘queasy’ experiments – after the sense of unease they can instil in purist researchers who prefer randomised designs. These methods have deep roots in social science and statistics, providing ways to test and learn about policy ideas that are already ongoing or in areas of decision-making where randomised approaches may not be possible. These type of designs also allow us to exploit unplanned opportunities for learning about social change, in what are termed ‘natural experiments’. These are tests that explore a policy or practice change resulting from a ‘natural’ divergence or external event.

No approach is perfect, and no design offers easy answers. Many innovations fail to make a difference; making better policy takes hard work and investment. Most projects evaluated by an RCT show little impact. In education, 90 per cent of interventions evaluated in RCTs by the US Institute for Education Sciences had weak or no positive effects. This lack of progress should not, however, depress us. It is a reminder of how difficult it is to make a difference (and how unwise it is to assume that just because an innovation is new and sounds attractive, that it will improve outcomes in practice).

Technopolis in the spotlight

All articles All news