Running a Randomised Control Trial of Our Programmes – Balloon Ventures

Running a Randomised Control Trial of our Programme

If you’re unfamiliar with randomised control trials, you can find lots of useful information here about what and RCT is and how they are often used in development.

What will the Balloon RCT look like?

In our case, we have two groups created through random allocation: the group which receives the Balloon programme (intervention group) and a group which is being waitlisted to receive the programme i.e. is getting no intervention for now (control group).

The goal of the RCT is to know whether the Balloon programme causes entrepreneurs to grow their businesses. We will therefore measure business performance (income, profit, customers etc.) across the study.

We are working with academics at Stanford University who are leading on the research design. The evaluation will take place in Kenya and Uganda from 2017 to 2020. The evaluated programmes will run in Kenya and Uganda in 2017 and 2018, and the end of evaluation data collection will happen in 2020

Why run an RCT?

There are lots of types of evaluation but RCTs are considered the gold standard of evidence. In other words, they are the best way the international development community has come up with to evaluate impact.

At Balloon, we’re serious about impact, our paramount priority is doing work that we have unshakeable confidence in. That way, when we make promises to funders, beneficiaries and other stakeholders about our impact, we are making genuine promises. Otherwise, we run the risk of thinking we create impact (and telling people we do) when our actual impact is limited. So it makes sense for us to run an RCT.

Running an RCT also sends a statement that we are confident in our programming. There’s no hiding from the results (our academic colleagues will publish no matter what the outcome!). So, we would only be doing this if we were pretty sure that our programmes create impact.

Usually, massive organisations run RCTs. How comes Balloon is running one?

There’s a few reasons why RCTs are usually run by larger organisations. First, for the stats to work, the number of beneficiaries going through the RCT (sample size) needs to be quite large, and there are few small organisations that run programmes with enough beneficiaries to merit doing an RCT. Secondly, you need a good evaluation/research team to be able to implement the RCT, there’s a lot to manage and control. Smaller organisations are less likely to have this capacity. Finally, (linked to the first two) the cost is normally very high. Running programmes at scale, taking on additional capacity, investing in rigorous data collection (external evaluators, training, etc.), keeping track of people over a year or more all costs a lot of money.

For Balloon, all of these three are relevant but we’ve managed to find ways to overcome the biggest challenges. On sample size, ~90% of Balloon’s work is comparable in that it’s the same programme delivered to beneficiaries, this allows us to be able to group all these individuals together and run an RCT on them. Regarding research team, we’re lucky to be able to partner with colleagues from Stanford University who have run several RCTs before. Our internal team also has experience running RCTs making it easier to deploy this one. Finally, on cost, we have approached the RCT using an entrepreneurial mind-set i.e. what is the leanest way we can do what we need to do. This has allowed us to strip out costs and still get what we need. At the same time, our colleagues from Stanford were able to secure funding to part-finance the RCT, making it easier for us to run this project.

How's it going so far?

So far so good! Half of the programmes are underway and the first is wrapping up now. We’ve learnt a lot already as we’re monitoring things on the RCT, that we never did before. For example, a new activity tracker shows us how our teams are working, with who, doing what and when. That’s all hugely interesting for understanding how we create impact.

We are eagerly counting down the days until the full results are in!