Life’s a Pitch (Part 2) – Looking for bias in our pitching panels

In the first part of this blog post, we looked at the evaluations made by Pitching Panels on the Balloon ICS programme. These panels, made up of experts, former volunteers, Balloon Team Members and members of the local community, assess the pitches made by entrepreneurs who have been through the Balloon programme, to ascertain their suitability to receive a low-interest loan to help them achieve their business goals.

In this week’s blog, we take another look at some of these data, and try to answer the question:

Can we see any hint of bias in the pitching panels' decision making?

Allocating loans is an important part of the Balloon process and highly valuable to entrepreneurs. It is important that there are no biases in making this decision. We therefore wanted to use the data available to us to see whether there was any discernible bias at play.

To do this, we analysed the relationship between variables that could potentially trigger biases (age, gender, level of English, and whether they are start-up or have an existing business) and the evaluations provided by our panel[1]. There were no statistically significant relationships between overall scores and age, gender, and whether they are a start-up or have an existing business. This means that, for example, being male or female does not appear to influence the score that you receive from the panel. This would indicate  no bias based on these variables.

However, there was a relationship between level of English and pitch score – the higher you were rated in terms of English language ability, the higher the pitch score you tended to receive from the panel.

This is an important finding. There are two hypotheses that could explain what’s going on:

  1. Evaluators are confusing entrepreneurial potential with English language ability. In other words, you give someone a higher score for being able to articulate their idea well, rather than considering the quality of the idea itself;
  2. English language ability is a proxy for things that actually make you a better entrepreneur. For example, if you speak better English, you are more likely to have attended university. Attending university also makes you more likely to run a successful business[2].

Now, if hypothesis 1 turned out to be true, we would be very concerned as this would be an example of biased decision making. So, we set out to find out more. What we discovered is quite interesting.

When we added variables representing ratings of the businesses’ potential to the analytical model, the relationship between English and total ratings of the entrepreneur disappeared[3]. Therefore, English language wasn’t actually linked to overall ratings, it was more the factors related to business potential. This suggests to us that hypothesis two is valid. In other words, an entrepreneur’s ability to speak English is a proxy for factors which might make you a better entrepreneur overall.

Although we do not currently know what these factors are because we did not collect data on these factors we have some ideas.

We can cluster our ideas into two groups:

  1. Pre-programme factors

These factors include anything that could be different between people of varying English ability at the point they join the programme. These could include: being more educated, having spent time abroad, having greater financial resources, etc. 

  1. On-programme factors

These are factors which affect how well our programme works. For example, despite providing support to entrepreneurs who may struggle with English, it could be that they do not get the same out of the programme compared to entrepreneurs with better levels of English. This would mean that their pitches at the end of the programme are weaker.

We are currently thinking about how we can better understand what is going on here. Once we have a more detailed picture we can think about how to better support everyone in the communities that we work in, and we’ll be sure to report back in a future blog post.

[1] We used multiple linear regression with two blocks to test this (block 1: demographics; block 2: pitch criteria). The outcome variable was the overall rating provided by our evaluators.

[2] This is a complicated relationship. It’s not only about education but for example, if you can attend university, you have access to money. If you have access to money, you are more likely to resist shocks that could potentially cripple your business (e.g. theft).

[3] I.e. the variance that English previously explained in overall rating was now better explained by the entrepreneur potential variables

Leave a Reply

Your email address will not be published. Required fields are marked *