Measuring Changes in Financial Understanding in Balloon Entrepreneurs

At Balloon, our mission is to “Build businesses, change lives”, and we believe that a critical part of achieving this is developing the skills and knowledge of entrepreneurs who take part in our programme. As a result, we created a curriculum based on sales/marketing, strategy and finance. Finance, in particular, seems to be critical. Understanding finance is one of the main reasons why local entrepreneurs choose to embark on the Balloon programme. Almost 50% of them rank it as the first or second most important reason for joining the programme.

To track how well we are doing against this goal, we’ve been tinkering with our monitoring and evaluation surveys. Our initial methodology was to ask entrepreneurs to rate themselves at the beginning and at the end of the programme on financial understanding. We take two measurements to see the change in score across the programme and 70% of them said it was one of the areas where they improved the most. Self-report is one way to measure impact, but there are several biases with this method affecting how much we can trust these results. For example, an overly-optimistic outlook could affect their self-ratings.

We therefore decided to create a more objective assessment. This is a test of financial understanding focussing on the profit equation, record keeping and financial decision making (core parts of our curriculum). We call it the Entrepreneur Financial Knowledge Assessment (EFKA) and every entrepreneur completes this at the beginning and at the end of the programme (the tests are slightly different but measure the same thing).

Our analysis of the EFKA data across programmes run in Kenya in the first quarter of 2017 yielded interesting results. On average entrepreneurs increase their score by 1.31 points, moving from a general average of 14.1/20 to 15.4. This result is statistically significant[1], which means that the probability that this outcome is purely due to random factors is extremely low. Furthermore, not only do entrepreneurs improve their score, but they are also quicker in doing the test. As we use categories to measure time (e.g. 10-15 minutes, 15-20 minutes…) it is hard to get a precise estimate of how much quicker they are, but they move from a rough median completion time of 20 minutes to around 15 minutes.

To explore further, we analysed whether the improvement was similar across locations. This was the result (locations kept anonymous):

LOCATIONAVERAGE SCORE CHANGE[2]
Location 1+3.1
Location 2+2.3
Location 3+1.4
Location 4~0
Location 5~0

As you can see, in some locations, entrepreneurs improve much more than others (it’s also encouraging that no location sees a decrease in score!). As entrepreneurs are trained in finance by volunteers (who are in turn trained by us), we thought location differences might be down to the teams in each location.

To assess if this hypothesis might be valid, we asked our ICS Programme Manager which teams they thought were the strongest among these, without sharing the EFKA results. They replied: “Definitely location 1 and 2, and maybe location 3 and then the rest”. This is remarkably similar to the ranking of the change in EFKA scores. While this evidence is not enough to draw definitive conclusions, the matching of our manager’s impression and our data suggests that teams can have an impact on the degree of change in entrepreneurs’ financial understanding.

In summary, we have strong self-report and objective data to suggest that entrepreneurs improve their financial understanding on a Balloon ICS programme. Interestingly, there is some evidence to suggest that this is influenced by the nature of the team in each location. Moving forward, we will monitor this in more detail and adjust how we deliver our training based on any insights that emerge.

 

[1] The t-test with H₀: test1=test2 has p-value of 0.016.

[2] 95% Confidence intervals for these values are, respectively:  1/5.2 ; 0.7/3.9 ; 0.3/2.6 ; -1.8/+1.7 ; -1.4/+1.2.

Leave a Reply

Your email address will not be published. Required fields are marked *