SoCap09: Metric-tastic

My final post from (10000km and a two weeks after) SoCap.

Sorry this is late– I’ve just been indundated by the types of things that inundate your life at the end of five months on the road: I hadn’t slept in the same place (and mostly in the same country) for more than three weeks between April and September. A lot of laundry, email management, and general decompressing time needed to happen.

Metrics
MBA-ism is filled with metrics. In fact, there is some research on the impact of metric-heavy MBA thinking on entrepreneurship, and the basic idea is that the two schools of thought are are diametrically opposed.

Dr. Saras Sarasvathy from the Darden School of Business posts her findings at effectuation.org, which broadly suggest that for a certain type of entrepreneur, the whole idea of finding markets, estimating market sizes, and attempting to essentially pre-plan the value of a business is opposite to disruptive models of innovation where marekts are created– i.e., no MBA would have invested in an ice machine because the value of the harvested ice market was too big.

Metrics, Development-style
Monitoring and Evaluation (M&E) is a tricky, hard thing. It’s a nightmare for lots of international NGOs as the data collection is done from far away, sometimes by people who don’t understand the value of the information they are gathering (Appfrica may be on to something with their new approach to this…).

Now, add to the difficulty of collecting metrics the ability to compare several projects:

– A clean water project
– A new ICT training centre
– A new delivery system for vaccines
– A new collective agriculture scheme

All of these may cost the same, and have variable outputs– some will have smaller effects on more people, others might be drastically transformational for the lives of a very few. How do you compare them? How you choose between them? How do you decide which was more successful after they’ve run?

Measurement metrics
There are a few different approaches to this, many of the latest (ish) are in things like Blended Value and SROI. These attempt to monetise the outputs, so if you increase someone’s productivity (training), then their increased productivity can be measured, turned into dollars, and compared against the increased sales prices that the agriculture scheme generates for its members, and then you report them along with your financial statements. You can add them so that you achieve “profit” in “social returns”. It’s crude but it’s something.

(if you have any better ideas, then by all means suggest away– I don’t think anyone’s particularly tied to these, although they do yield some kind of apples-to-apples comparisons, although the methodologies for each monetisation vary so they end up being apples-to-pears all to often…)

Operational metrics
These are, generally, Key Performance Indicators (KPIs). These should be the sorts of things that tell you what’s going on while you’re working– cost per vaccine delivery, cost per avoided pregnancy, cost per trained person. This lets you know if you’re on- or off-target, and then you can start to adjust your thinking and approach based on what’s happening in the field.

My question
The question I had with all of this was “What does it all tell you?”

Here’s the thing: Saying you trained 100 people to use the Internet is almost meaningless when you’re 5000 miles away. Saying that you created $50,000 worth of increased productivity is even worse. What you’re trying to get to is a deeper understanding about the effect thatyouvé had on people’s lives.

You want to get to people’s stories. You need the context to know exactly what it means to increase someone’s effective wage $100 a month.

There’s an important thing happening at and around SoCap about creating better, more comparable metrics, but it’s important to not just be stuck in the Excel sheet, but to also communicate what it is that you’re trying to change. There are too many “successful” projects on the ground that achieved exactly what they set out to on the Excel sheet, but didn’t create lasting change.