I attended the The Muskoka Initiative Consortium Results Symposium in Toronto on 23 and 24 June 2015. A number of NGOs and university partners that have received funding from the Canadian government to do MNCH (maternal, newborn and child health) programming in Africa gathered to share experiences, results and lessons learned.

In one of the first talks, Stan Zlotkin from University of Toronto told this story, as an analogy of the state of the data that we have available for program monitoring. He said imagine that the Federal Minister of Agriculture of Canada and the Deputy Minister were discussing the impact of a government program to support agricultural production in Canada. The conversation may have gone like this (as best as I remember Stan’s words):

Minister: So tell me about the impact of our support program on agricultural production in Canada!
Deputy Minister: Oh well it is very complicated. We supported fruit and grains and livestock…

Minister: Ok, tell me about the impact of our support program on fruit production!
Deputy Minister: Well, the results were different for peaches and apples and cherries.

Minister: Fine, fine. Just tell me the impact for something! Tell me about apples.
Deputy Minister: Well, there are apples grown in BC, and Ontario and Nova Scotia, and it is a different story in each province.

Minister: OK, then just tell me about apples in Ontario.
Deputy Minister: Well it is complicated. There are Mcintosh apples, and Granny Smith and…

Minister: Tell me about the McIntosh!!
Deputy Minister: Well, in the Niagara region it was a very dry year, but it was very wet in Collingwood, so it is a very complex story.

Minister: Aaagh. What I really need to know: was our money well spent?
Deputy Minister: Oh yes, the money was definitely well spent!

Stan’s point was that we wouldn’t expect such a scenario in Canada, and yet that is directly analogous to what often happens in the monitoring of Canadian funded health programming in low income countries.

I had a chance to speak at the conference about some of the work I have done with Care Canada in evaluating their monitoring data from an MNCH program in Malawi. I used a variation of Stan’s story as a starting point to explain what my experience was like working with the CARE monitoring data. Frankly, it was similar to experiences I’ve had working with data from various NGO health interventions over the last few years.

Minister: What can you tell me about the impact of our fruit program?
Deputy Minister: Oh, we have lots of data about our fruit program!

Minister: Great! What can you tell me about apples?
Deputy Minister: Well… we aren’t quite sure whether this data set is for apples or peaches. But the units reported for production are in “bushels”, and we know that apple production is measured in bushels, so we are confident the data are for apples.

Minister: OK, how was the apple production in British Columbia?
Deputy Minister: Well, the code for “province” was lost. Province was coded as 1, 2 or 3, and we aren’t sure if “1” is for BC or Ontario. However there is another question in the data set about migrant worker health conditions and some of the answers are given in Spanish. We know that there are Mexican migrant workers in BC, so we think these data are for BC.

Minister: OK, then how is apple production now in British Columbia?
Deputy Minister: Well… we aren’t sure when the data were collected. The date variable was corrupted and so the data were collected on either December 1st 2015 or 15th of January 2012…

I was later asked if that was really a fair analogy. Surely, I must have been exaggerating?? Unfortunately, no. We at HealthBridge are often asked to help other organizations make sense of their data and it is always the case that there is something wrong with the data. Usually there is something recoverable about the data. In a recent example from Honduras, the child health data were unusable, but the maternal health data were in good shape. In another case from Malawi the 2013 data were unusable, but much of the 2012, 2014 and 2015 were ok. There is so much wasted effort in collecting data, but not managing the data appropriately so that it can be usefully analysed.

I summarized the problems and suggested ways to address it in the following slide. My talk, and all the talks from the conference, will soon be online and I will post the link so you can hear the talk if it is of interest to you.