Wednesday, December 5, 2012
"Monitoring and evaluation. It doesn't sound like a whole heap of fun..."
Today "TheGuardian" website published an article on monitoring and evaluation of development projects. The subtitle of the article is: "Providing evidence for funders might seem unimportant when you're saving lives, but it is vital to improving development work." Publication of this kind in such a well-known and popular media is an achievement in itself. It's even more pleasing that the article is well-written (this is my personal my opinion, indeed). The author (Louise Tickle) starts with the following statement: "Monitoring and evaluation. It doesn't sound like a whole heap of fun, or even particularly relevant when you're busy trying to turn lives around." Point is well taken.
Monday, December 3, 2012
CREAM or SMART or both?
Good goals should meet SMART criteria. They should be:
- Specific,
- Measurable,
- Attainable,
- Relevant,
- Time-bound.
Good indicators should meet CREAM criteria. They should be:
– Clear (precise and unambiguous)
– Relevant (appropriate to the subject at hand)
– Economic (available at reasonable cost)
– Adequate (able to provide sufficient basis to assess
performance)
– Monitorable (amenable to independent validation).
SMART criteria are very well known. CREAM criteria are becoming more and more popular. They were introduced by Kusek and Rist in their book "Ten Steps to a Results-Based Monitoring and Evaluation System"
One of the common issues we face while evaluating programs is: when people try really hard to develop SMART goals they tend to use indicators instead of goals. This mistake has a number of negative consequences. The best and most comprehensive evidence-based description of those consequences I am aware of was provided by Ordóñez, L. D., Schweitzer, M. E., Galinsky, A. D., & Bazerman, M. H. in their article Goals Gone Wild: The Systematic Side Effects of Over-Prescribing Goal Setting (2009). In particular, they argue that when people are supposed to achieve a goal that is formulated as an indicator they often demonstrate unethical behavior. For instance, they can fix cars that are not broken to generate expected income per person ("SMART" goal).
Interestingly, there are more and more publications that apply SMART criteria to indicators but not to goals. I think, this is a very good idea. Try to google "SMART indicators", and you will get over 25000 links.
- Specific,
- Measurable,
- Attainable,
- Relevant,
- Time-bound.
Good indicators should meet CREAM criteria. They should be:
– Clear (precise and unambiguous)
– Relevant (appropriate to the subject at hand)
– Economic (available at reasonable cost)
– Adequate (able to provide sufficient basis to assess
performance)
– Monitorable (amenable to independent validation).
SMART criteria are very well known. CREAM criteria are becoming more and more popular. They were introduced by Kusek and Rist in their book "Ten Steps to a Results-Based Monitoring and Evaluation System"
One of the common issues we face while evaluating programs is: when people try really hard to develop SMART goals they tend to use indicators instead of goals. This mistake has a number of negative consequences. The best and most comprehensive evidence-based description of those consequences I am aware of was provided by Ordóñez, L. D., Schweitzer, M. E., Galinsky, A. D., & Bazerman, M. H. in their article Goals Gone Wild: The Systematic Side Effects of Over-Prescribing Goal Setting (2009). In particular, they argue that when people are supposed to achieve a goal that is formulated as an indicator they often demonstrate unethical behavior. For instance, they can fix cars that are not broken to generate expected income per person ("SMART" goal).
Interestingly, there are more and more publications that apply SMART criteria to indicators but not to goals. I think, this is a very good idea. Try to google "SMART indicators", and you will get over 25000 links.
Monday, October 29, 2012
What's real?
This kid is playing guitar. Guitar is fake. Music in his soul is real. Are evaluators always able to make such a distinction?
Here is the story. This boy was 2,5 years old at that time. He said: "Let me play my guitar." Then he took the 'guitar' and added: "This will be sad music." And I took a picture.
Saturday, October 27, 2012
Threat
Yesterday in the presentation on evaluator competencies I mentioned a worrisome trend in the evaluation market that I described as increased quasi-demand for quasi-evaluation of quasi-programs. It seemed to resonate. Several people talked to me after the presentation and confirmed that they shared my concern.
Wednesday, October 24, 2012
One thing evaluation associations should never do
Evaluation associations should not provide evaluation services to the external clients. This is extremely important for sustainability of associations. Let me explain why.
Evaluation associations serve their members and develop profession. If they start to provide evaluation services to the external clients they immediately get into competition with their members.
Possible consequences are very negative:
I am not saying indeed that income-generating activities are completely forbidden for associations. On the opposite, they can organize workshops for their members, they can provide certification services, they can issue journals and organize conferences etc. But it is crucial for associations to remember that their key clientele are their memebrs. In this respect associations have internal focus as opposed to business organizations.
Evaluation associations serve their members and develop profession. If they start to provide evaluation services to the external clients they immediately get into competition with their members.
Possible consequences are very negative:
- unequal access to information on potential contracts and financial resources inside association,
- conflicts between association and some of its members,
- problems with recruiting new members (it does not make much sense to join organization that instead of serving you competes with you),
- loosing focus,
- complete collapse.
I am not saying indeed that income-generating activities are completely forbidden for associations. On the opposite, they can organize workshops for their members, they can provide certification services, they can issue journals and organize conferences etc. But it is crucial for associations to remember that their key clientele are their memebrs. In this respect associations have internal focus as opposed to business organizations.
Friday, October 19, 2012
Are evaluation associations competing with each other?
The answer is YES, they are!
Of course, evaluation associations are uniqe organizations that serve their members and are oriented towards development of evaluation profession. But it would be a big mistake to deny the existence of competition between the associations since such a competition is one of the essential characteristics of the nature of their relationships. This is very similar to the relationships between the members of those associations who belong to the same professional organization (where they don't compete with each other) and compete in the evaluation market.
Why does competion between evaluation associations exist? Because they suggest similar services to the same and relatively small target group. Let me provide a personal example. Due to the very obvious reasons I can not afford attending more than one major international evaluation conference a year. When I choose which conference(s) to attend, it's a sign of competition between associations that host those conferences. In my case this year I had to choose between the conferences of the European Evaluation Society and the American Evaluation Association. When I decide which evaluation association to join, this is about my membership fees and most likely fees for some other services provided by the association. This is what we call competition.
There is another kind of competition between evaluation associations these days: regional associations compete for incorporating national associations. For instance, the emerging Asian Evaluation Society (not sure about the title - sorry!) invited evaluation networks from Kyrgzystan and Tajikistan to join. It means that our Central Asian colleagues who currently belong to the International Program Evaluation Network/IPEN (Newly Independent States) will need to decide if they still want to stay with IPEN or to quit IPEN and join another regional association (or do both).
Competition is good. It helps organizations improve their products and services. When the competition is healthy and honest, it is beneficial for all the parties involved (including customers indeed). Mature organizations value competition and respect their competitors. They know that competition is in fact one of the forms of collaboration in developing their business - very similar to sports. I do believe though that it is crucial to consider the competitive dimension of relationships between evaluation associations while developing their individual strategies and promoting collaboration bewteen them.
Of course, evaluation associations are uniqe organizations that serve their members and are oriented towards development of evaluation profession. But it would be a big mistake to deny the existence of competition between the associations since such a competition is one of the essential characteristics of the nature of their relationships. This is very similar to the relationships between the members of those associations who belong to the same professional organization (where they don't compete with each other) and compete in the evaluation market.
Why does competion between evaluation associations exist? Because they suggest similar services to the same and relatively small target group. Let me provide a personal example. Due to the very obvious reasons I can not afford attending more than one major international evaluation conference a year. When I choose which conference(s) to attend, it's a sign of competition between associations that host those conferences. In my case this year I had to choose between the conferences of the European Evaluation Society and the American Evaluation Association. When I decide which evaluation association to join, this is about my membership fees and most likely fees for some other services provided by the association. This is what we call competition.
There is another kind of competition between evaluation associations these days: regional associations compete for incorporating national associations. For instance, the emerging Asian Evaluation Society (not sure about the title - sorry!) invited evaluation networks from Kyrgzystan and Tajikistan to join. It means that our Central Asian colleagues who currently belong to the International Program Evaluation Network/IPEN (Newly Independent States) will need to decide if they still want to stay with IPEN or to quit IPEN and join another regional association (or do both).
Competition is good. It helps organizations improve their products and services. When the competition is healthy and honest, it is beneficial for all the parties involved (including customers indeed). Mature organizations value competition and respect their competitors. They know that competition is in fact one of the forms of collaboration in developing their business - very similar to sports. I do believe though that it is crucial to consider the competitive dimension of relationships between evaluation associations while developing their individual strategies and promoting collaboration bewteen them.
Monday, October 15, 2012
Tracking consultant's time and expenses
As an evaluation consultant I often work on several projects at the same time. It means that during one day I may spend some time working on one project, then switch to another project, and so on. Sometimes I charge the same client on an hourly basis for distance consultations and on a daily basis for on-site consultations. Tracking time under such circumstances is quite a challenging task. Several years ago I found software that perfectly fits my business. Since then tracking time and expenses became an easy task and even fun. I really like it! The third version of Timeless Time & Expense software has a more user-friendly interface. You may download a free 30-day trial version or purchase it for 69 USD.
Here is a brief description of the software proposed by the manufacturer: "Timeless Time & Expense® gives you the flexibility to track your time and expenses in a way that works for you. Rather than forcing you to track your time the way someone else thinks you should, Timeless Time & Expense allows you to model your time tracking the way you do business. It's easy and quick to use, so you can start tracking your time immediately."
Here is a brief description of the software proposed by the manufacturer: "Timeless Time & Expense® gives you the flexibility to track your time and expenses in a way that works for you. Rather than forcing you to track your time the way someone else thinks you should, Timeless Time & Expense allows you to model your time tracking the way you do business. It's easy and quick to use, so you can start tracking your time immediately."
Subscribe to:
Posts (Atom)