Monday, March 5, 2012
IWA on ECD is available online
International Workshop Agreement on Evaluation Capacity Development is available online as a complete document and as separate sections. Many thanks to the Evaluation Capacity Development Group for publishing it!
Tuesday, February 28, 2012
Evaluation and Evidence-Based Management
Evidence-Based Management (EBM) "is an emerging movement to explicitly use the current, best evidence in management decision-making. Its roots are in evidence-based medicine, a quality movement to apply the scientific method to medical practice." (Wiki)
The key principles of EBM are:
The key principles of EBM are:
- Face the hard facts, and build a culture in which people are encouraged to tell the truth, even if it is unpleasant.
- Be committed to "fact based" decision making -- which means being committed to getting the best evidence and using it to guide actions.
- Treat your organization as an unfinished prototype -- encourage experimentation and learning by doing.
- Look for the risks and drawbacks in what people recommend -- even the best medicine has side effects.
- Avoid basing decisions on untested but strongly held beliefs, what you have done in the past, or on uncritical "benchmarking" of what winners do.
EBM seems to be pretty much compatible with evaluation. In fact, EBM presumes mainstreaming evaluation.
Sunday, February 26, 2012
Being smart about SMART
Everybody knows SMART criteria for formulating goals (which should be Specific, Measurable, Attainable, Relevant and Time-bound). From my point of view, there is a common misinterpretation of SMART criteria that leads to substitition of goals by indicators. The latter is not good. It has some serious undesirable consequences, for example, implementors put too much emphasis on the indicators and less (or no) emphasis on making a difference.
Goals should be described in the terms of desired results (improvements). "Increased number of women served" is not an improvement. This is a sign of improvement, indicator (one of indicators) that could be used to measure achievement of the following goal: "Women and men have equal access to services." The project is about equality, but not about numbers. Numbers may indicate social change, but are not social change. Thus, goals become SMART when there are indicators that allow to measure achievement of the goals. Indicators help to make the goals SMART, but should not be used instead of goals.
Goals should be described in the terms of desired results (improvements). "Increased number of women served" is not an improvement. This is a sign of improvement, indicator (one of indicators) that could be used to measure achievement of the following goal: "Women and men have equal access to services." The project is about equality, but not about numbers. Numbers may indicate social change, but are not social change. Thus, goals become SMART when there are indicators that allow to measure achievement of the goals. Indicators help to make the goals SMART, but should not be used instead of goals.
Tuesday, February 14, 2012
Article on PATEM is available online in Evaluation and Program Planning
In this article I described Participatory Training Evaluation Method (PATEM) of measuring participants’ reaction to the training. PATEM provides rich information; allows to document evaluation findings; becomes organic part of the training that helps participants process their experience individually and as a group; makes sense to participants; is an interesting and exciting group activity; becomes a part of the learning process; is flexible enough to accommodate what are often dynamic and less-than-predictable training contexts; and can be used in groups of various sizes. The article includes step-by-step instructions on how to use PATEM and illustrative examples of participants’ feedback.
Monday, February 6, 2012
Freedom to blog
For two weeks I was working in Kazakhstan. Together with a local colleague we conducted two trainings for local NGOs, government officials and representatives of the oil company that initiated and supported those events. Kazakhstan is a great country that is developing really fast. Working there is always fun for me, and I see how the results of our trainings and consultations are used by various stakeholders which is very pleasing. Unfortunately this time I faced a problem there: most popular blogging sites are blocked by the government: one can not access Blogspot or Livejournal from Kazakhstan. Lesson learned: in two weeks I realized that freedom to blog (includes both reading blogs and writing) is important for me.
Sunday, January 22, 2012
17 out of 25 deleted posts recovered with the help of a fellow blogger
Today I received an e-mail from Christopher Lysy who created the Eval Central. He folllowed my blog for awhile and used Google Reader which acts as a type of backup. Chris kindly copied 17 of my lost 25 posts and e-mailed them to me. Thank you, Chris! The recovered posts are available below.
Consultations with stakeholders on sampling in program evaluation
When we are starting evaluation design we always consult with the key stakeholders and the program staff in particular. At that point in time we know almost nothing about the program, and program staff who know the program really well could be extremely helpful in planning evaluation. Sampling is one of the areas where we rely on the program people' opinion (I mean situations when we use purposeful sampling strategy indeed). They know much better where we should go and whom we need to talk to in order to learn about the program. People are also able and willing to explain rationale for choosing sites to visit and informants to interview. The question we ask is: where should we go and whom should we talk to in order to learn about your program as much as possible during the given (relatively short) period?
Follow by e-mail option
This is just a technical note. My blog has a new option now: follow by e-mail. If you wish to receieve new postings via e-mail you may use "Follow by e-mail" option in the menu. I find it very convenient. For example, I follow the AEA blog by e-mail.
My first OD mentor
This time I would like to write about a person who was my first mentor in organization development (OD). I met Margaret Molinari in Russia in 1990 at the so called Family Camp where American and Russian psychologists, social workers and OD consultants came with their families to spend time together, learn from each other and grow. The idea of such family camps was developed by the famous American family psychotherapist Virginia Satir. Family Camps in Russia were organized by two NGOs: Institute of International Connections (US) (IIC) and Institute for Professional Development / IPD (Russia).
At that time Margaret had PhD in psychology and worked in the capacity of a senior OD consultant at the General Motors headquarters in Detroit. I was just starting my independent consulting business in Russia. A quick reminder for the readers from outside our region: it was the very beginning of market economy in Russia, and we registered one of the first private consulting companies in the country.
My guess is that in 1990 Margaret could feel our energy and strong desire to try new things and to build professional consulting practice. But as an experienced person she could definitely see that we had no proper training and had extremely limited access to the special literature and other sources of information, let alone opportunities for training and formal education in the OD area.
Here is what she did in 1991 as a part of IIC exchange program: she invited four (!) Russian consultants (including myself) to the US to learn about OD for over 3 months. Margaret was responsible literally for everything: she hosted all four of us at her home, she developed and coordinated our fellowships programs, she raised money to support our stay in the US and cover all the expenses related to our visit. She even found a car for the two of us who attended an OD consulting training (several weeks) at the training center outside Detroit. At the same time Margaret was our mentor, helping us in a very intelligent and nice manner to find our ways in the profession.
When I arrived to the US in 1991 a had a pretty vague picture of my professional career and the future of my company. By the time I left the US I had a clear vision. That was a turning point for me - self-identification as an organization development consultant. Margaret played a key role in that process.
After leaving GM Margaret joined the Peace Corps and spent several years in the Latin America and then in Moldova, where she was responsible for training and development. Lucky Peace Corps-Moldova! It was very close to Moscow and I wanted to visit Margaret in Chisinau this year, but will not be able to do so because Margaret Molinary passed away on January 4, 2012 when she was on vacation in the US.
Margaret's son Jonah said: "She traveled around the world bringing amazing change and growth to so many lives." This is very true. My life was one of those.
Unusual role
Next year I will be working for a client in a very unusual (at least, for me) capacity. They hired me to help select an external evaluator for their organization. Will be fun to work on the client's side and on behalf of the client. I expect an interesting learning experience and appreciate my client for inviting me to do that.
In fact, this could be a niche for evaluation consultants if and when the evaluation market is developed enough in the terms of both demand and supply. There are pretty unique ethical issues related to such a role as well. In this case we have to take into consideration relationships between the consultant and his/her colleagues, while usually ethical considerations in evaluation business are related to client-consultant relationships and protection of informants.
Bloggers Series on the AEA's blog
In January, 2010 the American Evaluation Associationa established its own blog. This initiative turned out to be extremely successful. From my point of view the AEA's Tip-a-Day blog that publishes new information EVERY day (!) is the #1 evaluation blog in the world. One can find a lot of interesting and useful information there. Last Saturday AEA started "Bloggers Series" with Susan Kistler's post on Writing Weekly for aea365. Publication is worth reading if you are interested in evaluation and/or blogging.
Becoming an evaluation consultant in Russia
Once a year a colleague of mine invites me to meet his students at the Higher School of Economics in Moscow and talk to them about evaluation profession. This year's meeting took place yesterday. It went well, students asked many good questions and seemed to be interested. I like their questions because they help to reflect on various important aspects of my profession.
Yesterday they asked several questions about how to become an evaluation consultant. While answering I realized that it's not that easy to become an evaluator in Russia today.
First of all, it's impossible to get a degree in evaluation here. But it is possible to get a degree that would be relevant for evaluation work. For instance, it could be a degree in sociology, psychology or economics. Good knowledge of research methodology will be an asset for those who want to enter evaluation field. Two realistic strategies for learing about evaluation are self-education (very few publicationas are available in Russian though) and short trainng courses (not easy to find, may be expensive).
Second problem is indeed lack of evaluation and evaluation related vacancies. Even when companies need evaluation specialists (my company hires consultants on a short-term basis pretty often), they look for people with some practical experience rather than univesity graduates.
The conclusion is: it's not an easy thing to become an evaluator in Russia. If you are interested in evaluation profession, you should focus on studying research methodology and developing your research skills while being a student. Your first position after graduation can hardly be an evaluation one. It will take a while to get there. Read about evaluation, participate in the evaluation conferences and workshops if and when possible. Join evaluation association. Get in touch with the lead evaluation practitioners. At some point in time your door of opportunity will open.
Chain of results?
Numerous manuals on program planning and evaluation describe the so called chain of results as "inputs (resources) - activities - outputs - outcomes - impact." From my point of view there is one mistake and one inaccuracy in such a description.
Mistake: inputs and activities are not results. Programs are designed to convert inputs into results by implementing activities.
Inaccuracy: we need to make difference between a chain of expected results (goals) and a chain of actual results.
Chain of expected results reflects the program's intent and could be described as a hierarchy of goals: "expected outputs - espected outcome - expected impact".
Chain of actual results reflects what has been achieved by the program and could be described as follows: "actual outputs - actual outcome - actual impact."
The traditional presentation of chain of results in a form of "outputs-outcomes-impact" looks like a chain of actual results and is most often used to describe expected results.
Cynefin Framework
In this short video-presentation Dave Snowden from the Cognitive Edge "introduces the Cynefin Framework with a brief explanation of its origin and evolution and a detailed discussion of its architecture and function." Cynefin Framework helps "to see things from new viewpoints, assimilate complex concepts, and address real-world problems and opportunities".
Second International Conference on National Evaluation Capacities: presentations available online
UNDP Evaluation Office and the Public Service Commission (PSC) of South Africa co-hosted the second International Conference on National Evaluation Capacities on September 12-14, 2011 in Johannesburg. Papers and presentations are now available online.
ISO IWA: work in progress
According to the ISO rules, workshop participants now have 3 months to develop a document and submit it to ISO for publication. In order to do that we formed several working groups. Now we have pretty tight time line because the draft version of the document should be ready in December, 2011.
Mind mapping Evaluation Capacity
To get a flavor of what is going on at the workshop watch this mind map of Evaluation Capacity...
P.S. ISO IWA on ECD on the facebook
You may visit ECDG Facebook page at http://facebook.com/ECDG.net to receive daily updates from the ISO International Workshop Agreement (IWA) on Evaluation Capacity Development
Drafting ISO standard on evaluation capacity developmpent
Tomorrow morning I am heading to Geneva to participate in a 5-day ISO International Workshop Agreement (IWA) on Evaluation Capacity Development.
I decided to participate in that event due to several reasons:
1. I am interested in this sphere both as a researcher (my PhD dissertation was related to ECD) and a practitioner (I have been involved in evaluation capacity building in our region for quite a few years).
2. More and more clients these days ask to help them develop evaluation standards/policies/systems for their internal use.
3. The second issue of the Journal of Planning, Monitoring and Evaluation will be about evaluation guiding principles and standards. It will be good to learn more about that area as I am currently chairing the editorial board.
4. I have never participated in ISO standrads development. It will be intersting to try and get an idea of the process.
5. I know metting organizers and facilitator in person and respect them.
ECDG will partially support my travel and accomodation, which is indeed very nice of them.
It could be pretty much guranteed that in any case the outcomes of the joint work of a group of experienced and highly motivated people will be interesting and useful. I will share my impressions and tell about the results achieved in this blog.
Evaluation Week
The Independent Evaluation Group (IEG) of the World Bank Group, hosts its Fifth Annual Evaluation Week on October 24-27, 2011. The agenda looks great.
Discussion in Riga
The discussion in Riga went very well. There were two presenters: Bettina Windau from Bertelsman Stiffung (Germany) and me. Our presentations were complementary and it turned out that we agreed on the key issues - no fight between the panelists this time :) You can look at my presentation here to get a flavour of what I was speaking about. To my big surprise nobody argued with the suggestion not to publish evaluation reports.
Bettina is actively involved with the newly established Social Impact Analysts Association (SIAA). Interesting initiative. I invite the visitors of my blog to have a look at the SIAA website.
Grantakers Forum East in Riga
Grantakers Forum East in Riga will start tomorrow morning. One of the panel discsussions will address a number of questions related to evaluation:
• Is there a correlation between effort and the final results?
• When does it make sense to do the evaluation yourself, and when should an external evaluator be assigned?
• When should a summative, or end-of-project, evaluation be used and what advantages does a parallel evaluation have?
• How can results be disseminated and considered when developing future projects?
• How can the success of a project be measured as a contribution to an abstract goal?
• When do you find that you are evaluating something just for the sake of evaluation?
• How can evaluations contribute to the transparency of funding institutions?
I will participate in that discussion and will share my impressions in this blog.
Plagiarism unlimited
In 2007 I presented a paper on Evaluation Capacity Building at the conference that was organized by the Moscow Higher School of Economics (HSE). My paper was published on the HSE website (in Russian indeed). Recently I was surprised to discover that the entire paper was posted on the website of the Moscow based company called TeConsulting as their product. My name is not mentioned there at all. Amazing! Interestingly, there is no way to get in touch with the TeConsulting. Maybe, they do not exist any more?
Consultations with stakeholders on sampling in program evaluation
When we are starting evaluation design we always consult with the key stakeholders and the program staff in particular. At that point in time we know almost nothing about the program, and program staff who know the program really well could be extremely helpful in planning evaluation. Sampling is one of the areas where we rely on the program people' opinion (I mean situations when we use purposeful sampling strategy indeed). They know much better where we should go and whom we need to talk to in order to learn about the program. People are also able and willing to explain rationale for choosing sites to visit and informants to interview. The question we ask is: where should we go and whom should we talk to in order to learn about your program as much as possible during the given (relatively short) period?
Follow by e-mail option
This is just a technical note. My blog has a new option now: follow by e-mail. If you wish to receieve new postings via e-mail you may use "Follow by e-mail" option in the menu. I find it very convenient. For example, I follow the AEA blog by e-mail.
My first OD mentor
This time I would like to write about a person who was my first mentor in organization development (OD). I met Margaret Molinari in Russia in 1990 at the so called Family Camp where American and Russian psychologists, social workers and OD consultants came with their families to spend time together, learn from each other and grow. The idea of such family camps was developed by the famous American family psychotherapist Virginia Satir. Family Camps in Russia were organized by two NGOs: Institute of International Connections (US) (IIC) and Institute for Professional Development / IPD (Russia).
At that time Margaret had PhD in psychology and worked in the capacity of a senior OD consultant at the General Motors headquarters in Detroit. I was just starting my independent consulting business in Russia. A quick reminder for the readers from outside our region: it was the very beginning of market economy in Russia, and we registered one of the first private consulting companies in the country.
My guess is that in 1990 Margaret could feel our energy and strong desire to try new things and to build professional consulting practice. But as an experienced person she could definitely see that we had no proper training and had extremely limited access to the special literature and other sources of information, let alone opportunities for training and formal education in the OD area.
Here is what she did in 1991 as a part of IIC exchange program: she invited four (!) Russian consultants (including myself) to the US to learn about OD for over 3 months. Margaret was responsible literally for everything: she hosted all four of us at her home, she developed and coordinated our fellowships programs, she raised money to support our stay in the US and cover all the expenses related to our visit. She even found a car for the two of us who attended an OD consulting training (several weeks) at the training center outside Detroit. At the same time Margaret was our mentor, helping us in a very intelligent and nice manner to find our ways in the profession.
When I arrived to the US in 1991 a had a pretty vague picture of my professional career and the future of my company. By the time I left the US I had a clear vision. That was a turning point for me - self-identification as an organization development consultant. Margaret played a key role in that process.
After leaving GM Margaret joined the Peace Corps and spent several years in the Latin America and then in Moldova, where she was responsible for training and development. Lucky Peace Corps-Moldova! It was very close to Moscow and I wanted to visit Margaret in Chisinau this year, but will not be able to do so because Margaret Molinary passed away on January 4, 2012 when she was on vacation in the US.
Margaret's son Jonah said: "She traveled around the world bringing amazing change and growth to so many lives." This is very true. My life was one of those.
Unusual role
Next year I will be working for a client in a very unusual (at least, for me) capacity. They hired me to help select an external evaluator for their organization. Will be fun to work on the client's side and on behalf of the client. I expect an interesting learning experience and appreciate my client for inviting me to do that.
In fact, this could be a niche for evaluation consultants if and when the evaluation market is developed enough in the terms of both demand and supply. There are pretty unique ethical issues related to such a role as well. In this case we have to take into consideration relationships between the consultant and his/her colleagues, while usually ethical considerations in evaluation business are related to client-consultant relationships and protection of informants.
Bloggers Series on the AEA's blog
In January, 2010 the American Evaluation Associationa established its own blog. This initiative turned out to be extremely successful. From my point of view the AEA's Tip-a-Day blog that publishes new information EVERY day (!) is the #1 evaluation blog in the world. One can find a lot of interesting and useful information there. Last Saturday AEA started "Bloggers Series" with Susan Kistler's post on Writing Weekly for aea365. Publication is worth reading if you are interested in evaluation and/or blogging.
Becoming an evaluation consultant in Russia
Once a year a colleague of mine invites me to meet his students at the Higher School of Economics in Moscow and talk to them about evaluation profession. This year's meeting took place yesterday. It went well, students asked many good questions and seemed to be interested. I like their questions because they help to reflect on various important aspects of my profession.
Yesterday they asked several questions about how to become an evaluation consultant. While answering I realized that it's not that easy to become an evaluator in Russia today.
First of all, it's impossible to get a degree in evaluation here. But it is possible to get a degree that would be relevant for evaluation work. For instance, it could be a degree in sociology, psychology or economics. Good knowledge of research methodology will be an asset for those who want to enter evaluation field. Two realistic strategies for learing about evaluation are self-education (very few publicationas are available in Russian though) and short trainng courses (not easy to find, may be expensive).
Second problem is indeed lack of evaluation and evaluation related vacancies. Even when companies need evaluation specialists (my company hires consultants on a short-term basis pretty often), they look for people with some practical experience rather than univesity graduates.
The conclusion is: it's not an easy thing to become an evaluator in Russia. If you are interested in evaluation profession, you should focus on studying research methodology and developing your research skills while being a student. Your first position after graduation can hardly be an evaluation one. It will take a while to get there. Read about evaluation, participate in the evaluation conferences and workshops if and when possible. Join evaluation association. Get in touch with the lead evaluation practitioners. At some point in time your door of opportunity will open.
Chain of results?
Numerous manuals on program planning and evaluation describe the so called chain of results as "inputs (resources) - activities - outputs - outcomes - impact." From my point of view there is one mistake and one inaccuracy in such a description.
Mistake: inputs and activities are not results. Programs are designed to convert inputs into results by implementing activities.
Inaccuracy: we need to make difference between a chain of expected results (goals) and a chain of actual results.
Chain of expected results reflects the program's intent and could be described as a hierarchy of goals: "expected outputs - espected outcome - expected impact".
Chain of actual results reflects what has been achieved by the program and could be described as follows: "actual outputs - actual outcome - actual impact."
The traditional presentation of chain of results in a form of "outputs-outcomes-impact" looks like a chain of actual results and is most often used to describe expected results.
Cynefin Framework
In this short video-presentation Dave Snowden from the Cognitive Edge "introduces the Cynefin Framework with a brief explanation of its origin and evolution and a detailed discussion of its architecture and function." Cynefin Framework helps "to see things from new viewpoints, assimilate complex concepts, and address real-world problems and opportunities".
Second International Conference on National Evaluation Capacities: presentations available online
UNDP Evaluation Office and the Public Service Commission (PSC) of South Africa co-hosted the second International Conference on National Evaluation Capacities on September 12-14, 2011 in Johannesburg. Papers and presentations are now available online.
ISO IWA: work in progress
According to the ISO rules, workshop participants now have 3 months to develop a document and submit it to ISO for publication. In order to do that we formed several working groups. Now we have pretty tight time line because the draft version of the document should be ready in December, 2011.
Mind mapping Evaluation Capacity
To get a flavor of what is going on at the workshop watch this mind map of Evaluation Capacity...
P.S. ISO IWA on ECD on the facebook
You may visit ECDG Facebook page at http://facebook.com/ECDG.net to receive daily updates from the ISO International Workshop Agreement (IWA) on Evaluation Capacity Development
Drafting ISO standard on evaluation capacity developmpent
Tomorrow morning I am heading to Geneva to participate in a 5-day ISO International Workshop Agreement (IWA) on Evaluation Capacity Development.
I decided to participate in that event due to several reasons:
1. I am interested in this sphere both as a researcher (my PhD dissertation was related to ECD) and a practitioner (I have been involved in evaluation capacity building in our region for quite a few years).
2. More and more clients these days ask to help them develop evaluation standards/policies/systems for their internal use.
3. The second issue of the Journal of Planning, Monitoring and Evaluation will be about evaluation guiding principles and standards. It will be good to learn more about that area as I am currently chairing the editorial board.
4. I have never participated in ISO standrads development. It will be intersting to try and get an idea of the process.
5. I know metting organizers and facilitator in person and respect them.
ECDG will partially support my travel and accomodation, which is indeed very nice of them.
It could be pretty much guranteed that in any case the outcomes of the joint work of a group of experienced and highly motivated people will be interesting and useful. I will share my impressions and tell about the results achieved in this blog.
Evaluation Week
The Independent Evaluation Group (IEG) of the World Bank Group, hosts its Fifth Annual Evaluation Week on October 24-27, 2011. The agenda looks great.
Discussion in Riga
The discussion in Riga went very well. There were two presenters: Bettina Windau from Bertelsman Stiffung (Germany) and me. Our presentations were complementary and it turned out that we agreed on the key issues - no fight between the panelists this time :) You can look at my presentation here to get a flavour of what I was speaking about. To my big surprise nobody argued with the suggestion not to publish evaluation reports.
Bettina is actively involved with the newly established Social Impact Analysts Association (SIAA). Interesting initiative. I invite the visitors of my blog to have a look at the SIAA website.
Grantakers Forum East in Riga
Grantakers Forum East in Riga will start tomorrow morning. One of the panel discsussions will address a number of questions related to evaluation:
• Is there a correlation between effort and the final results?
• When does it make sense to do the evaluation yourself, and when should an external evaluator be assigned?
• When should a summative, or end-of-project, evaluation be used and what advantages does a parallel evaluation have?
• How can results be disseminated and considered when developing future projects?
• How can the success of a project be measured as a contribution to an abstract goal?
• When do you find that you are evaluating something just for the sake of evaluation?
• How can evaluations contribute to the transparency of funding institutions?
I will participate in that discussion and will share my impressions in this blog.
Plagiarism unlimited
In 2007 I presented a paper on Evaluation Capacity Building at the conference that was organized by the Moscow Higher School of Economics (HSE). My paper was published on the HSE website (in Russian indeed). Recently I was surprised to discover that the entire paper was posted on the website of the Moscow based company called TeConsulting as their product. My name is not mentioned there at all. Amazing! Interestingly, there is no way to get in touch with the TeConsulting. Maybe, they do not exist any more?
Saturday, January 21, 2012
Undeleting posts
Today I clicked the wrong button and unintentionally deleted the last 25 posts from my blog. It seems like there is no "Undelete" option here. Of course this will not be an extremely dramatic event for the evaluation profession worldwide, but it was somewhat important for me. Lesson learned: backup your posts regularly.
Subscribe to:
Posts (Atom)