Evaluation Space

Freedom to blog

2012-02-06T14:25:00.003+04:00

For two weeks I was working in Kazakhstan. Together with a local colleague we conducted two trainings for local NGOs, government officials and representatives of the oil company that initiated and supported those events. Kazakhstan is a great country that is developing really fast. Working there is always fun for me, and I see how the results of our trainings and consultations are used by various stakeholders which is very pleasing. Unfortunately this time I faced a problem there: most popular blogging sites are blocked by the government: one can not access Blogspot or Livejournal from Kazakhstan. Lesson learned: in two weeks I realized that freedom to blog (includes both reading blogs and writing) is important for me.

17 out of 25 deleted posts recovered with the help of a fellow blogger

2012-01-22T19:16:00.000+04:00

Today I received an e-mail from Christopher Lysy who created the Eval Central. He folllowed my blog for awhile and used Google Reader which acts as a type of backup. Chris kindly copied 17 of my lost 25 posts and e-mailed them to me. Thank you, Chris! The recovered posts are available below.

Consultations with stakeholders on sampling in program evaluation

When we are starting evaluation design we always consult with the key stakeholders and the program staff in particular. At that point in time we know almost nothing about the program, and program staff who know the program really well could be extremely helpful in planning evaluation. Sampling is one of the areas where we rely on the program people' opinion (I mean situations when we use purposeful sampling strategy indeed). They know much better where we should go and whom we need to talk to in order to learn about the program. People are also able and willing to explain rationale for choosing sites to visit and informants to interview. The question we ask is: where should we go and whom should we talk to in order to learn about your program as much as possible during the given (relatively short) period?

Follow by e-mail option

This is just a technical note. My blog has a new option now: follow by e-mail. If you wish to receieve new postings via e-mail you may use "Follow by e-mail" option in the menu. I find it very convenient. For example, I follow the AEA blog by e-mail.

My first OD mentor

This time I would like to write about a person who was my first mentor in organization development (OD). I met Margaret Molinari in Russia in 1990 at the so called Family Camp where American and Russian psychologists, social workers and OD consultants came with their families to spend time together, learn from each other and grow. The idea of such family camps was developed by the famous American family psychotherapist Virginia Satir. Family Camps in Russia were organized by two NGOs: Institute of International Connections (US) (IIC) and Institute for Professional Development / IPD (Russia).

At that time Margaret had PhD in psychology and worked in the capacity of a senior OD consultant at the General Motors headquarters in Detroit. I was just starting my independent consulting business in Russia. A quick reminder for the readers from outside our region: it was the very beginning of market economy in Russia, and we registered one of the first private consulting companies in the country.

My guess is that in 1990 Margaret could feel our energy and strong desire to try new things and to build professional consulting practice. But as an experienced person she could definitely see that we had no proper training and had extremely limited access to the special literature and other sources of information, let alone opportunities for training and formal education in the OD area.

Here is what she did in 1991 as a part of IIC exchange program: she invited four (!) Russian consultants (including myself) to the US to learn about OD for over 3 months. Margaret was responsible literally for everything: she hosted all four of us at her home, she developed and coordinated our fellowships programs, she raised money to support our stay in the US and cover all the expenses related to our visit. She even found a car for the two of us who attended an OD consulting training (several weeks) at the training center outside Detroit. At the same time Margaret was our mentor, helping us in a very intelligent and nice manner to find our ways in the profession.

When I arrived to the US in 1991 a had a pretty vague picture of my professional career and the future of my company. By the time I left the US I had a clear vision. That was a turning point for me - self-identification as an organization development consultant. Margaret played a key role in that process.

After leaving GM Margaret joined the Peace Corps and spent several years in the Latin America and then in Moldova, where she was responsible for training and development. Lucky Peace Corps-Moldova! It was very close to Moscow and I wanted to visit Margaret in Chisinau this year, but will not be able to do so because Margaret Molinary passed away on January 4, 2012 when she was on vacation in the US.

Margaret's son Jonah said: "She traveled around the world bringing amazing change and growth to so many lives." This is very true. My life was one of those.

Unusual role

Next year I will be working for a client in a very unusual (at least, for me) capacity. They hired me to help select an external evaluator for their organization. Will be fun to work on the client's side and on behalf of the client. I expect an interesting learning experience and appreciate my client for inviting me to do that.

In fact, this could be a niche for evaluation consultants if and when the evaluation market is developed enough in the terms of both demand and supply. There are pretty unique ethical issues related to such a role as well. In this case we have to take into consideration relationships between the consultant and his/her colleagues, while usually ethical considerations in evaluation business are related to client-consultant relationships and protection of informants.

Bloggers Series on the AEA's blog

In January, 2010 the American Evaluation Associationa established its own blog. This initiative turned out to be extremely successful. From my point of view the AEA's Tip-a-Day blog that publishes new information EVERY day (!) is the #1 evaluation blog in the world. One can find a lot of interesting and useful information there. Last Saturday AEA started "Bloggers Series" with Susan Kistler's post on Writing Weekly for aea365. Publication is worth reading if you are interested in evaluation and/or blogging.

Becoming an evaluation consultant in Russia

Once a year a colleague of mine invites me to meet his students at the Higher School of Economics in Moscow and talk to them about evaluation profession. This year's meeting took place yesterday. It went well, students asked many good questions and seemed to be interested. I like their questions because they help to reflect on various important aspects of my profession.

Yesterday they asked several questions about how to become an evaluation consultant. While answering I realized that it's not that easy to become an evaluator in Russia today.

First of all, it's impossible to get a degree in evaluation here. But it is possible to get a degree that would be relevant for evaluation work. For instance, it could be a degree in sociology, psychology or economics. Good knowledge of research methodology will be an asset for those who want to enter evaluation field. Two realistic strategies for learing about evaluation are self-education (very few publicationas are available in Russian though) and short trainng courses (not easy to find, may be expensive).

Second problem is indeed lack of evaluation and evaluation related vacancies. Even when companies need evaluation specialists (my company hires consultants on a short-term basis pretty often), they look for people with some practical experience rather than univesity graduates.

The conclusion is: it's not an easy thing to become an evaluator in Russia. If you are interested in evaluation profession, you should focus on studying research methodology and developing your research skills while being a student. Your first position after graduation can hardly be an evaluation one. It will take a while to get there. Read about evaluation, participate in the evaluation conferences and workshops if and when possible. Join evaluation association. Get in touch with the lead evaluation practitioners. At some point in time your door of opportunity will open.

Chain of results?

Numerous manuals on program planning and evaluation describe the so called chain of results as "inputs (resources) - activities - outputs - outcomes - impact." From my point of view there is one mistake and one inaccuracy in such a description.

Mistake: inputs and activities are not results. Programs are designed to convert inputs into results by implementing activities.

Inaccuracy: we need to make difference between a chain of expected results (goals) and a chain of actual results.

Chain of expected results reflects the program's intent and could be described as a hierarchy of goals: "expected outputs - espected outcome - expected impact".

Chain of actual results reflects what has been achieved by the program and could be described as follows: "actual outputs - actual outcome - actual impact."

The traditional presentation of chain of results in a form of "outputs-outcomes-impact" looks like a chain of actual results and is most often used to describe expected results.

Cynefin Framework

In this short video-presentation Dave Snowden from the Cognitive Edge "introduces the Cynefin Framework with a brief explanation of its origin and evolution and a detailed discussion of its architecture and function." Cynefin Framework helps "to see things from new viewpoints, assimilate complex concepts, and address real-world problems and opportunities".

Second International Conference on National Evaluation Capacities: presentations available online

UNDP Evaluation Office and the Public Service Commission (PSC) of South Africa co-hosted the second International Conference on National Evaluation Capacities on September 12-14, 2011 in Johannesburg. Papers and presentations are now available online.

ISO IWA: work in progress

According to the ISO rules, workshop participants now have 3 months to develop a document and submit it to ISO for publication. In order to do that we formed several working groups. Now we have pretty tight time line because the draft version of the document should be ready in December, 2011.

Mind mapping Evaluation Capacity

To get a flavor of what is going on at the workshop watch this mind map of Evaluation Capacity...

P.S. ISO IWA on ECD on the facebook

You may visit ECDG Facebook page at http://facebook.com/ECDG.net to receive daily updates from the ISO International Workshop Agreement (IWA) on Evaluation Capacity Development

Drafting ISO standard on evaluation capacity developmpent

Tomorrow morning I am heading to Geneva to participate in a 5-day ISO International Workshop Agreement (IWA) on Evaluation Capacity Development.

I decided to participate in that event due to several reasons:

1. I am interested in this sphere both as a researcher (my PhD dissertation was related to ECD) and a practitioner (I have been involved in evaluation capacity building in our region for quite a few years).

2. More and more clients these days ask to help them develop evaluation standards/policies/systems for their internal use.

3. The second issue of the Journal of Planning, Monitoring and Evaluation will be about evaluation guiding principles and standards. It will be good to learn more about that area as I am currently chairing the editorial board.

4. I have never participated in ISO standrads development. It will be intersting to try and get an idea of the process.

5. I know metting organizers and facilitator in person and respect them.

ECDG will partially support my travel and accomodation, which is indeed very nice of them.

It could be pretty much guranteed that in any case the outcomes of the joint work of a group of experienced and highly motivated people will be interesting and useful. I will share my impressions and tell about the results achieved in this blog.

Evaluation Week

The Independent Evaluation Group (IEG) of the World Bank Group, hosts its Fifth Annual Evaluation Week on October 24-27, 2011. The agenda looks great.

Discussion in Riga

The discussion in Riga went very well. There were two presenters: Bettina Windau from Bertelsman Stiffung (Germany) and me. Our presentations were complementary and it turned out that we agreed on the key issues - no fight between the panelists this time :) You can look at my presentation here to get a flavour of what I was speaking about. To my big surprise nobody argued with the suggestion not to publish evaluation reports.
Bettina is actively involved with the newly established Social Impact Analysts Association (SIAA). Interesting initiative. I invite the visitors of my blog to have a look at the SIAA website.

Grantakers Forum East in Riga

Grantakers Forum East in Riga will start tomorrow morning. One of the panel discsussions will address a number of questions related to evaluation:

• Is there a correlation between effort and the final results?

• When does it make sense to do the evaluation yourself, and when should an external evaluator be assigned?

• When should a summative, or end-of-project, evaluation be used and what advantages does a parallel evaluation have?

• How can results be disseminated and considered when developing future projects?

• How can the success of a project be measured as a contribution to an abstract goal?

• When do you find that you are evaluating something just for the sake of evaluation?

• How can evaluations contribute to the transparency of funding institutions?

I will participate in that discussion and will share my impressions in this blog.

Plagiarism unlimited

In 2007 I presented a paper on Evaluation Capacity Building at the conference that was organized by the Moscow Higher School of Economics (HSE). My paper was published on the HSE website (in Russian indeed). Recently I was surprised to discover that the entire paper was posted on the website of the Moscow based company called TeConsulting as their product. My name is not mentioned there at all. Amazing! Interestingly, there is no way to get in touch with the TeConsulting. Maybe, they do not exist any more?

Undeleting posts

2012-01-21T15:24:00.002+04:00

Today I clicked the wrong button and unintentionally deleted the last 25 posts from my blog. It seems like there is no "Undelete" option here. Of course this will not be an extremely dramatic event for the evaluation profession worldwide, but it was somewhat important for me. Lesson learned: backup your posts regularly.

Business ethics

2011-09-02T10:02:00.004+04:00

Some time ago my company was invited to be included as a local partner (subcontractor) into somebody's proposal. We agreed and sent resumes of our key staff along with other documents. Next morning I received a message saying that they really liked one of my partners' CVs and would like to hire that person to a full time position. In the end they asked for my opinion regarding that idea in a very polite manner.

I found the request flagrant and shared only some thoughts I had about it in a very clear and polite way.

Interstingly, they did not apologize, which of course told me even more about their style than the initial request.

Vacation of evaluation consultant

2011-08-26T15:08:00.000+04:00

I am working in consulting business for about 25 years. One of the lessons learned is: you need to take care of yourself if you wish to sustain your business. Some applications of this lesson are:

a) Plan your vacation in advance and consider it an essential part of your annual schedule. Otherwise you will have no vacation.

b) Do not try to "work a little bit" on vacation. If you do that, most likely you will have no vacation.

I have not written anything here for a while since I was on vacation. Now I am back :)

Rates for "local" and "international" consultants

2011-08-02T18:20:00.000+04:00

In the early 90-s I felt very uncomfortable when my international colleagues asked to explain the difference in the Russian hotel rates for foreigners and local citizens. Our international guests had to pay much more for the same room. And probably the hotels' logic was pretty straight forward: "we should charge more because they have more money out there."

Recently I had a conversation with an international organization that has one of its offices in Moscow. They explained me that they have different rates for "international" consultants and "local" consultants. Locals are paid less indeed. If I am based in Moscow and work for them here, I am "local". If they invite me to work for them in Ukraine, I become "international." My qualification, experience (including international) and degree (PhD received in the US) do not matter. They are supposed to pay in accordance with where I live.

This made me think of the hotel rates. Maybe we should get back to the "international" rates for foreign guests? That would be logical :)

Seriously speaking, I think that international organizations should re-consider their policies in the area of consultants' rates and pay them in accordance with professional qualification and track record.

AEA listening to the world

2011-07-19T10:25:00.000+04:00

An invitation to participate in AEA's International Listening Project has been posted recently by Victor Kuo in the AEA's blog. This is an interesting and unique initiative that is supposed to contribute to the development of AEA international policy. Data collection should be completed by August 1, 2011.
AEA is one of the key players in the development of evaluation profession worldwide. Thus, it will be very interesting to see the results of this survey and the AEA's reaction. An online questionnaire for those who are willing to be listened is accessible here.

Evaluation'2011 Conference Program available online

2011-07-11T20:17:00.000+04:00

As usual, the program is very rich if not overwhelming (600 sessions, over 1000 presenters). I would highly recommend Jane Davidson's pre-conference workshop. She decided to conduct a 2-day workshop for the first time. Must be very good!

ISO International Workshop Agreement (IWA) on Evaluation Capacity Development

2011-07-06T13:04:00.000+04:00

Evaluation Capacity Development Group (ECDG) will host an ISO International Workshop Agreement (IWA) on Evaluation Capacity Development on October17-21, 2011 in Geneva, Switzerland. Organizers expect about 60 participants from all over the world. The workshop will be facilitated by Bob Williams.

Proceedings from the 2009 international conference on national evaluation capacities

2011-06-29T07:34:00.000+04:00

UNDP announced publication of the Proceedings from the 2009 international conference on national evaluation capacities (available online for free download). I had a pleasure to participate in that conference and to present a paper "Evaluation Capacity Building Strategy: Towards a Mature Profession", which is now included into the conference proceedings.

Comments on the previous post

2011-06-20T20:31:00.002+04:00

I sent my previous post to the international network of evaluators and received a number of interesting responses. Some of them explained why the TOR could look so strange, and another ones suggested possible strategies for evaluation consultant who receives such a TOR. Let me share with you a summary of comments I received. By the way, you may also want to sbuscribe to XCeval at Yahoo groups.

What is it?

an extreme example of a bad ToR! What are these people thinking when they issue such unrealistic requests for how evaluations should and can be conducted?!
mistake in TOR
an NGO (or NGO staff person) who does not have experience in evaluations
misinformed understanding of the nature and purpose of evaluation
if it is not a mistake, then it is a direct -poor- product of the evaluation supply side system
an extreme case of a common problem, complex, impact evaluations requested that are totally under budgeted in time and money… everyone knows this is a problem, except the people who commission evaluations.
a way to see if applying consultants are honest (I can do the work, but not in the proposed time frame)
a project that can only afford to pay for 10 working days
they ran short of money
they have a deadline
this report is probably an auditing requirement, so you have some room to negotiate
:) these are Venus days not Earth days. (A day on Venus takes 243 Earth days)

What to do?

don’t take the assigment
write saying you are interested but need to have that question addressed.
ask for 20 days of paid field work and then another two weeks to write it up. Suggest the draft at the end of three weeks field work and ten days writing and then wait for comments--no more than a week. Then send the final report. You will be paid for 20 days but working for 40.
suggest conducting exploratory evaluation. For example, evaluability assessment, rapid feedback evaluation, or an evaluation synthesis, if there are prior research and evaluation studies available
:) put a monkey behind a keybord, and when is done, draft the title page send the report, take the cheque and make a donation!!

Speaking of high quality evaluations...

2011-06-17T00:10:00.001+04:00

Today I received a TOR from a very well-kown international organization. They are looking for an international evaluation consultant who will conduct a final outcome evaluation of a 7-year project in a pretty complex (post-conflict) environment. Evaluation should include analysis of numerous documents and a field study. During the field study evaluator should conduct interviews with representatives of about 10 different groups of stakeholders (not less than 20 interviewees) and use individual and group interviews as well as observation. Changes should be assessed both at the community and regional levels.
And ... can you believe this? ... they allow 10 (!) days for everything. TOR says that the draft report should be ready in five days after start of consultancy (I guess, by the middle of the field study) and that the final report should be ready in 10 days after start of consultancy (by the end of the field study).
Since they produced such a TOR, there must be evaluation consultants who will accept it. This is sad.

Using virtual expert panels in evaluation

2011-06-08T23:45:00.001+04:00

Nobody can be expert in everything. Often when we conduct evaluations we need expert contributions in the subject areas we do not have enough knowledge about. Sometimes it is possible to work with experts face-to-face. Sometimes experts whom we need are located far from the place where we work and are too busy and expensive to become our team members. In such cases we use virtual expert panels: experts work with us on the key issues related to their respective areas via e-mail or skype. Skype allows to have teleconferences with several experts. And it does not matter where they are. Another important thing about virtual expert panels is that you can afford hiring world-class experts on an hourly basis since they work from home, which turns out to be a cost-effective approach.
One example. Some time ago we conducted an evaluation in Russia related to economic think tanks. We needed additional expertise and found two outstanding experts, one of whom was in the US and another one - in Egypt. They helped us a lot.

Customer is not always right

2011-06-03T09:24:00.000+04:00

Good evaluation should be customer/intended user focused indeed. It means in particular that the customer's needs should be clarified and considered as the top priority by evaluation consultant. But it does not mean that evaluation consultant should perceive everything that customer says as the 'right thing'. Customer can not be always right simply because in many cases he knows less about evaluation than evaluation consultant. That is why the consultant's responsibility sometimes is to disagree with the customer and explain why the customer is wrong. Not an easy thing to do.

KPIs as an illusion of the real world

2011-05-24T12:42:00.000+04:00

"I've always believed that people who rely too much on indicators don't have real courage in their convictions and try to blame their own bad mistakes on "the indicators". As someone's already said, the markets are simply too complex to be boiled down to a set of statistics." This statement related to equity markets has been made on the LinkedIn.
I would not blame anybody for the lack of courage but it is very true for the programs we are dealing with: life is too complex to be boiled down to a set of statistics. Or KPIs. I am afraid that people are getting too much excited about KPIs. Indicators present a very simplified picture of the world. This is good for some purposes, but it should not be percieved as the real world. Such perception could have extremely negative consequences.

Randomized trials as the hottest thing in the fight against poverty

2011-05-20T09:53:00.004+04:00

On May 18 Nicholas Kristof published an article "Getting Smart on Aid" in NY Times. Some of his thoughts might be really interesting for the evaluation community in the US and beyond:

Economics "possesses a rigor that other fields in the social sciences don’t — and often greater relevance as well. That’s why economists are shaping national debates about everything from health care to poverty, while political scientists often seem increasingly theoretical and irrelevant."
"Economists are successful imperialists of other disciplines because they have better tools. Educators know far more about schools, but economists have used rigorous statistical methods to answer basic questions"
"How can we most effectively break cycles of poverty? For decades, we had answers that were mostly anecdotal or hot air. But, increasingly, we are now seeing economists provide answers that are rigorously field-tested, akin to the way drugs are tested in randomized controlled trials, yielding results that are particularly credible and persuasive."
"Randomized trials are the hottest thing in the fight against poverty"

Comments on the article are a nice addition to the original text and demonstrate variety of opinions.

PS And ... I am back to my blog

Extraterrestrials and the use of information

2010-10-04T10:13:00.003+04:00

Let me share an interesting citation from a fiction novel 'Beetle in the Anthill' written by well-known Russian authors Arkadi and Boris Strugatski in 1979. Translated into English and available at Amazon
One of the characters is an extraterrestrial representing a completely different civilization (his name is Shekn Itrch). He got to know our civilization very well and was strucked by the fact that people (the mankind) explore the world extensively and do not use information they collect.
Here is what he says: “I wonder why do you aspire to learn about things that do not make sense? What will you do with that? You learn more and more and do not use what you know.”
(this is my translation of his statement)
Surprisingly it reminded me some of Patton's books :) Extraterrestrials (at least some of
them) turn to be pretty much utilization focused.

Use of proverbs in evaluation: "A word is enough to the wise"

2010-03-31T21:20:00.008+04:00

I like proverbs. One of my favorite books is collection of Russian proverbs by Vladimir Dal'. One of the possible ways of using proverbs in evaluation is presentation of lessons learned in the form of proverbs. I start with a proverb that briefly decribe some idea and then add a paragraph specific to the program under study. Below are some examples that illustrate the proposed use of proverbs taken from our recent evaluation reports. I excluded the names of the agencies and the countries due to ethical considerations.

“Well begun is half done” Evaluability of the program substantially depends on the quality of its design. If the programme logic is not coherent and there are no good indicators and baseline data, the program will face serious challenges in measuring its progress and results. It’s much more effective to conduct evaluability assessment at the design stage of the program than to undertake it when the program is in progress or coming to an end.

“Nothing comes out of the sack but what was in it” You can’t have a strong country program without having a strong country team. Country representatives play crucial role at all stages of the country program life cycle including needs assessment, design, implementation and evaluation. While making its country representatives responsible for the program, the Agency should delegate them enough authority and build their capacity. Agency should also invest into building positive professional image of its country representatives. Delegation, capacity building and creation of positive image do not require huge financial investments on the Agency side.

“No bees, no honey; no work, no money” In order to implement the program in accordance with the country priorities, fundraising should be done at the country level by the country representatives/offices and constituents, as neither regional, nor sub-regional office are able to find money to meet all the countries’ needs. Such fundraising should be encouraged, supported and ‘legalized’ (i.e. incorporated into the Agency’s strategy, policies and procedures).

By the way, screening dozens of proverbs to find the right one that decribes lesson leared really well is FUN! And it helps to better formulate the lesson itself because it makes you think about the essense of it.

Outstanding publication on RCTs :)

2010-03-03T23:24:00.004+03:00

The title of the article is "Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials".
Outstanding! :))))
Here is the "Conclusions" section of it:
"...As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute."

Prague vs. San-Antonio: financial dimension of international conferences

2010-03-01T13:51:00.011+03:00

European Evaluation Society will have its annual conference in Prague on October 6-8, 2010 (3 full days). Pre-conference workshops will take place on October 4 and 5.
American Evaluation Association will have its annual conference in San-Antonio on November 10-13 (3,5 days). Pre-conference workshops will take place on November 8 through 10.
I decided to compare the costs of attending these conferences for a person who lives in Moscow, Russia.
Calculation shows min and max options. Minimum option includes a 1/2 day workshop and the conference + stay at an inexpensive hotel (4 nights). Maximum option includes a 2 day workshop, conference, luncheon + stay at the conference hotel (more expensive, 7 nights).
The table above shows that registration fee for the European conference is 5,5 times higher than for the American one. But the cost of travel to the US for me is much higher than the cost of travel to Prague.
The costs of visas and meals were not included since they could be considered similar.
Interestingly, the total amount of money I would have to spend does not depend on which conference I decide to attend.

International Evaluation Conference in Kyrgyzstan

2010-02-18T12:40:00.002+03:00

International Program Evaluation Network (IPEN) will have its 10th annual conference in Bishkek, Kyrgyzstan on September 24-25, 2010. Pre-conference workshops will take place on September 23rd. The conference theme this year is "Design and implementation of monitoring and evaluation systems." Application form is available at the conference website. I attended all the IPEN conferences since 2000 and taught at three Evaluation Schools sponsored by IPEN. IPEN conferences are always fun: great atmosphere, interesting presentations and discsussions. As an IPEN 'veteran' and fan I am biased indeed, but this is not just my opinion :)

AEA365: Hot Tips, Cool Tricks, and Rad Resources for Evaluators

2010-01-11T16:04:00.003+03:00

AEA365 is a new blog sponsored by the American Evaluation Association. I think, this is a great idea to invite evaluators share resources they find useful. Bravo, AEA!

A New Year post :)

2009-12-29T00:42:00.006+03:00

What I like about my job:
- creativity, need to think and analyze,
- dealing with many very different people,
- working in a good team,
- travelling, working internationally,
- opportunity to learn a lot about different spheres of life and to learn continually,
- solving complex problems that become even more complex as time goes by,
- opportunity to be useful for organizations, programs, projects and individuals.
What I don't like about my job:
- when there is too much travelling,
- when the tasks are either unrealistic or not real (fictitious),
- when there is no enough time to do the job with the highest possible quality,
- when I have night flights.
I had a lot of interesting work in different countries in 2009. Sometimes there was too much travelling, several times - serious time pressure, and I had a few night flights :) Overall it has been a very good year.

Conference on 'National Evaluation Capacities' in Marocco

2009-12-10T22:14:00.003+03:00

International conference on evaluation capacity building will take place in Casablanca, Marocco next week (Dec 15-17). Conference is organized by UNDP. Information on the conference program, participants and presentations is available here. I will present a concept of comprehensive evaluation capacity building strategy based on developing evaluation into a mature profession.

Evaluation book in Russian: who is interested?

2009-12-06T08:28:00.000+03:00

In the end of September a new evaluation book in Russian "Program evaluation: Methodology and practice" became available online. Below is geography of the visitors as of December 6, 2009. The total number of visitors is 463. According to my information some visitors downloaded the book and then shared it with as amny as 40 colleagues. Thus we don't really know how many people actually have the book. But nevertheless the statistics is interesting. Most vistors represented Russian Federation - no surprises. Over a 100 of those were from Moscow. Ukraine goes second with 44 visitors. Interestingly, there were 40 visitors from Kyrgyzstan (3rd), where the International Program Evaluation Network will have its 10th conference next year. If we consider that the population of Russia is 145 mln., population of Ukraine is 46 mln., and population of Kyrgyzstan is 5 mln., we'll have to admit that Kyrgyzstan is far ahead all other countries having 8 visitors/million people. Russia is only the fourth with 1,53 books/millon. And Ukraine is the 7th with about 1 book/million. So, the place for the next IPEN conference seems to be chosen correctly.

Number and geography of vistors (as of Dec 6, 2009)
Russian Federation (RU) 222 (Moscow - over 100)
Ukraine (UA) 44
Kyrgyzstan (KG) 40
United States (US) 37
Kazakstan (KZ) 27
Uzbekistan (UZ), Germany (DE), Tajikistan (TJ) 9
Belarus (BY) 6
Armenia (AM) 5
Lithuania (LT) 4
Philippines (PH), Georgia (GE), United Kingdom (GB) 3
Moldova, Republic of (MD), Canada (CA), Azerbaijan (AZ), Switzerland (CH) 2
Denmark (DK), Poland (PL), China (CN), Japan (JP), Israel (IL), Iraq (IQ), Turkey (TR), Romania (RO), Italy (IT), Australia (AU), Serbia (RS) 1

Meeting in Minnesota

2009-11-17T04:40:00.000+03:00

I did not know that Minnesota Evaluation Association was founded informally over 20 years ago. Although it was formally established only in 2003. Today it has over 100 members. Interstingly, this is somewhat closer to what we have in the Newly Independent States than AEA with its 5000 members. So, maybe the whole notion of international partnertships should be reconsidered at the level of AEA affiliates. It simply might work better at this level. It just crossed my mind on the way back from the meeting with evaluation colleagues in Minnesota. The meeting was informal and friendly. Participants asked very good questions that lead our dicsussion far beyond the evaluation training issues. We talked about development of evaluation profession, values, vison, difficult decisions we have to make from time to time. I enjoyed the meeting and would like to appreciate Minnesota Evaluation Association for this opportunity. Hopefully, at least some of the ideas I shared will prove useful for those who attended.

Innovative ways of teaching evaluation

2009-11-16T23:55:00.000+03:00

This afternoon I will talk to the members of Minnesota Evaluation Association about innovations in teaching evaluation. My PPT presentation is available here.

"Getting-to-maybe" project proposals?

2009-11-15T15:11:00.000+03:00

Conversation around complexity and use of system approach in evaluation sounds to me as a call for new approach to planning and programming. I wonder, what the new guidelines for program design should look like. If the existing approach based on linear model and logical framework is too simplistic, what should the new approach be? Will the donors encourage "getting-to-maybe" proposals?

Evaluators' skills - note from the AEA conference

2009-11-13T03:58:00.000+03:00

Evaluators are contionously looking for new ways of encreasing effectiveness of their work. In their search they naturally go beyond evaluation. For instance, they get involved in evaluation capacity building in organizations; they participate in program design; they facilitate organizational change interventions. I think, all these areas are important for evaluation development. BUT it's important for evaluators to be aware of the fact that in order to implement non-evaluation activities in a professional manner they need to have proper skills. For example, today at one of the sessions we discussed how to use coaching in developing evaluation capacity. I have a strong OD background and do not consider myslef a coach - I know just a little bit about that. Some of my friends are doing coaching on a professional basis. It requires SPECIAL SKILLS. Good evaluator, good researcher or good trainer is not a good a good coach by default. I think, evaluators should take that seriously and should not implement non-evaluation activties without proper training and quality assurance.

Presentation on Organization Paradigms and Evaluation

2009-11-11T23:21:00.000+03:00

Here is my PPT presentation at the AEA conference for those who might be interested.

Bloggers Roundtable at Evaluation 2009

2009-11-03T19:08:00.000+03:00

The Bloggers Roundtable at Evaluation 2009 to be held on Saturday from 11:50 am to 12:35 pm in Suwanee 21.

Participants will discuss:

The state of evaluation blogging (whose contributing, how much, etc.)
Interest and opportunities for group action and improving the state of evaluator blogging (for instance through posting on a common topic such as the conference theme or having an evaluation collation that brings together posts from across sites)
Interest and opportunities for collaborative learning (for instance through learning about tools and tips)
Other items from the group

Presentation on teaching evaluation and good Ukrainian food in Minnesota

2009-10-28T23:23:00.000+03:00

Minnesota Evaluation Association kindly invited me to visit and make a presentation in Minneapolis immediately after the AEA conference. It's an honor for me indeed.
I intend to share innovative methods for teaching evaluation that were developed and applied in my international work. This presentation will start with a brief history of my work in the evaluation profession and then focus on several methods of teaching evaluation that are based on the use of video, detective stories, metaphors and participatory approaches (learning by doing). Presentation will include short demonstrations and discussion.
Here is a message from Sara Thompson, MN EA Program Chair:
"Are you unable to attend AEA but would love to hear an international speaker? Are you going to AEA and want more? We are pleased to host Alexey Kuzmin, president of the Process Consulting company, Moscow, on Monday, November 16th. The program will be held from 5-7pm at Kramaczuk's Deli.Kramaczuk's, which was founded in 1954 by a couple from the Ukraine, today offers a range of delicious foods -- sausages, cabbage rolls, goulash stew,borscht and other soups, varenyky, potato salad, pastries, etc."
The food should definitely be delicious.

Evaluating 'programs' that are not programs

2009-10-22T12:17:00.000+04:00

More and more often we face situations when we are invited to evaluate a 'program' that does not have essential features of a program (in particular does not have any program logic/theory), and which actually is not a program. Quite challenging. Under such circumstances we follow one the two possible strategies:
(1) program reconstruction - on the basis of what we can learn from the documents and people we reconstruct program logic and then evaluate the program with consideration of that logic,
(2) evaluation of whatever we have to evaluate - we accept the fact that the subject of evaluation is not a program and evaluate it accordingly. For instance, consulting services provided on an ad-hoc basis could hardly be considered a program. But can definitely be evaluated.

Partnerships Yield Program Evaluation: Methodology and Practice

2009-09-29T18:04:00.000+04:00

Here is an article from the AEA's Newsletter (Volume 9, Issue 9, September 2009): "Partnerships Yield Program Evaluation: Methodology and Practice"

AEA members Alexey Kuzmin, Rita O'Sullivan, and Natalia Kosheleva are co-editors of a new book that is the first of its kind in Russia. Program Evaluation: Methodology and Practice was published by Presto-RK and is the result of a project initiated and implemented by the International Program Evaluation Network (IPEN). The project was supported by International Research and Exchange Board (IREX) through its program "Dialogue" in Russia and funds were provided by USAID-Russia. Several AEA members contributed both as writers and editors."There are very few publications in Russian available for people interested in program evaluation in the Newly Independent States," says Kuzmin. "The Board of Trustees of IPEN included the book into the Network strategy a couple years ago. Since then we had series of conversations with IPEN members in different countries about their needs and with potential donors. By the time the book concept became clear we found a donor. It was good luck :)""Most rewarding," he adds, "was the collaborative nature of the project and the number of people and organizations that supported us in Russia and in the US. We have a very interesting combination of conceptual chapters, written by U.S. and local authors, and a good collection of case studies that are region specific. We wanted to partner with our American colleagues since we have some history of collaboration and one of the IPEN strategic priorities is international partnerships. "Four fellow AEA contributors include Michael Quinn Patton, Ross Conner, Tom Grayson and Jean Quesnel, along with Kuzmin, O'Sullivan and more than a dozen writers who are regional to the area. "This is the first program evaluation book of this quality and size in Russian. This is also the first Russian-American publication on evaluation in Russian," says Kuzmin. "As compared to other books I know (in English), our book provides a pretty comprehensive overview of what evaluation is and illustrates evaluation uses in various sectors including case studies from government, business, foundations and non-governmental organizations." Kuzmin, president of the Process Consulting Company, teaches at Moscow School of Social and Economic Sciences (MSSES). He has been working as a management and organization development consultant for more than 20 years and since the mid-90s has specialized in program and project evaluation. He has conducted more than 100 evaluations of projects and programs in Russia, CIS, Central and Eastern Europe. O'Sullivan, an associate professor at the University of North Caroline, specializes in collaborative evaluation techniques that enhance evaluation capacity and utilization among educators and public service providers. In 2004, she completed Practicing Evaluation: A Collaborative Approach, published by SAGE, and is senior author of Programs for At-Risk Students: A Guide to Evaluation published by Corwin Press in 1993. Kosheleva, an evaluation consultant with Process Consulting Company, has more than twenty years experience in the field of program and project development and management. She has conducted a number of evaluation projects in Russia, Tajikistan and Ukraine and published several articles on evaluation in Russia. The book is available online free of charge.

"Program evaluation: Methodology and practice" available online

2009-09-28T22:41:00.000+04:00

Our new book in Russian (400 pages) is now available online for free download.

International Evaluation Conference in Kiev, Ukraine

2009-09-21T09:22:00.001+04:00

International Conference"Methodology of Program Monitoring and Evaluation" will take place in Kiev this week (September 24-26). This will be the 9th annual conference of the International Program evaluation Network (IPEN). Conference program and information on the pre-conference workshops is avaialable in English. We expect over 100 participants. I attended all IPEN conferences since 2000 and found them very productive and empowering.

Monitoring and Evaluation of E-Governments

2009-09-12T22:55:00.000+04:00

Some time ago we conducted a study related to monitoring and evaluation of e-governments. It occured to me that I can share two documents in English developed on the basis of that study: "E-Government Checklist" (unpublished) and an article "Two dimensions of e-Government" published in the Russian edition of PC-Week. Hopefully, somebody will find these texts useful. Our report on the results of that study is available (only in Russian) here.

Use of training in evaluation capacity development checklist

2009-09-09T07:56:00.001+04:00

This checklist was one of the results of my PhD study completed in 2004. I am happy to share it with people interested in increasing effectiveness of evaluation training in evaluation capacity development. Based on research, the checklist can be used by practitioners.

"Organization paradigms and evaluation" - my presentation at the AEA conference in Orlando

2009-09-04T15:08:00.001+04:00

Here is what I am going to talk about at the Evaluation'09 conference in Orlando on Saturday, Nov 14, 11:50 AM to 12:35 PM, Sebastian L-1. Unfortunately my participation in the bloggers' roundtable will not be possible as it is sheduled for the same time :(
So, here is the brief description of my presentation.
Organizations are different. They have different “personalities” or organizational cultures, they are built and operate in different ways. According to Larry Constantine (1993) an organizational paradigm is “both a standard or model for an organization and a world view, a way to make sense of organizational reality”. Since evaluation should be built in the organizational reality, it’s important to identify and consider organization paradigm while choosing the most relevant evaluation approach for a particular program or an organization. In this presentation we shall explore the four organization paradigms described by Constantine (closed, synchronous, random, open) from an evaluator’s point of view and suggest how to design useful evaluations with consideration of organizational context.

random.org

2009-08-31T08:28:00.001+04:00

Internet resource that might be useful for evaluators and researchers. The website offers even a broader variety of applications: "RANDOM.ORG offers true random numbers to anyone on the Internet. The randomness comes from atmospheric noise, which for many purposes is better than the pseudo-random number algorithms typically used in computer programs. People use RANDOM.ORG for holding drawings, lotteries and sweepstakes, to drive games and gambling sites, for scientific applications and for art and music".

History of evaluation development in the Newly Independent States

2009-08-19T14:29:00.000+04:00

Back in 2005 UNCEF published a brochure on the new trends in development evaluation. There was an article on the evaluation development in the Newly Independent States included into that brochure. Although it's out of date it might be interesting to those looking for information on the history of evaluation development.

Participatory Training Evaluation Method (PATEM)

2009-08-12T08:46:00.000+04:00

The PATEM method might be useful for those who teach evaluation and /or conduct any kind of training and wish to actively involve participants in training evaluation. PATEM process nicely concludes the training. I will be happy to provide any additional information on how to use the method. Any feedback and comments on the PATEM use in your work will be appreciated.

Use of detective stories in teaching evaluation

2009-07-30T09:19:00.000+04:00

Smart detecives like Conan Doyle's Sherlock Holmes work pretty much like evaluators: to identify and find the criminal they collect data, analyze it, develop conclusions and recommendations. I find it very productive to use detective stories in teaching evaluation especially when we discuss data analysis. Sherlock Holmes helps a lot and makes it fun. Paper describing how I do that was presented at the AEA conference in Denver in 2008. In includes a handout for students. Please feel free to use it. I will appreciate feedback on how it works in your settings.

New evaluation book in Russian - success of a joint Russian-American project

2009-07-26T21:50:00.000+04:00

A new book "Program Evaluation: Methodology and Practice" has been published in Moscow. This is the first book of this size and quality in Russian. The book appeared as a result of a project initiated and implemented by the International Program Evaluation Network in collaboration with a few members of the American Evaluation Association. The project was supported by International Research and Exchange Board (IREX) through its program "Dialogue" in Russia. Funds were provided by USAID-Russia.

Kuzmin, A., O'Sullivan, R., Kosheleva, N. (Ed.). (2009).Program evaluation: Methodology and practice. Moscow, Russia: Presto-RK. (396 pages)

Program Evaluation: Methodology and Practice

Contents

Introduction

Foundations of Program Evaluation Methodology
What is a program – Kuzmin A.
Evaluation as a program management function - Kuzmin A.
Evaluation and audit: similarities and differences – by J.S. Quesnel

Evaluation at the design stage of the program - Kuzmin A.
Evaluation at the implementation stage of the program - Kuzmin A.
Use of logic models in program evaluation – by T. Grayson
Program monitoring - Kuzmin A.
Guiding principles for evaluators - Kuzmin A.

Making evaluation useful
Utilization-focused evaluation – by M.Q. Patton
Collaborative evaluation – by R. O’Sallivan

Evaluation practice in the governmental sector
Integrating evaluation in the practice of Russian government – Tsygankov D.
Evaluating efficiency of the programs promoting employment – Kartseva M.
Specifics of evaluation of federal and municipal programs – Makarova T.
Evaluation within the framework of the Krasnoyarsk region grant program – Vladyko L., Malitskaya E.

Evaluating corporate social programs
Evaluation of corporate social programs – Balakirev V., Kuzmin A.
Evaluation of RUSAL 100 Class Projects Program (case study) – Balakirev V.
Evaluation of Sakhalin Energy corporate social program (case study) – Tiushkevitch N., Mikhailova M., Lygina A.

Evaluation in foundations
Modern approaches to evaluating effectiveness of foundations – Palivoda L.
Evaluation use by donors working in Russia – Kaminarskaya N.
Development of a system of monitoring for the Program fighting tuberculosis in Donetsk region (case study) – Zabolotny A., Denisenko I., Soshenko I.

Evaluation in NGO sector
Program evaluation in NGO sector - Kuzmin A.
Development of evaluation capacity in Siberian Civic Initiative Support Center (case study) - Kuzmin A.

Evaluation as a professional field
Professionalization of evaluation – by J.S. Quesnel
On the way towards professionalization: creation of evaluation associations - Kuzmin A.
Facilitating Evaluation around the World: IOCE case study - by Ross Conner

Future outlook
Future of evaluation – by M.Q. Patton
Tendencies of program evaluation development in the Newly Independent States – Asanova J.

Author information

Evaluation Conference in Kiev, Ukraine

2009-07-26T21:25:00.000+04:00

International Program Evaluation Network (IPEN) will have its 9th annual conference in Kiev, Ukraine on September 24-26. Please attend the conference website (in English and Russian) for more details.

Are evaluators blogging?

2009-07-26T20:49:00.000+04:00

Just to begin with I added evaluation related blogs from the AEA's blog list and another source sent to me be a Moscow colleague yesterday. Since I was interested only in evaluation related blogs I excluded blogs with personal or non-evaluation content. All the blogs I selected (in English and Russian - these are the only two languages I understand) are now included into "My blog list" on this page. One thing is obvious: evaluators are not overly active in using blogs :( The total number of blogs is 18 (including 2 in Russian). Not too many. Only 4 blogs out of 17 were updated about a week ago or less. 7 blogs were updated 2-5 weeks ago. 4 blogs were updated 2-5 months ago. And 3 blogs were updated from 7 months to 1 year ago.
What shall we call an 'active' blog? Is it OK to have a blog that is updated once a year?

Getting connected

2009-07-26T19:39:00.000+04:00

Some evaluators use blogs. I also have a blog since 2007. Unfortunately, our blogs are not connected. I decided to create a new blog here and try to get connetced to blogging evaluators.