CASE STUDY FOR ORGANIZATIONAL DEVELOPMENT

The Impact of Using One Appreciative Question to

Evaluate a City Health Department

Presented by
EnCompass LLC

AEA Annual Conference, Orlando, FL

Wednesday, Nov 11, 2009

This case example illustrates a situation where the evaluation team was only able to add one appreciative question
to an already developed interview guide that included other non-AI questions.

Background

The District of Columbia Department of Health (DC DOH) commissioned an evaluation to determine what
improvements were needed to help the DC DOH take on its emerging role of moving out of direct health service
delivery and into a role of planning, policy development, and quality assurance. Soon after developing an interview
guide using non-AI questions, the evaluation (made up of evaluators and department staff) learned about
Appreciative Inquiry and decided to add the following appreciative question to the beginning of the interviews with
DC DOH staff: Tell me about a time in the last year when you have been particularly proud of the work you are
doing – a time when you felt most committed to the DC DOH. The remainder of the interview guide included
questions about the problems that staff members were facing, their level of compliance with job descriptions and
standards, their ideas of how various processes could be redesigned, and the systems that needed to be developed.

Conducting the Interviews

Once the evaluations were completed, the 15 evaluators who had conducted a total of approximately 60
individual interviews reported how the one appreciative question had changed the tone of the interviews and the
usefulness of the resulting data. The evaluators admitted that they had gone into the interviews somewhat critical of
the client as a result of the D.C. government’s recent takeover by an independent board. As a result, the city
employees’ morale was extremely low. The appreciative question that sought information about interviewees’
successes resulted in the DC DOH staff sharing their own passion for their clients (D.C. residents), and the ways in
which they had gone above and beyond the call of duty to make DC DOH work in the difficult times. This not only
energized the DC DOH staff, but hearing their stories enabled the interviewers to see the staff in a more positive
light, which resulted in their resolved to contribute constructively to DC DOH’s future through this evaluation.

Benefits of Using Appreciative Inquiry

This example also demonstrates that one not need employ the full AI process, nor ask many appreciative
questions to benefit from an application of Appreciative Inquiry for gathering important and useful information. At
a time when the D.C. city services were under attach across departments, starting the interviews with a study of
exceptional service by DC DOH staff, respondents were able to reveal some of the critical strengths in the system,
and how they had employed those to provide exceptional service. The department staff that conducted the
interviews began with a negative bias about the city services. As a result of conducting the interviews, however, this
perception shifted dramatically. Using the appreciative question helped them hear information in a way that non-AI
questions would not have been able to ask or address.

1
 CASE STUDY FOR SYSTEMS THINKING

U.S. National Academy of Sciences

The African Science Academy Development Initiative (ASADI)
EnCompass LLC
AEA Annual Conference, Orlando, FL
Wednesday, Nov 11, 2009

The African Science Academy Development Initiative (ASADI) is a ten-year program of the US National Academy
of Sciences (US NAS) with funding from the Bill & Melinda Gates Foundation to engage with African science
academies in a collaborative relationship to promote evidence-based decision making and policy formulation.
The ASADI’s principle goal is to advance the ability of African academies of science to provide independent,
evidence-based policy advice to their governments by achieving a series of benchmarks. The benchmarks include:
(1) infrastructure and internal capacity, (2) convening activities, (3) enhancement of and or clarity of membership,
(4) governance functions, and (5) greater financial sustainability.
EnCompass LLC and its partner OtherWISE: Research and Evaluation conducted a mid-term evaluation to assess
the African science academies’ movement towards playing a significant role in influencing health and health-related
program and policy decision-making. The evaluation provides in-depth information on the three primary partner
academies in Uganda, Nigeria and South Africa. Based on the findings of the mid-term evaluation and ongoing
reflection on the part of the US NAS staff, the following conclusions were drawn:
 All three academies produced high output in terms of convening activities and reports
 The importance of partnerships and relationships emerged as a key factor of success. Of critical importance was
the quality of the relationship between the African academies and US NAS, between the African academies
during the learning collaborative, and between the academies and their governments and policy stakeholders.
 Capacity building required more than just technical assistance, but strong communications, and an
understanding of cultural contexts. Convening meetings and forums are only one piece of capacity building,
and in the end, the ASADI team learned a great deal about the cultural context and existing way administrative
and policy decision are made in each country.
 The changing composition of memberships and their role in the academies as “service” organizations was
recognized as a critical factor, and ASADI turned its attention and made important investments in building
relationships with members, and engaging in knowledge transfer.
 Academies exerted substantial efforts in communications and outreach, as a way to position them for policy
advising and in using research to advocate for evidence-based policy making. Academies should now consider
developing a communications strategy to enhance academies’ role as service organizations.
 As noted in Uganda, progress depends on attending to needs in different areas simultaneously. That is,
infrastructure building is linked to increased communications/outreach which in turn raised the academies’
profile and convening power and contributed to policy advising activities and greater sustainability. ASADI
increasingly worked on different aspects of the whole system simultaneously, which was an effective strategy.
Future lines of inquiry could include consideration of how the cultural context in the three countries influence their
emerging model of policy advising.

CASE STUDY FOR ORGANIZATIONAL DEVELOPMENT

2
 Using Appreciative Inquiry to Transform Evaluative Thinking and Build
Evaluation Capacity
“The Albania Experience”

Presented by
EnCompass LLC

AEA Annual Conference, Orlando, FL

Wednesday, Nov 11, 2009

Since 2003, EnCompass has supported the Coordinated Action Against Human Trafficking (CAAHT) program in
Albania in its innovative work to strengthen civil society and increase coordination of anti-trafficking work in
Albania. EnCompass has provided mentoring, coaching and capacity building to both CAAHT staff and local
NGOs in monitoring and evaluation, knowledge management and participatory learning using Appreciative Inquiry
techniques and approaches. This approach has transformed the attitudes and relationships within the anti-
trafficking community of Albania and contributed to a new level of cooperation between civil society and
government actors.

Description of Methodologies Employed and Activities Undertaken:

In the first three years of the project, EnCompass supported the CAAHT initiative in its efforts to enhance
collaboration among local government and civil society stakeholders to improve programming and services to
prevent, protect and assist in the reintegration of trafficking victims into their communities. EnCompass designed
and facilitated three annual conferences with key governmental and non-governmental stakeholders and conducted
a participatory evaluation of the coordination of anti-trafficking work nationwide. In the last three years of the
project, EnCompass’ role was expanded to include training, coaching and capacity building in monitoring and
evaluation using a participatory approach to incorporate appreciative evaluation. Two training workshops were
conducted with key governmental and NGO partners to establish a management information system and ongoing
coaching used to support implementation of the system to improve program management and planning. The final
phase focuses on impact assessment of the anti-trafficking work and factors that contribute to sustainability.

Description of Accomplishments:
EnCompass’ contributions to the CAAHT initiative has supported its work as one of the most innovative programs
in anti-trafficking. The evaluation of the coordination component conducted by EnCompass gave solid evidence of
the increased collaboration between government and civil society which resulted in the establishment of Regional
Anti-trafficking Committees in each prefecture. The use of a participatory approach to monitoring and evaluation
and the introduction of Appreciative Inquiry techniques transformed the thinking about how information can be
used to improve program management and planning and contribute to a learning community in anti-trafficking.
Systematic coaching and capacity building has also strengthened the technical ability of both the CAAHT staff and
NGO partners in using effective questions and Appreciative Inquiry in deepening the evaluative process to really
examine substantive issues and processes that can contribute to the sustainability of anti-trafficking work in
Albania.

3
 Evaluation activity AI/Systems tools used/how Results of use Insights Challenges

Strongly and clearly Evaluating anti- Transitioning

Used a participatory approach to design documented the power of trafficking work from a very
and implement a survey, based on AI cooperation and through the "lens" of numerical PMP to
Assessment of the techniques, of key stakeholders on the coordination to motivate appreciative inquiry a rich,
Coordination Component benefits of CAAHT sponsored coordination social action and enable contributed to a appreciative
activities on the anti-trafficking community synergistic activity on sense of inquiry approach
in Albania trafficking issues empowerment in civil in evaluation
society

Resulted in an improved AI process helped to Lack of

Conducted two workshop with CAAHT staff
Development of a M&E system with indicators reframe M&E as a evaluation
and partners to introduce AI concepts and
utilization focused M&E that are more directly tool for learning and capacity among
used AI to focus the design of a utilization-
system linked to project goals and program planning local partners
focused M&E system
outcomes and management

Transformed the way the AI and systems Sustaining the

partners are processing and approach to capacity building
Ongoing coaching and mentoring of both using M&E data, and evaluation changed after completion
CAAHT staff and NGO partners on core contributed to a new level the relationship of the CAAHT
evaluative functions such as asking of collaboration and among key project will be a
Building evaluation capacity evaluative questions, collecting and coordination within the stakeholders and challenge
analyzing data, preparing reports and using anti-trafficking community made it possible to
M&E data for program planning and create new systems
management that could address
trafficking issues in a
more effective way