Discussion

1. Discuss the OECD five six key evaluation criteria showing how they
contribute to evaluation understanding. [25]

NOV 2019 Q1 / MAY/JUNE 2019 Q6 / MAY/JUNE 2018 Q6

Define each of the following terms as they are used in development evaluation;

(a) Relevance (5 marks)

(b) Effectiveness (5 marks)

(c) Efficiency (5 marks)

(d) Impact (5 marks)

(e) Sustainability (5 marks)

(f) COHERENCE (5 MARKS)

MAY/JUNE 2016 Q6

RE-ICES

(a)Relevance (is the intervention doing the right things)

• The extent to which the intervention objectives and design respond to beneficiaries’, global, country,
and partner/institution needs, policies, and priorities, and continue to do so if circumstances change.

To what extent are the objectives of the program still valid?

Are the activities and outputs of the program consistent with the overall goal and the attainment of
its objectives?

Are the activities and outputs of the program consistent with the intended impacts and effects?

Evaluating relevance helps users to understand if an intervention is doing the right thing. It allows
evaluators to assess how clearly an intervention’s goals and implementation are aligned with
beneficiary and stakeholder needs, and the priorities underpinning the intervention. It investigates if
target stakeholders view the intervention as useful and valuable.

Relevance is a pertinent consideration across the programme or policy cycle from design to
implementation. Relevance can also be considered in relation to global goals such as the Sustainable
Development Goals (SDGs)

(b)Effectiveness (Is the intervention achieving its objectives?)

• Effectiveness is a measure of the extent to which a development programme or project achieves the
specific objectives it set. If, for example, we set out to improve the qualifications of all the high
school teachers in a particular area, did we succeed?

The extent to which the intervention achieved, or is expected to achieve, its objectives, and its results,
including any differential results across groups.

Effectiveness should analyse progress towards objectives along the results chain /causal pathway.

• In contrast to impact, which looks at higher-order effects and broader changes to which an
intervention may be contributing, effectiveness is concerned with more closely attributable results

• “Results” and “differential results” to open the door to asking important questions about the

distribution of results across different groups, and to look beyond intended objectives.

• This is in line with the policy priority to leave no-one behind.

• It encourages evaluators to examine equity issues and results for groups that have been marginalized,

while not assuming that equity is an objective of the intervention.

(c)Efficiency (how well are resources being used?)

Efficiency tells you that the input into the work is appropriate in terms of the output. This could be
input in terms of money, time, staff, equipment and so on. When you run a project and are concerned
about its replicability or about going to scale, then it is very important to get the efficiency element
right.
The extent to which the intervention delivers, or is likely to deliver, results in an economic and timely
way.

• “Economic” is the conversion of inputs (funds, expertise, natural resources, time, etc.) into outputs,
outcomes and impacts, in the most cost-effective way possible, as compared to feasible alternatives
in the context.

• “Timely” delivery is within the intended timeframe, or a timeframe reasonably adjusted to the
demands of the evolving context.

• This may include assessing operational efficiency (how well the intervention was managed).

• It is important to know the cost of program outcomes - whether the benefits justify costs

• Assessing for efficiency is a very technical exercise.

TWO TYPES OF EFFICIENCY ANALYSIS

Cost Benefit Analysis

• In this analysis, the outcomes of programs are expressed in monetary terms e.g compare amount
spent on anti-smoking program versus amount spent on healthcare for patients with smoking related
disorders, days lost from work etc.

• Cost effectiveness analysis

• Outcomes are expressed in substantive terms

• E.g. this evaluation would calculate the amount spent in converting each individual from a smoker
into a non-smoker

(d) Impact (what difference does the intervention make?)

• Impact tells you whether or not what you did made a difference to the problem situation you were
trying to address. In other words, was your strategy useful?

• The extent to which the intervention has generated or is expected to generate significant positive or
negative, intended or unintended, higher-level effects.

• Impact addresses the ultimate significance and potentially transformative effects of the intervention.

• It seeks to identify social, environmental and economic effects of the intervention that are longer
term or broader in scope than those already captured under the effectiveness criterion.

• Did ensuring that teachers were better qualified improve the pass rate in the final year of school?
Before you decide to get bigger, or to replicate the project elsewhere, you need to be sure that what
you are doing makes sense in terms of the impact you want to achieve.

(e)Sustainability (will the benefits last?)

• Sustainability is concerned with measuring whether the benefits of an activity are likely to continue
after donor funding has been withdrawn.
• The extent to which the net benefits of the intervention continue, or are likely to continue.
Considering the exiting plan

• “The continuation of benefits from a development intervention after major development assistance
has been completed.

• The probability of continued long-term benefits. The resilience to risk of the net benefit flows over
time.”

• Includes an examination of the financial, economic, social, environmental, and institutional

capacities of the systems needed to sustain net benefits over time.

– Design robustness

– Financial considerations

– Socio-cultural compatibility

– Environmental considerations

– Community Ownership and participation

– Exit strategy and capacity development

– Technological considerations

(f) Coherence: (How well does the intervention fit?)

The compatibility of the intervention with other interventions in a country, sector or institution.

The extent to which other interventions (particularly policies) support or undermine the intervention
and vice versa. This includes internal coherence and external coherence.

Internal coherence addresses the synergies and interlinkages between the intervention and other
interventions carried out by the same institution/government, as well as the consistency of the
intervention with the relevant international norms and standards to which that institution/government
adheres.

Internal policy coherence considers two factors: the alignment with the wider policy frameworks of
the institutions; and the alignment with other interventions implemented by the institution including
those of other departments responsible for implementing development interventions or interventions
which may affect the same operating context.1 It should consider how harmonised these activities are,
if duplication of effort and activities occurs, and if the interventions complement each other.

External coherence considers the consistency of the intervention with other actors’ interventions in
the same context. This includes complementarity, harmonisation and co-ordination with others, and
the extent to which the intervention is adding value while avoiding duplication of effort.