KPI 15 Extent ICF Intervention Lead Transformational Change
KPI 15 Extent ICF Intervention Lead Transformational Change
KPI 15 Extent ICF Intervention Lead Transformational Change
Acronyms ........................................................................................................................................ 3
Rationale ......................................................................................................................................... 4
Summary table ............................................................................................................................... 4
Technical Definition ....................................................................................................................... 5
Methodological Summary.............................................................................................................. 6
Methodology ................................................................................................................................... 7
Worked Example ......................................................................................................................... 10
Data Management ........................................................................................................................ 13
Data disaggregation ..................................................................................................................... 14
Annex 1: Why report on KPI 15 ................................................................................................. 14
Annex 2: Examples of indicators against each of the TC criteria ............................................ 15
Annex 3: Comparability and synergies with other external indicators ................................... 17
Annex 4: Definitions of key methodological terms used across Methodology Notes ............ 18
Annex 5: Optional KPI 15 Reporting Template ........................................................................ 18
Acronyms
The ICF will have greater impact if it can be ‘transformational’ by, for example, encouraging others to
replicate and scale-up successful activities and facilitating substantive institutional and policy change
toward a low carbon and climate resilient future. A challenge for this indicator is to capture these
different, often country-specific, dimensions of Transformational Change (TC), while remaining
sufficiently simple, to be unambiguous and pragmatic.
This indicator recognises that “transformation” is multi-dimensional, and that the indicator will not be
able to capture everything that, in time, may contribute to TC. Rather, the objective is to capture enough
evidence to form a reasonable qualitative picture of ICF effectiveness in this area.
Further information and a broader explanation of why one should report on KPI 15 is in Annex 1.
Summary table
Table 1: KPI 15 summary table
Units Box marking (i.e. 0, 1, 2, 3 or 4)
Disaggregation NA
summary
Headline data The self-assessment box marking (for each relevant criterion and an overall
to be reported marking) with explanatory text presenting evidence of transformation against
relevant criteria, both to justify the assessment and evaluate the reliability of the
evidence. See Annex 5 for an optional KPI 15 reporting template.
Latest revision September 2018.
Timing issues When to report: ICF programmes will be required to report ICF results once each
year in March. Please bear in mind how much time is needed to collect data
required to report ICF results and plan accordingly.
Reporting lags: Your programme may have produced results estimates earlier in
the year, for example during your programme’s Annual Review. It is acceptable to
provide these results as long as they were produced in the 12 months preceding
the March results commission. In some cases, data required for producing results
estimates will be available after the results were achieved – if because of this,
results estimates are only available more than a year away from when results are
delivered, this should be noted in the results return.
Links across KPI 15 supplements other IFC KPIs, which report on more tangible results (e.g.
KPI portfolio avoided GHG emissions or hectares of deforestation avoided) and gives an
indication of the likelihood that programme results will continue beyond the
funding period of the ICF programmes. For example, it could be deduced that
emissions reductions will continue beyond the end date of a GHG emissions
reduction programme, if the programme is deemed to be highly likely to be
transformational. This could be the result of a policy change or encouraging
others to replicate and scale up successful activities because of the ICF
programme.
Technical Definition
Transformational Change is ‘change which catalyses further changes’, enabling either, a shift from one
state to another (e.g. from conventional to lower carbon or more climate-resilient patterns of
development), or faster change (e.g. speeding up progress on cutting the rate of deforestation). However,
it can entail a range of simultaneous transformations to political power, social relations, decision-making
processes, equitable markets and technology.
Many of the transformations the ICF is seeking to bring about will only be evident after a period of time,
and most are unlikely to materialise within the period of ICF support. This indicator therefore tracks
early signs of transformation, or the extent to which ICF activities are being, or have a good likelihood of
being, transformational. It does so by using proxies for drivers of transformation, to assess the extent to
which ICF support can be linked, if not attributed, to likely Transformational Change.
These proxies (henceforth called the ‘criteria’, as set out in the ‘Methodology’ section) are based on the
Theory of Change (ToC) for Transformational Change (see ‘Rationale’ section).
The ICF is likely to be more transformational in developing countries if several of the following criteria
prevail, and if at least one criterion exists for each level of the Theory of Change (see Figure 1 below for
details):
• Political will and local ownership: Where the need for change is agreed locally, and the
process is locally owned. Where high-level political buy-in and broad support from across
societies, cultures, and interest groups enable widespread changes to patterns of development;
• Capacity and capability can be increased: Where a target country and target communities
have the capacities and capabilities necessary to bring about the change;
• Innovation: Where wider and sustained change comes from innovative new technologies with
the potential to demonstrate new ways of doing things;
• Evidence of effectiveness is shared: Where approaches which have proven successful in one
location are disseminated widely, and lessons on their usefulness are credible;
• Leverage / create incentives for others to act: Where the costs of climate action are
reduced to the point that acting on climate change risks and challenges is a sensible decision for
public agencies, commercial firms, and private individuals. These cost reductions may need to be
steep enough to overcome behavioural inertia;
• Replicable: Where good ideas piloted by the ICF are replicated by others in the same country,
and more widely;
• At scale: Where interventions (such as national, sectoral or regional programmes) have
sufficient reach to achieve progressive institutional and policy reform, or drive down the costs of
technology deployment;
• Sustainable: Where activities are likely to be sustained once ICF support ends.
Ultimately, many truly transformational changes will require a critical mass, to overcome political, market
and other sources of inertia. Many of the points above relate to achieving this critical mass and the more
of the above an intervention can promote, the greater the likelihood that it will lead to transformational
change.
The Theory of Change for Transformational Change below groups criteria at three different levels
(drivers, mechanism and enablers).
Further information and a broader explanation of why one should report on Key Performance Indicator
(KPI) 15 is in Annex 1.
Methodological Summary
This KPI is a qualitative process indicator. The expectation is that it will normally be assessed at the
level of a significant ICF programme, or a country/thematic portfolio, rather than for individual projects.
Expected results
A qualitative assessment of the type and nature of expected Transformational Change should be provided
at the start of the programme (or portfolio of programmes). This assessment should be guided by the
criteria included in the ‘Methodology’ section. It is not necessary to provide a box marking for the
expected result at this stage, the assumption being that this would be ‘4 – transformation judged very
likely’, since all ICF programmes should be designed to be transformational. A baseline assessment should
also be conducted, based on qualitative judgement assessment.
Actual results
ICF programme/portfolio managers should annually report:
• An overall assessment score of the likelihood that transformation linked to the ICF support
(programme, country, region or sector portfolio) as follows:
o 0: Transformational Change judged unlikely
o 1: No evidence yet available – too early to assess, but Transformational Change
expected
o 2: Some early evidence suggests Transformational Change judged likely
Transformational Change evidence will not be aggregated at the overall ICF level in the same way as
other ICF KPIs. Scores might be synthesised to identify patterns and trends to assess overall progress
and to tease out lessons, rather than to form a view on the ICF’s expected future global transformational
impact. This KPI therefore adopts a qualitative approach to monitoring (not measuring) likelihood of
transformation, relative to expected change.
A summary of key methodological steps, which are expanded in the methodology section further below,
is demonstrated in the following diagrams:
Where there is more than one ICF project programme, in a country, region or sector portfolio, the
assessment should be scored at the aggregate level, to reflect expected synergies (and reduce the risk of
double-counting).
Methodology
Expected Results
At the start of the programme or project, define:
• what successful transformation looks like for the programme/portfolio (including its Theory of
Change);
• the key stakeholders involved;
• which of the TC criteria are relevant to report against; and
• the programme/portfolio-specific indicators.
This step should list and very briefly describe – at impact and outcome levels and noting £ values – the
projects or programmes comprising the portfolio. This may be wider than just ICF programmes and
include other influencing activities.
This should not require extra analysis beyond the Strategic Cases of the main interventions but may need
amending if new projects are added to the portfolio, which address new issues. The baseline is based on a
qualitative judgement assessment.
3. Describe the Theory of Change that links the programme / portfolio activities and
the expected Transformation
Though this step will draw heavily on the ToC of the main interventions, it may require additional work
given that Transformational Change should sit above those ToC interventions. The project ToCs should
be nested within the overall Transformational Change ToC.
This step contextualises the UK support and allows a political economy analysis of the change to be
summarised. Other stakeholders could be considered: a) those whose engagement is a necessary pre-
condition for change; b) those who have been (or need to be) engaged during implementation; c) those
who are not essential, but whose engagement presents opportunities which can / have been made use of.
This may need amending as additional key players are identified during programme / portfolio
implementation.
This step has two purposes: (i) to set out what eventual impact is expected, and when it may be realised
(drawing on impact statements of the portfolio interventions); (ii) to set out the criteria and indicators
used to assess the likelihood of TC, drawing on relevant indicators and KPIs from project / programme
logframes.
Actual Results
At each reporting round, provide a narrative and assessment (score) of progress towards transformation.
At least one criterion should be included from each level of the Theory of Change for Transformational
Change (see Rationale section).
The categories are not intended to be of equal importance, and may not all be relevant in every case.
However, an absence of some (notably ‘political will’ and ‘capability and capacity’) are likely to be major
constraints on Transformational Change. ‘Replication’, though clearly important, is likely to be a later
stage indicator. In turn, ‘sustainability’ is likely to rely on changes to many of the other criteria to be a
truly Transformational Change.
Weightings should be applied to the individual criteria depending on how many are present at each level
of the Theory of Change (i.e. apply a lower weighting if there are multiple criteria from the same level of
the Theory of Change). For example, a programme with four criteria relevant for KPI 15 might have one
criteria weighted at 30% from the TOC level: Enabler (e.g. Sustainable), one criteria weighted at 30%
from the TOC level: Mechanism (e.g. At Scale) and two criteria weighted at 20% from the TOC level:
Drivers (e.g. Innovation and Evidence of Effectiveness).
This should draw on the programmes/projects logframe and other relevant ICF KPIs if appropriate, for
each of the selected criteria. If not, then they should be formulated at the time a baseline is set for the
intervention’s expected Transformational Change.
Examples of useful metrics for each criterion can be found in the table in Annex 2 and in the “KPI 15
Discussion Paper on How Programme Managers Can Communicate Case Studies and Evidence of
Progress Towards Transformational Change”, which is a supplementary document to this KPI
methodology note.
Please be aware that though these two sources suggest the types of evidence that could be used to
assess each criterion, programme managers should treat these as a guide and think carefully about what
types of evidence are most relevant to their particular programme and local circumstances. This is
important given that the barriers to systemic change are often hierarchical or local, or specific to
particular sectors.
Using the programmes/projects’ logframes, and other relevant ICF KPIs if appropriate, collect data against
each of the metrics.
Examples can be found in the “KPI 15 Discussion Paper on How Programme Managers Can
Communicate Case Studies and Evidence of Progress Towards Transformational Change”, which is a
supplementary document to this indicator.
The reporting template for KPI 15 (found in Annex 5) might be helpful to guide reporting. This step is
optional, and alternate reporting approaches may be used.
Assign an overall KPI 15 score, based on the weighted mean of the respective individual criteria.
Note that where there is more than one ICF project in a country, region or sector portfolio, the
assessment should be scored at the aggregate level, to reflect expected synergies (and reduce the risk of
double-counting):
As far as possible, reporting should be at the level of a significant programme or country (or similar)
portfolio, to help ensure that the links between different activities are understood, and an assessment
made of the likelihood that a critical mass of support for change is emerging.
This indicator seeks to track the transformational impact of HMG climate change “activities”. Though the
bulk of these will involve bilateral funding through the ICF, it will be important to recognise the role of
wider influencing and policy support provided by HMG. The contributions of others to the likely
Transformational Change – notably national and decentralized governments, but also other domestic and
international donors and organisations – should also be recorded as part of expected and actual results.
The methodology acknowledges that some ICF activities may inadvertently have an adverse effect on
Transformational Change (pilots might go wrong and undermine the case/support for change;
interventions may build capacity in one area by denuding it in another, etc.). It will be important that the
evidence presented is balanced, and also that any such negative influences are reported on.
Worked Example
Worked example 1
Based on a fictitious programme that provides financing for energy efficiency and small-scale renewable
energy projects in developing countries.
Relevant proxy criteria included: evidence of effectiveness shared; capacity & capability increased;
replicability; scale; sustainability.
3. Assign metrics to each of the selected criteria; 4. Collect criteria data; 5. Score each
criteria individually, based on data; 6. Fill in KPI 15 reporting template; and 7. Assign
overall KPI 15 score – these results of these steps are shown in the optional reporting
template below
The following metrics were assigned against each TC criteria based on field evidence, and data was
reported and scored as below. All scores were based on evidence against the metrics:
“Energy Efficiency and Renewable Energy for SMEs and Households” (EERESH)
Programme/Project summary:
Based on a fictitious programme that provides financing for energy efficiency and small-scale renewable
energy projects in developing countries. The programme aims to increase the flow of finance to small
and medium enterprises (SMEs) and households by creating a new partnership with the private sector,
specifically investing into a fund to leverage greater amounts of private finance. The fund tackles the
common barrier of institutional finance not being readily available to SMEs and home-based micro-
enterprise for low carbon projects in developing economies.
Metrics:
Number of new website visitors deemed an appropriate measure of whether evidence of effectiveness
is shared. Sign-up / log-in system helps track whether relevant users have accessed the website.
Number of activities (e.g. workshops, key publications) delivered to disseminate programme
information is a standard measure of whether effectiveness is shared.
Metric Data
Number of new website visitors 2,000
Number of activities (e.g. workshops, key 20
publications) delivered to disseminate
programme information
Justification of score:
A score of 3 (Tentative evidence of change – Transformational Change judged likely) was awarded, as
based on higher than average number of visitors to a programme’s website and a slightly higher than
average number of activities to disseminate programme information for a programme of this size.
Metrics:
Metrics relating to Technical Assistance (TA) selected as most relevant to capacity building.
Metric Data
Total number of TA programmes approved 100
since inception
Total amount of TA funding approved $3.82m
Total number of people trained through the Over 500 per participating country (mean
TA Fund average)
Justification of score:
A score of 2 (Some early evidence suggests Transformational Change judged likely) was awarded, as a
good level of TA programmes were approved but the total funding for each TA programme was
relatively low and more people could be trained via the TA fund.
Metrics:
The potential for replicability assessed on whether low carbon loans developed beyond the
programme and whether the programme led to increased institutional knowledge of low carbon
investments, which has significant potential to instigate replicability.
Metric Data
Have the Financial Institutions developed Yes
low carbon loans beyond the programme
Did the programme lead to increased Yes, to some extent
institutional knowledge of low carbon
investments
Justification of score:
A score of 3 (Tentative evidence of change – Transformational Change judged likely) was awarded. A
maximum score of 4 would have been awarded if it was reported that the programme had
categorically led to increased institutional knowledge of low carbon investments.
Metrics:
The potential for scalability assessed not only via total number of individual sub-loans disbursed but
also by geographic spread and total number of financial institutions receiving investment.
Metric Data
Total number of financial institutions 30
receiving investment (loans and direct
investments)
Total number of countries receiving 25
investment
Total number of individual sub-loans 20,000
disbursed
Justification of score:
A score of 4 (Clear evidence of change) was awarded, as many developing countries and financial
institutions were reached.
Metrics:
Total number of financial institutions expected to extend their programme to new MSME recipients
was deemed the most appropriate proxy of EERESH’s sustainability.
Metric Data
Total number of financial institutions 79%
expected to extend their programme to
new MSME recipients
Justification of score:
A score of 4 (Clear evidence of change – Transformational Change judged very likely) was awarded, as
over three quarters of the financial institutions expected to extend their programme to new MSME
recipients.
Overall score
Overall EERESH scored a weighted mean score of 3 based on equal weightings of the
above criteria.
Real-life examples can be found in the “KPI 15 Discussion Paper on How Programme Managers Can
Communicate Case Studies and Evidence of Progress Towards Transformational Change”, which is an
internal HMG document, and supplements this indicator.
Data Management
Data Sources
Some data will be available directly from programmes, for example from project-level M&E. Ideally, the
duty to collect data should be the responsibility of recipients of ICF funding, or a third-party auditing
entity. This information will need to be kept up to date by liaising with programme managers.
Quality Assurance
All results estimates should be quality assured before they are submitted during the annual ICF results
return, ideally at each stage data is received or manipulated. For example, if data is provided by partners,
this data should be interrogated by the ICF programme team for accuracy, or are the very least data
should be sense checked for plausibility. When converting any provided data into KPI results data, quality
assurance should be undertaken by someone suitable and not directly involved in the reporting
programme. Suitable persons vary by department; this could be an analyst, a results / stats / climate and
environment adviser / economist.
Central ICF analysts will quality assure results that are submitted and this may lead to follow up requests
during this stage.
To avoid inherent reporting biases, it is strongly recommended that, where possible, data collection is
undertaken by a third party that is not directly involved with implementing the project. Where not
possible, consider using independent evaluations or alternative means to periodically check the validity of
results claims.
Any concerns about data quality or other concerns should be raised with your departmental ICF analysts
and recorded in documentation related to your results return.
Data disaggregation
NA
Transformations are hard to predict, multi-faceted, multi-causative, and take time to unfold. For these
reasons, monitoring transformation is inherently difficult, and this KPI is complex. It is implicitly linked to
other KPIs and embedded within programme design. It is therefore less important for accountability as to
the purpose and ‘big-picture’ thinking of projects. However, it is not easily aggregated across the
portfolio, or connected to more quantitative metrics such as Value for Money (VfM).
The purpose of monitoring Transformational Change via KPI 15 is not so that a programme can be
completed and report “transformation achieved”. Rather, the purpose is to encourage thinking and
programme design toward the sorts of transformations needed – to ‘design in’ learning from UK ICF
programmes as they are implemented.
The transformations the ICF seeks to bring about will only be evident after a period of time. Though it
will be necessary to monitor these longer-term changes, they are unlikely to materialise within the ICF
programme, typically up to 5 years. This indicator therefore tracks early signs of transformation, or the
extent to which key ICF activities either are being, or have a good likelihood of being, “transformational”.
It does so by using proxies for drivers of transformation, to assess the extent to which ICF support can
be linked, if not attributed, to likely Transformational Change. Proxies include political will and local
ownership; capacity and capability; innovation; sharing of effectiveness; leverage / incentives for others to
act; replicability; scalability and sustainability.
Applied appropriately, KPI 15 can provide more than a simple accounting function. KPI 15 can help ICF
programme developers and managers think through what is needed for a programme to catalyse deeper,
faster and/or wider changes in a particular sector, area or institutional context. Within these contexts, it
is essential to establish the set of conditions that should catalyse or at least contribute to the particular
Transformational Change sought.
Within this context, KPI 15 as it is formulated and presented in the Methodology Note, above, provides
insights into progress made against Transformational Change.
* These measures could equally fit under the ‘leverage/ incentives for others to
act’ criterion. Which one the programme manager chooses to put them under
will depend on what elements of the generic theory of change are most relevant
to the portfolio in question.
At Scale Ideally this will be a quantitative assessment of resources mobilised relative to the
assessed funding amount necessary to effect the desired change. It will be
location and context-specific.
Such measures may well draw on other criteria and could include:
• Proportion of population at risk whose climate adaptive resilience is judged to
have been markedly improved [drawing on other relevant KPIs]
• X% of infrastructure at risk built to higher standard [e.g. X% of roads
constructed or up-graded to cope with a climate-induced 1 in 5-10 year rain
storm]
• A particular renewable technology accounts for X% of market share
• X% of potential farmers are able to access a particular improved seed variety,
or Y% of farmers have been trained in new flood or drought-adaptive or lower
carbon practices
Sustainable A view on the likely sustainability of ICF-funded activities could comprise a
[Activities are synthesis of the evidence presented on each of the indicators listed above (and
likely to be should certainly draw on the other criteria).
sustained once Where relevant, other evidence should be included in this assessment [defined by
HMG funding programme or project].
ends] Such measures could include:
• Local government representatives paid to continue promotion of climate
risk management measures in farming practices/watershed management
• Target community assumes responsibility for management of solar/wind
projects e.g. within a decentralised 5-year plan
PROGRAMME/PROJECT NAME
Programme/Project summary:
Metrics:
[insert short narrative introducing relevant metrics below (delete / add extra rows as appropriate –
note that at least one criteria should be selected from each level of the KPI 15 Theory of Change,
which can be found above in the Methodology Note). Narrative on metrics should provide an
explanation as to why the metrics have been selected within the context of the
programme/fund/country level intervention]:
Justification of score:
A score of [insert score] was awarded, as [insert narrative justifying score and a qualitative judgement
on the strength of evidence].
Metrics:
[insert short narrative introducing relevant metrics below (delete / add extra rows as appropriate –
note that at least one criteria should be selected from each level of the KPI 15 TOC, which can be
found in the Methodology Note). Narrative on metrics should provide an explanation as to why the
metrics have been selected within the context of the programme/fund]:
Justification of score:
A score of [insert score] was awarded, as [insert narrative justifying score and a qualitative judgement
on the strength of evidence].
*Repeat step for as many Transformational Change criteria that are relevant
Overall score
Overall [insert programme name] scored a weighted mean score of [insert score and note on
criteria weightings].
1
E.g. Drivers, Mechanism, Enablers
2
E.g. Drivers, Mechanism, Enablers
Photo credits
Photos used in this KPI guidance note series were sourced from two websites, Climate Visuals and Unsplash (except the
photo for KPI 14 which belongs to IMC Worldwide). They are available for use under a Creative Commons license, which
enables organisations provided that photographers are credited. Photographers for this KPI guidance note series are
credited below.