M&E Training Manual

A Training Manual
on
Monitoring and Evaluation
Concepts, Tools and Strategies
for
Social Sector Programmes
Tools Series II
Ministry of Social Development
Monitoring & Evaluation Division
September 2003
TABLE OF CONTENTS
PREFACE ...................................................................................................................................... 1
INTRODUCTION ........................................................................................................................... 2
What is the Purpose of the Training Workshop? .................................................................... 3

What are the Expected Benefits? ............................................................................................ 3
The Training Programme ....................................................................................................... 4
MODULE 1: DEFINING THE CONCEPTS OF MONITORING AND EVALUATION................ 5
Origins of Programme Evaluation.......................................................................................... 5

What is Evaluation?................................................................................................................ 5
Need for Reliable Data ........................................................................................................... 6
Key Evaluation Issues ............................................................................................................. 7
Evaluation versus Social Science Research............................................................................ 8
What is Monitoring? ............................................................................................................... 8
Steps in a Monitoring Process ................................................................................................ 8
Role of Performance Indicators in Monitoring ...................................................................... 9
MODULE 2: PROGRAMME PLANNING AND DESIGN........................................................... 10
Needs Assessment.................................................................................................................. 10
Baseline Assessment.............................................................................................................. 11
Conceptual Model as an Integral Part of Programme Design............................................. 11
Establishing a Programme Theory ....................................................................................... 12
Developing a Logical Framework ........................................................................................ 12
Defining Goals and Objectives ............................................................................................. 13
Impact Objectives.................................................................................................................. 14
Choosing Activities ............................................................................................................... 15
Identifying Indicators ............................................................................................................ 15
Process Indicators................................................................................................................. 15
Results Indicators.................................................................................................................. 16
Criteria for Selecting Indicators ........................................................................................... 16
MODULE 3: PLANNING A USEFUL EVALUATION ................................................................ 19
The Evaluation Process ........................................................................................................ 19

Types of Evaluation............................................................................................................... 19
Cost-Benefit Analysis ............................................................................................................ 23
Cost-Effectiveness................................................................................................................. 23
Comparing the two Concepts................................................................................................ 24
Evaluation Assessment.......................................................................................................... 26
Scope of the Evaluation......................................................................................................... 29
MODULE 4: EVALUATION DESIGN ........................................................................................ 31
Experimental Design............................................................................................................. 31
Non Experimental Design .................................................................................................. 31
Pre Test and Post Test Design ........................................................................................ 32
Static Group Comparison ..................................................................................................... 32
Quasi Experimental Design ............................................................................................... 32
Ministry of Social Development/Monitoring and Evaluation Division September 2003
Time Series Design................................................................................................................ 32

Issues to Consider when Selecting an Evaluation Design .................................................... 33
Other Evaluation Tools and Methods ................................................................................... 34
Sampling................................................................................................................................ 35
The Probability Sample......................................................................................................... 35
NonProbability Sample ....................................................................................................... 36
Sample Size ........................................................................................................................... 36
Confidence Level................................................................................................................... 37
MODULE 5: DATA COLLECTION AND ANALYSIS................................................................ 38
Quantitative Data.................................................................................................................. 38
Qualitative Data.................................................................................................................... 38
Tips for Quantitative and Qualitative Data.......................................................................... 39
Coding................................................................................................................................... 40
Data Analysis ........................................................................................................................ 40
Analysing Qualitative Data................................................................................................... 40
Analysing Quantitative Data................................................................................................. 40
Preparing an Evaluation Report ........................................................................................... 42
Developing a Solutions Framework...................................................................................... 42
ii
Preface
This training manual was developed as a reference document for an introductory training course
in Monitoring and Evaluation Concepts, Tools and Strategies for Social Sector Programmes. It
is intended that the course material covered will assist in building monitoring and evaluation
capacity in the social sector, through the exposure of participants to five modules of training and
the transfer of knowledge by these persons to others in their respective Ministries.
This publication is the second in a series of Tools being developed by the Monitoring and
Evaluation Division of the Ministry of Social Development, with the support of the Evaluation
Advisor seconded to the Ministry through The Commonwealth Fund for Technical Cooperation
(C.F.T.C.).
Permanent Secretary
Ministry of Social Development
A Training Manual on Monitoring and Evaluation Concepts, Tools and

Strategies for Social Sector Programmes
INTRODUCTION
Most of you must have heard of, or participated in, an evaluation exercise. You may even have
initiated an evaluation by carrying it out using internal resources or by contracting a consultant to
do it. As managers of social sector programmes, you will, in the course of your career, be called
upon to make decisions about the viability of a particular programme, or to compare the relative
benefits of various programmes. When this happens, you would be required to support whatever
decision you make with concrete facts and you will realise that evaluation can be a useful tool to
achieve your objective. If you are not too familiar with evaluation, you may begin to wonder
why there is a need for it.
In their seminal book on evaluation, Peter Rossi et al, argued limited resources for social
programmes in every country, including the United States, make it critical that such investments
yield demonstrable and proportionate social benefits1
Evaluation is also undertaken to strengthen governance and institutional development. Findings
from evaluations help to ascertain the extent to which ongoing programmes are (i) achieving the
objectives for which they were designed; (ii) responding to the actual needs of the beneficiaries;
(iii) conforming to existing policies and procedures; and (iv) contributing to institutional learning
and capacity development.
Evaluation has become a key component of results-based and performance-based management.
Findings from evaluations are used to initiate policy and programme changes. Also, they can
contribute to organisational learning through constant exchange of information among key
stakeholders.
Other reasons for undertaking an evaluation are:
1
2
to help in determining whether or not a programme should be continued, improved,

expanded or discontinued;
to assess the usefulness of a new initiative;
to increase the effectiveness of programme management;
to satisfy the accountability requirements of sponsors; and
to contribute to knowledge in social science. 2
Peter Rossi et al, Evaluation. A Systematic Approach, 6th Edition, 1999, pp6-7
Ibid
What is the Purpose of the Training Workshop?

The immediate objective is to introduce participants to foundational notions of monitoring and
evaluation. It is hoped that participants would, by the end of this training workshop, have
enhanced their capacity to collect, analyse, and use data to make decisions/judgments about the
effectiveness and appropriateness of social interventions.
The medium-term objective is to facilitate the development of an evaluation culture. By this it is
implied that monitoring and evaluation would be seen as a key component in the delivery of
social sector interventions and that there would be a formal process within Ministries to support
the undertaking of evaluations on a regular basis. Above all, decisions concerning social
interventions would be based on credible data.
What are the Expected Benefits?
At the end of the training workshop, it is anticipated that participants would have acquired a
good understanding of Monitoring and Evaluation (M&E) concepts, tools and strategies. They
would have acquired some basic knowledge about programme design, especially how to define
SMART objectives and how to identify indicators that are appropriate. They would have had a
good understanding of data collection methods, data analysis and preparation of final evaluation
reports.
A good grasp of the M&E concepts and tools should enable programme managers to interact
effectively with external consultants who are contracted to undertake evaluations. If consultants
are recruited to undertake an evaluation, programme managers should be able to discuss on equal
footing and be abreast of the issues to be evaluated. They should have a good grasp of the
methods that the consultants propose to use for data collection and analysis. This is important
for quality control of the final product. Also, it is anticipated that participants would appreciate
the use of evaluation findings to foster organisational learning, transparency and accountability.
Feedback from evaluations could be used to improve existing interventions or to design new
ones. Evaluation findings can help to reposition a Ministry by using the information from
evaluation to develop a new strategic plan or simply to bargain for additional resources.
The Training Programme

The training programme is based on a modular concept. There will be five main modules:
Module 1: Defining the Concepts of Monitoring and Evaluation
Module 2: Programme Planning and Design
Module 3: Planning a Useful Evaluation
Module 4: Evaluation Design
Module 5: Data Collection and Analysis
MODULE 1: DEFINING THE CONCEPTS OF MONITORING AND EVALUATION

It is important that we have a good understanding of the origins of monitoring and evaluation.
This will enable us to appreciate the usefulness of the concept in the context of policy and
decision- making and the effective management of social sector programmes.
Origins of Programme Evaluation
During the administration of Presidents Kennedy and Johnson, the United States Government
allocated considerable resources to social programmes to combat poverty and to stem the social
and economic marginalisation of minority groups, especially blacks. Also known as war on
poverty and great society programmes, massive resources were made available to deal with
unemployment, crime, urban decay, access to medical care, mental health treatment, and
housing. Due to the size of the programmes, there was interest in finding out if their goals and
objectives had been achieved. Social commentators, political activists and politicians wanted to
know the overall benefits of these programmes in terms of what worked and what did not work.
Some of these programmes were hurriedly designed and implemented, and were often thought to
be too expensive in relation to the outcomes and impact. The rise of fiscal conservatism in the
1970s; the reduction in federal expenditure in the 1980s; and the emphasis on results-based
management in the 1990s, further reinforced the interest in evaluation.
What is Evaluation?
Evaluation is the use of social science methods to collect, analyse, interpret and communicate
information about the effectiveness of social programmes, which are initiated to improve human
conditions. 3 However, it should be noted that the contexts of social programmes do not lend
themselves to rigorous social science methods and standards. The key is for every evaluator to
be adaptable as well as sensitive to the information needs of decision-makers who call for
evaluations. It is important to remember that evaluation requires flexibility in approach and
thought, which implies that its purpose and audience will influence the scope. It is imperative
that stakeholders needs/questions be paramount when designing the evaluation. In other words,
programme evaluation should focus on issues that are of importance to the stakeholders. Having
said this, it behoves evaluators to ask appropriate questions and to use the highest possible
standards. After all, the bottom line of programme evaluation is to use findings to improve
social conditions 4 .
3
4
Ibid
Ibid
With increasing emphasis on results-based management, evaluation findings are being used as a
management tool to determine if a programme should be continued, discontinued, improved,
expanded or curtailed.
It should be noted that in addition to using evaluation findings to improve programmes,
evaluations could be conducted to contribute to social science knowledge. This may necessitate
the use of rigorous social science research methods such as experimental and quasi-experimental
design.
Furthermore, evaluation findings can be used for public relations purposes. In this context,
evaluation can be launched to justify a political decision or simply to promote the image of the
organisation that initiates it.
Need for Reliable Data
Whatever the purpose, an evaluation should produce reliable data that can stand both internal and
external scrutiny. Evaluation findings should guide programme managers and decision- makers
to make decisions to improve ongoing/future interventions. A fundamental principle, when
considering undertaking an evaluation, is to aim for quality in the design of the study so that the
findings can stand the test of time. Also, evaluation must be credible, impartial and costeffective.
Credibility
It is advised that the process for data collection be transparent. As indicated before, key
stakeholders must be consulted and their views must be taken into account when determining the
questions to focus on. Most importantly, the core competencies or the skills set of the evalua tors
must be sound.
Impartiality and Independence
Evaluators should remain impartial throughout the evaluation process. Above all, findings and
conclusions should be free of personal biases. Judgements must be based on proven and reliable
data that can withstand scrutiny.
Cost-Effectiveness
The benefits of evaluation should outweigh the costs. The evaluation should be well tailored and
focused to minimise costs. Under most circumstances, the total cost of an evaluation should not
exceed 10% of the annual expenditure on the programme being assessed.
Key Evaluation Issues

Evaluation can focus on any of the following issues:
Issue
Question (s)
Programme relevance
Is the intervention still relevant in relation to the

original problem? If yes how? And if not, why not?
Programme design and effectiveness
To what extent has the intervention met the stated

objectives? What are the internal workings of the
project?
Programme efficiency
How cost-effective were the various activities that

were implemented? Could there be an alternative
strategy to implement the intervention? If yes, how?
And if not, why not?
Programme accountability
How well did the intervention adhere to established

guidelines, procedures and policies during the
implementation of the activities?
Programme impact
Did the intervention result in changing the

circumstances of the beneficiary group in a
substantial way? If yes, how? And if not, why not?
Catalytic effect
Did the programme result in changes that were not

anticipated?
Programme sustainability
Can the intervention survive when funding ceases?

If yes, how? If not why not? Does the community
support the programme?
It should be noted that evaluation could cover any or all of the issues raised. When all of the
issues are covered it is known as a comprehensive evaluation or an in-depth evaluation. The
type of evaluation to be undertaken is invariably determined by the uses of the evaluation
findings. Above all, whoever is requesting the evaluation could also influence the types of
questions that could be asked. For instance, if the request comes from sponsors of the
intervention such as the government, they may want to know whether the established policies
and procedures are being followed as well as the ultimate effects of the intervention on the
beneficiary group. If it comes from those who are managing the intervention, they might be
interested in finding out about the internal workings of the intervention in terms of the timeliness
and cost effectiveness of the activities being implemented. These issues are discussed further in
Module 3.
Remember this caveat: Keep the evaluation as focused and simple as possible. Too many
questions could result in a superficial assessment and/or inefficient use of valuable time as well
as human and financial resources.
Evaluation versus Social Science Research
Evaluation makes use of social science data collection methods and as mentioned earlier, every
effort must be made to ensure that the data, which is collected during evaluation, is reliable and
can stand any scrutiny. However, evaluation is more art than science because it is often tailored
to meet the needs of decision- makers and programme implementers who may be impatient for
the results. The nature of the request may not give the evaluator sufficient time to apply the most
rigorous of methods.
Social science research is undertaken to produce knowledge about a given problem. The
methods used tend to be more rigorous (for example: experimental design) than the tools used to
design social interventions. The reason being that it may be unethical to withhold services from
a segment of a population just for experimental purposes! It may even be politically
unacceptable to do so.
One cannot possibly undertake a thorough evaluation if appropriate mechanisms have not been
put in place to facilitate the monitoring of the intervention on a regular and systematic basis.
Without a good monitoring system it would be difficult to compare pre- intervention and postintervention conditions to ascertain if an intervention has indeed made a difference in the lives of
the beneficiaries.
What is Monitoring?
Monitoring is used to determine how well a programme is carried out at different le vels and at
what cost. It tracks changes that occur over time in terms of resource inputs, production, and
use of services. Above all, a monitoring system provides information on progress towards
the achievement of stated objectives.
Steps in a Monitoring Process
For a monitoring system to work effectively, it requires the development of a management
information system (MIS) for data capture, storage, retrieval and analysis. This could be based
on manual and/or electronic templates. It may be advisable to develop electronic templates for
more complex monitoring requirements. For instance, to date, there are over 150 social
interventions across Trinidad and Tobago aimed at improving the lives of the most marginalized
segments of society. These interventions are carried out through several Government Agencies
and reach thousands of beneficiaries. The data for such interventions should be available in
electronic formats and stakeholders should be able to access the data at any point in time.
Role of Performance Indicators in Monitoring
Monitoring is relatively straight forward if right from the on-set of an intervention, thought is
given to developing indicators for the defined objectives. The data collected should be based on
the agreed indicators. The information derived could then be used to improve the activities (see
Module 2 for additional discussion on objectives and indicators). In conclusion, it should be
underlined that routine collection of intervention data is necessary. It helps to improve on
programme management and performance and it enables us to know how well a programme is
doing.
If lapses are detected midstream, measures could be taken to streamline them.
Moreover, it facilitates accountability in terms of determining if established policies and
procedures are being adhered to.
References:
1.
2.
3.
4.
Peter H. Rossi et al, Evaluation: A Systematic Approach, 6th Edition, Sage Publication,
1999.
International Programme for Development Evaluation Training (IPDET) July 2003,
Module 1 Introduction to Development Evaluation and Module 2 Evaluation
Models.
World Bank, Operations and Evaluation Department, Monitoring and Evaluation: Some
Tools, Methods and Approaches, 2002.
Carol H. Weiss, Evaluation Research: Methods of Assessing Programme Effectiveness,
1972, Princeton Hall.
MODULE 2: PROGRAMME PLANNING AND DESIGN

This session focuses on programme initiation, planning and design. The underlying assumption
is that evaluation should be a key consideration when programmes are being designed. It should
form an integral part of the programme to facilitate the collection of appropriate data to inform
decision- making. There will be emphasis on conceptual models, logical framework, writing of
goals and objectives, identifying indicators, and defining activities. Also, participants will be
introduced to programme theory, in particular how to identify programme assumptions and logic,
including establishing causal relationships. This should help in having an understanding of how
a programme works. This is because the underlying assumptions of a programme may or may
not be valid. There is a need to review a programme theory in order to establish plausible
explanations for what has worked and what has not worked.
Considerable thought must go into conceptualising and designing a programme. Programme
theory or assumptions underlying a programme/project should be carefully considered. This
module is predicated on the assumption that what goes into the initiation and preparation of a
programme would go a long way in determining its ultimate success in terms of meeting the
goals and objectives for which it was designed. In particular, if the programme is well designed,
it becomes relatively straightforward to initiate formative and/or summative evaluations. The
purpose of this module is to review the various pre implementation steps programme managers
should follow before designing a programme.
Needs Assessment
Prior to designing a programme it is advisable that an attempt is made to review existing social
and economic conditions and to determine how these impact on the proposed beneficiary group.
Information gathered through a needs assessment can form the basis of project conceptualization
and design. A needs assessment helps to probe and identify specific problems of a group or an
entire community after which an appropriate intervention/strategy could be designed to respond
to those challenges. Ideally, project implementation strategies should be based on the results of
a needs assessment.
When is it appropriate to conduct a Needs Assessment?
In an ideal world, a needs assessment is undertaken prior to designing an intervention. However,
it is possible that an organisation may have acquired considerable information and knowledge
about the proble m for which an intervention will be designed to solve. Under such
circumstances, it may not be necessary to do a needs assessment. Alternatively, there may be
other organisations that may have done extensive research on the subject matter. These findings
could therefore form a basis for the design of the intervention; but should it become necessary to
10
do a needs assessment, consideration should be given to choosing the most appropriate

methodology. Both quantitative and qualitative data can be gathered (See Module 5 for a
description of the two approaches).
Baseline Assessment
This method is used to collect information on the focus beneficiaries of a proposed intervention.
The findings from a baseline assessment should enable a comparison of change resulting from
the intervention. The pre- and post intervention data is compared to enable us to determine how
and to what extent the intervention has had any tangible effects on the beneficiaries.
Baseline data example
Average rating of
50 Homes for the Aged
in Community A
January 2003 = 5.2
Baseline data
Intervention
Sensitization
& Training
Programme
Average rating of
the same Homes
June 2003 = 6.4
Post Intervention Data
Conceptual Model as an Integral Part of Programme Design

A conceptual model is a theoretical construction of how the proposed intervention will achieve
the desired results. It serves as the foundation of the project and it helps in defining objectives
and indicators. Above all, the conceptual model should explain how the strategies to be
employed during the intervention would result in a desired change. Information that is collected
during a needs assessment could be useful when developing a conceptual model.
Graphics can be used to depict the relationships between the intervention, results and goals:
Identification
of a problem
Design of an
intervention
Goals and objectives

to be addressed
Results
There are some key questions worth asking when designing a conceptual model. These are:
What are the specific problems affecting the focus population?
Which of these problems can the intervention realistically deal with?
11
What changes must occur in the beneficiary population to reduce the problem?
What activities should be carried out to bring about the change? etc.
Conceptual Model for the Civilian Conservation Corps
Exposure to Induction
training: eg. Physical
Training, Lectures in
life skills development
Participation
of Youth at
Risk in
6 month
CCC Program
Increased
Employability
Increased confidence,
development of
positive work ethic,
development of
relevant skills
Exposure to On
the Job
Training in one
of 10 skill areas
Transformation of
Youth at Risk into
productive, well
balanced and
responsible citizens
Development of selfemployment capacity
Reforestation and
conservation of the
environment
Establishing a Programme Theory

It is important that we understand the theory that underpins a programme. It may be that a
programme or its underlying assumptions may not be sound, hence the need to review it. Such
an exercise should help to establish plausible reasons for what has worked and what has not
worked. A clear understanding of the programme theory requires a description of the concepts,
assumptions and expectations that represent the programme as structured and operated. It also
requires an analysis of key components including objectives, activities, and the target population.
Note that programme theory establishes relationships between programme resources,
programme activities, and programme outcomes.
Developing a Logical Framework
After developing a conceptual model, the next step is to develop a logical framework.
What is a Logical Framework?
It is a method for organising a project in a graphic form. A logical framework provides details of
what the project will accomplish, how it will accomplish it and how one will know if it has been
accomplished.
12
What are the key components of a Logical Framework?

The key components of a logical framework are as follows:
- The problem(s) the specific intervention is designed to solve (goal)
- What the intervention is designed to achieve (expected effects/impacts)
- How it will be done (activities)
- How the objectives have been or will be attained (results and process indicators)
- The means of verification of indicators (data sources)
- How often data will be collected (frequency)
- Who will collect the data? (person/department/organisation responsible)
Extracts from a Logical Framework:
Project Title :
One Caring Adult
Project Goal:
To facilitate the partnering of an adult mentor with an at risk youth in order to

encourage the personal development and social integration of the mentee.
Objective 1:
To provide a source of guidance and support for a minimum of 30 at risk youth in yr 1.
Key Activities
Indicators
1.1 Advertise for mentors

1.2 Screen & approve 30
mentors
1.3 Train Mentors
1.4 Obtain list of names of
possible beneficiaries
from community
groups, NGOs,
probation, community
police etc.
1.5 Screen & approve
mentees
1.6 Match mentors &
mentees
Process:
No of mentor
applications approved.
No of groups
providing beneficiary
listings.
No of persons
participating in mentor
training.
Results:
Percentage of mentees
satisfied with mentor
support system.
Means of
Verification
Frequency Person
Responsible
Admin. Records
Annual
J.Smith/M.Doe
Admin. Records
Annual
J.Smith
Admin. Records
Annual
J.Smith /
Consultant
Post-intervention Annual
survey of mentees
J.Smith
Defining Goals and Objectives

These two concepts are often used interchangeably. Nonetheless their meanings are different.
What are Goals?
Goals represent long-term endeavours to bring about change or an improvement in the
circumstances of the target beneficiaries, the timeframe is often 3-5 years or longer. For
example: To eliminate abject poverty in Trinidad and Tobago by 2015, etc.
13
What are Objectives?

Objectives are more specific than goals. They are time-bound and are measurable. Here are few
examples: (i) To reduce the number of vagrants in St Anns by 10 per cent by 2005. (ii) To place
100 children in foster homes by the end of the 2003 financial year.
It is important that data is collected on each of the objectives to pin point if, and to what extent,
the specific objectives have been fulfilled.
Impact Objectives
Impact objectives are objectives that focus on measurable outcomes. To determine if there
has been a measurable change, it is preferable to use action verbs such as the following:
Reduce
Improve
Strengthen
Increase
Decrease
Enhance
These verbs can be used to denote interventions that are aimed at improving the general
operations of an organisation. Some examples:
To increase the number of foster homes by 50% by 2004.
To reduce staff turn over in the Research Unit from 10% to 3% by 2005.
If the main preoccupation is to introduce an intervention that will lead to a change in the
circumstances of the beneficiaries, then it is advisable to avoid inappropriate verbs such as the
following:
Train
Provide
Produce
Establish
Create
Conduct
However these verbs can be considered if the intervention is geared towards the provision of
services such as the following:
To serve 500 clients in the 2003 financial year
To train 300 peer educators by the end of the programme year.
14
Ultimately impact objectives are key since in practice most interventions aim to effect change in
the circumstances of the beneficiary population. We should endeavour at all times to write
smart objectives. SMART stands for:
Specific avoid differing interpretation
Measurable monitor and evaluate progress preferably in quantitative terms
Appropriate to the problem being addressed
Realistic- achievable yet challenging
Time-bound within a specific timeframe
Choosing Activities
After identifying the objectives, the next step is to choose activities. Activities refer to specific
tasks. They are what the organisation proposes to do to achieve the defined objectives. It is
advisable to review available skills and expertise before identifying the activities. Examples of
activities:
Organise counselling sessions for teenage girls who could be at risk of getting pregnant.
Conduct 10 hours of training for recipients of hampers under the SHARE programme.
Identifying Indicators
An indicator is a measure of a concept or behaviour. An indicator is used as a road map to assess
how far and the extent to which specific project objectives have or have not been attained.
There are two types of indicators, namely process and results indicators.
Process Indicators
Process indicators provide information on the activities that are being implemented in terms of
types of activities, the number, who the activities are directed at, etc. These indicators provide
information that would enable us to determine if an intervention is moving in the right direction
in order to achieve the stated objectives. This type of information is collected throughout the life
of the intervention. Examples of process indicators are: (i) Number of people who have visited
the San Juan Community Mediation Centre in the last quarter of 2002; (ii) Number of cases that
have been successfully resolved at the San Juan Community Mediation Centre in the first quarter
of 2003.
Process indicators are useful for monitoring. Data collected using process indicators help in
determining the reasons for the success or failure of an intervention. For example: Number of
participants at the M&E training programme organised by the Ministry of Social Development.
15
Results Indicators
These types of indicators are closely linked to the stated objectives of an intervention. They are
meant to provide a framework for assessing whether or not as a result of the intervention there
has been a visible change in the circumstances of the beneficiary population. The extent of
change can be measured at the programme level or the population level. Results indicators are
expressed as a percentage, ratio or proportion. It provides a basis for assessing the degree of
change in relation to the beneficiaries and/or their environment. Although results indicators are
meant to measure change, they should not be anticipative. For example: instead of reduction in
the number of teenage pregnancies it is more appropriate to write percentage of adolescent
girls of 10-19 years old who have had babies in the last year5
Results Indicators and Objectives
Since results indicators provide an indication of whether or not an objective has been achieved, it
is advisable to include at least one result indicator when designing the intervention.
Principles for selecting Results Indicators
Indicators should be precise and clear. If indicators are written as percentages both the
numerator and the denominator should be specified. For example: Percentage of beneficiaries
of the SHARE programme who have successfully set up a micro enterprise.
Number of beneficia ries who have successfully set up micro enterprises Numerator
The total number of beneficiaries participating in the SHARE programme Denominator
Criteria for Selecting Indicators
There should be emphasis on the selection of indicators that are clear and concise. The
following criteria must be considered when selecting indicators 6 :
Relevance
There should be a clear relationship between the indicator and the

objective being measured. Whatever information is collected must
be useful to decision- making. More information is not necessarily
more useful.
Reliability
Relates to the stability of the measurement process. The same

measurement process should produce the same findings even if the
data analysis is repeated several times over.
IPPF/WHR, Guided for Designing Results-Oriented Projects and Writing Successful Proposals,
December 2000, p18
Ibid p18
16
Validity
The indicator must be consistent and should represent what is

actually being measured. A host of factors can influence the
validity of the data being measured including poor design of the
data collection instrument, poorly trained data collection staff,
measurement errors, poor sampling, transcription errors, etc.
Availability of
information
There should be ready access to sources of data.
Ease in
measuring
The indicator does not require sophisticated methods of

measurement
Easy to
understand
The social planner/evaluator must clearly communicate what is

being measured and the user must understand what is being
measured.
Cost effectiveness
The cost of data collection in terms of both human and financial

resources should be considered when choosing an indicator. It
should not be too expensive to collect the data. The basic rule of
thumb is that costs associated with evaluation should range
between three and ten percent of the total cost of the intervention.
Robustness
The data that is generated must be reliable and replicable.
Timeliness in
data collection
Data collection and analysis should take place within a welldefined timeframe in terms of the frequency of data collection and
the currency of the data.
It may be prudent to select more than one indicator for an objective. This is because an objective
may have different dimensions and complexity. In view of time constraints and in order to be
cost effective, it is advisable to identify only a few indicators that are manageable.
An example of a poor indicator:
Increase in the percentage of youth using condoms during first sexual experience.
An appropriate indicator:
Percentage of sexually active youth (ages 15 to 24) participating in a sensitisation project that
report- using condoms at last intercourse by age and sex.
17
References:
1.
2.
3.
Peter H. Rossi et al, Evaluation: A Systematic Approach, 6th Edition, Sage Publication,
1999.
International Planned Parenthood Federation, Guide for Designing Results-Oriented
Projects and Writing Successful Proposals, Western Hemisphere Region, Inc, December
2002.
Susan Adamchack et al, A Guide to Monitoring and Evaluating Adolescent Reproductive
Health Programmes, Focus on Young Adults, Tools Series 5, June 2000.
18
MODULE 3: PLANNING A USEFUL EVALUATION

Social interventions are designed to achieve government-wide goals in relation to solving social
conditions or problems. Evaluations are initiated to determine the effectiveness of such
interventions in terms of achieving the desired goals and objectives. Participants will be
introduced to simple techniques for planning useful evaluations. Participants will be taken
through the essential steps in initiating, planning and undertaking an evaluation, including the
evaluation process, types of evaluations, evaluation assessment and scope of the evaluation.
The Evaluation Process
Determining what questions to select for evaluation is not easy, more so if there is no process in
place to facilitate the choice of issues on which to focus. Ideally, there should be a mechanism to
initiate the evaluation in terms of determining the issues, the preparation and approval of the
scope of work, the undertaking of the evaluation, the submission of the final report to the
initiator of the evaluation for approval, and the implementation of the recommendations.
In the specific context of Trinidad and Tobago, Parliament, the Prime Minister, the Cabinet, the
Minister of Social Development, the Co-ordinating Committee on the Social Sector and
Permanent Secretaries may initiate evaluations.
Types of Evaluation
An evaluation could focus on any of the following:
1.
Projects
2.
Programmes
3.
Themes
4.
Sectors
5.
Country
6.
Programme effectiveness
7.
Programme efficiency
8.
Programme impact
9.
Programme sustainability
10. Overall programme evaluation and management audit
1.
Project
This is a single intervention with defined goals and objectives, which can be implemented in a
specific location or in several locations (towns, villages, communities, etc.) For example, an
intervention designed to combat high incidence of teenage pregnancy in Morvant.
19
2.
Programme
A programme consists of several activities or projects with defined goals and objectives that aim
at improving the social and economic circumstance of the beneficiaries. An example is the
SHARE programme.
Note that both projects and programmes can be subjected to mid-term and final evaluations. A
mid-term evaluation is undertaken in order to determine the overall effectiveness, efficiency, and
impact of a project or programme. Findings from a mid-term evaluation could result in making
changes to the project/programme. A final evaluation or ex-post evaluation is an assessment of a
project/programme after it has been completed.
3.
Thematic Evaluation
This type of evaluation focuses on selected interventions within a sector that addresses specific
priorities. For example: teenage mothers, drug addicts, and alcoholism.
4.
Sector Evaluation
Sector evaluation focuses on a cluster of development interventions.
education, training, agriculture, micro-enterprise, etc.
For example: health,
5.
Country Evaluation
This type of evaluation is common with donor-funded programmes/projects.
organisation can decide to evaluate its activities in a given country.
A donor
6.
Programme Effectiveness
It is often assumed that once a policy/programme is initiated one can expect successful
implementation. This is always not the case since the intervention may not have been effectively
implemented. This could be due to poor design, inadequate inputs and a host of many reasons.
This explains why a key component of evaluation is to focus on how well a programme has been
implemented by looking at the inputs, processes, outputs and outcomes.
Inputs
This refers to resources (i.e. personnel, facilities, space, equipment and supplies) that make it
possible for successful implementation of a project/programme.
Programme Processes
This refers to the use of inputs to undertake activities, which will lead to the realisation of the
objectives that have been designed for the intervention. Programme processes range from
management procedures to research and evaluation systems.
20
Programme Outputs
These are the results obtained at the programme level through activities and use of inputs.
Programme outputs can be classified into functional outputs, service outputs and service
utilisation. Some examples of functional outputs are: the number of beneficiaries trained to set
up their own business; the number of training sessions conducted; etc. Service outputs focus on
access to services and quality of care, such as number of drug addicts who are currently being
treated for addiction and number of those treated who are satisfied with the type of services
being offered at the treatment centres. Service utilisation denotes the ability of a programme to
retain repeat clients and more significantly to attract new clients, such as the number of new
clients who have access to the SHARE programme.
Programme Outcomes
These are results that occur after implementation of activities. Outcomes can be at the
programme level or the population level and are medium term or long term in nature.
The relationship among the programme components can be illustrated as follows:
Input
Process
Output
Outcome
In sum, evaluation that focuses on programme effectiveness/processes attempt to find out what
works and what does not work when carrying out the implementation of activities. The
following issues are often considered: How well is the programme functioning? To what extent
are the services delivered as intended? Are the services being provided congr uent with the needs
of the target beneficiaries? What are the management capabilities of the implementers? 7
Bertrand, Magnani and Tutenberg, Evaluating Family Planning Programes. The Evaluation Project, September
1996
21
Example from the Adolescent Mothers Programme

Inputs
Teachers, social workers, child care trainers, skills trainers,
centres/service delivery points, teenagers, financing etc.
Process
Provision of academic/skills/parenting & child care training, psychosocial support to pregnant teenagers and teenage mothers who attend the
Centre 9am-3pm 5 days per week for 3 months.
Outputs
50 teenagers trained in skills, child care and parenting, more prepared for
new responsibilities.
Outcome
Reduction in birth rate in under 19 age group;
Reduction in child morbidity (for children of trainees);
Increase in awareness of risks of early child bearing
Increase in awareness of benefits of child spacing etc.
7.
Programme Efficiency
It is often said that knowledge of programme results is not sufficient enough to declare success in
producing outputs and outcomes. Results must be measured against their costs. Due to
competing demands on the resources of the government, it behoves programme managers to
demonstrate that their programmes are not excessively expensive and that all things considered,
their programmes are providing value for money. Programmes could be terminated or retained
on the basis of their comparative costs. Of course in the realm of politics, it is not always
feasible to kill a programme as a result of inefficiencies or cost over runs. Inefficient
programmes may be kept purely for political expediency.
The following questions should be asked when conducting efficiency assessment:
Is a programme producing sufficient results in relation to the overall costs/resources
deployed?
Which approach to providing the service results in the lower cost per unit in relation to
the outcome?
22
There are two main techniques for assessing programme efficiency, namely cost-benefit
analysis and cost-effectiveness analysis.
Cost-Benefit Analysis
Cost-benefit analysis focuses on the relationship between programme costs and outcomes with
both cost and outcomes expressed in monetary terms:
Hospital Care
Salaries
Overheads
Food
Utilities
Hospital bed, etc.
Community Care
Food
Rent
Training
Service delivery, etc.
Total cost $10, 500

Estimated Benefits $20,500
Net Benefits=$10,000
Total cost $15,500

Estimated Benefits $30,500
Net Benefits =$15,000
To over simplify, the benefits derived from a programme should outweigh the cost of providing
it. This method could be applied to a single programme, two programmes with the same goals
competing for the same funds, or two programmes with different goals competing for the same
funds.
Generally speaking, cost-benefit analysis is difficult to do because it requires making
assumptions about the monetary va lue of programme benefits.
Cost-Effectiveness
Cost-effectiveness analysis examines the relationship between programme costs and outcomes,
in terms of the cost per unit of outcomes achieved. This method compares two programmes
which both aim to achieve a comparable outcome. For example, the unit cost of rehabilitating
one drug addict could be $100,000 per year in Programme X but $80,000 per year in Programme
Y, making Programme Y more cost effective. Cost effectiveness could also be considered from
more than one point of view, for instance, the cost-effectiveness of two dispute resolution
options (i) to the client or (ii) to the state.
23
Resolution of dispute X via the

Mediation Centre
Resolution of dispute X via the

Court system
Cost to the Client:
Transport
Cost of Lawyer
Cost of working hours lost, etc.
Cost of working hours lost, etc.
Total cost $800

Rent
Salaries
Professional Fees, etc.
Total cost $3, 500
Total cost $7,500

Cost to the State:
Salaries
Cost of Legal Aid
Utilities, etc.
Total cost $ 10,000
Comparing the two Concepts

As indicated earlier, in cost-benefit analysis, programme outcomes are expressed in monetary
terms. For example, a cost-benefit analysis to reduce the high incidence of HIV/AIDS infection
could focus on the difference between the money spent on sensitisation campaigns and the
savings made by government in providing anti-retroviral drugs to persons living with AIDS
(PLWAs). In cost-effectiveness analysis, outcomes are expressed in substantive terms, for
example, the money spent per person on various sensitisation programmes to encourage people
to have fewer sexual partners. In doing cost-benefit and cost-effectiveness analyses, the
following must be kept in mind:
Analysis is undertaken to determine whether or not the net effects justify the cost of the
intervention.
Estimates of costs relative to benefits could be based on tangible and intangible factors
The cost of undertaking an intervention can be direct or indirect.
Above all, it should be noted that it is not always easy or possible to assign monetary value to
social programmes. 8
8.
Programme Impact Assessment
According to Peter H. Rossi et al, the central premise of any social programme is that the
services it delivers to the beneficiary group should induce some change that improves social
conditions 9 .
8
9
Peter H. Rossi et al, pp 365-391

Ibid, p 102
24
Impact assessment determines the extent to which a programme delivers on the intended
objectives in such a way that it results in improvements in the social conditions of the target
beneficiaries. Some questions to consider when doing an impact assessment:
To what extent could programme outcomes be attributed to the intervention?
To what extent did the programme succeed in producing change in the social conditions
of the beneficiaries?
To what extent can one attribute changes that have occurred to the specific interventions?
These questions seek to establish a cause and effect relationship. The bottom line is to
establish the net effect of an intervention. In order to do so, it is useful to define outcome
variables. It may be possible to use classic experimental design (treatment and control
group-subjects with the same characteristics randomly selected.) This type of design is
useful for a pilot project-See Module 4.
Programme objectives designed for impact assessment:
Changing behaviour e.g. sexual behaviour to avoid contracting sexually transmitted
diseases
Lowering functional illiteracy
Reducing homelessness
Minimising teenage pregnancies
Reducing incest
Reducing child abuse
Reducing unemployment among the youth
A Diagram Depicting Impact Theory/Assumptions:
Skills
Job
training
Job
search/Int
.
Better job
Improved
Economic
Status
Motivation
to work
25
9.
Programme Sustainability
It denotes the extent to which an intervention can continue to be viable and produce benefits after
the completion or closure of the interventio n.
10.
Overall Programme Evaluation and Management Audit (OPE/MA)
An OPE/MA is an evaluation that covers all facets of programme evaluation components such as
those described above.
Evaluation Assessment
Evaluation assessment studies are undertaken to ensure that resources designated for evaluation
are judiciously used to answer the right questions in an appropriate and credible manner. It may
be noted that evaluation assessment is not a substitute for the actual evaluation. It is meant to
underscore the importance of creating demand and a buy- in of the findings. The ultimate
objective is that the findings will be accepted and used.
What is the rationale for undertaking evaluation assessment?
The following are the underlying reasons for initiating an evaluation assessment:
To understand the intervention and the environment in which it operates
To define the purpose of the evaluation
To identify the users of the evaluation findings (stakeholders)
To define key evaluation issues and related questions
To identify key evaluation methods and strategies which would be utilized to determine
the availability of data and quality of data which is available (if any)
To estimate the cost and resources required for data collection, data analysis and
producing the report
To define the timeframe
To formulate options in terms of the questions and methods to be used to collect data
To formulate recommendations.
The Intervention and its Context
It is important to have a good understanding of the intervention in terms of its goals, objectives,
targets, the activities, the reach and the expected outcomes. It may be useful to establish
plausible relationships among the activities. Discuss the origins of the intervention by reviewing
legislative instruments, Acts of Parliament, etc. and assess whether or not the intervention is
26
doing what is it was set up to do. Review the appropriateness of the underlying assumptions, the
programme reach and determine if the expected outcomes are still realistic. An example of a
programme profile is presented in the box below:
Profile of an Intervention
Typically an intervention should have the following core profile:
Goal:
Establish the overarching goal of the programme - what the

intervention is supposed to focus on in relation to the
reach/beneficiaries and expected outcomes.
Description:
Outline the background to the intervention, such as the legal or other

authority under which it operates; the historical context and the
theory that underpins the intervention.
Objectives:
Describe what the intervention is designed to achieve and how it can

change the lives or circumstances of the beneficiaries.
Target Group: Identify who are the intended beneficiaries of the intervention.
Activities:
Expected
Outcomes:
Budget:
Describe the activities that will be undertaken to achieve the stated

objectives.
Describe the medium to long term benefits to be achieved.
Give a breakdown of the cost of the resources to be utilized for the
intervention (i.e. human, financial, infrastructure, etc.)
Stakeholder Analysis
Assess the main actors of the intervention such as programme managers, programme staff, policy
makers and beneficiaries. Indicate how the evaluation will be of immediate benefit to them.
Define Key Issues/Questions
There is a need to take into consideration the information requirements or needs of the
stakeholders, namely decision- makers, programme managers and beneficiaries. Try to tailor the
evaluation questions to answer the issues that are of importance to them. In other words, the
concerns of the stakeholders must be translated into specific questions. The questions will form
the basis of data collection. It is advised that the list of questions should be shared with
programme managers and policy makers for their input. Below are samples of questions:
27
Programme Rationale:
Is the intervention still relevant?
Is it meeting the purpose for which it was designed?
To what extent are the activities and outputs consistent with the stated goals?
Objectives Achievement:
Did the intervention achieve the stated objectives?
How realistic were the objectives?
To what extent were the targets achieved?
Were the objectives in line with the needs of the beneficiary population?
Outcomes and Effects:
Did the intervention lead to a change in the circumstances of the beneficiary population?
Were there any unintended effects?
Were there any catalytic effects?
Programme Efficiency:
To what extent were the resources (i.e. financial, human, equipment, etc) appropriately
utilized?
Were any savings made?
Is there an alternative way of achieving the stated objectives in a cost-effective manner?
Define Evaluation Methods/Strategies
The evaluation must describe the evaluation approaches that will be used to answer the defined
questions. The methods selected should be time-tested/rigorous to facilitate the collection of
appropriate information to meet the needs of the client(s). The methods should be consistent
enough to allow for both internal and external validity.
Identify Tools for Data Collection
Data collection tools must be rigorous. However it is to be noted that evaluation is not a
longitudinal research to produce knowledge or definitive statements. It is a tool for providing
timely information that is appropriate for policy and decision-making. Here are a few examples
of data collection tools: rapid assessment, focus group discussions, case studies, time series
analysis to detect changes over time, pre- and post-intervention surveys, etc. If the size of the
beneficiary population is too large, it may be useful to use some sampling techniques. There are
two types of sampling techniques: non-probability sampling (i.e. purposive samples and quota
samples) and probability sampling (i.e. random samples, stratified samples and cluster samples).
See Module 4 for further discussion on sampling.
28
Data can be obtained through interviews, survey questionnaire, direct observation, review of
documents and records, etc. It is advisable to use several sources of information to minimize
personal biases (i.e. data triangulation).
Estimation of Cost and other Resources
Estimate the total cost of the proposed evaluation, including other resources. Describe the team
composition and the competencies/skills required to undertake the evaluation. Do an
implementation plan with specific time lines/benchmarks.
How long does it take to do an Evaluation Assessment?
Depending on the scope and complexity of the intervention, an evaluation assessment may take
up to a month to complete.
Conclusions and Recommendations
The conclusion should entail a recommendation regarding the specific evaluation questions that
could be covered during the actual evaluation. Options should be given with regard to possible
evaluation approaches and the related costs of each approach.
Scope of the Evaluation
Once a decision is reached to undertake an evaluation, the next step is to prepare the scope of the
evaluation to clarify the programme context, what issues to focus on, who will do what, when it
will be done and the resources required for undertaking the evaluation.
Step 1:
Define the programme context. Define the goals and objectives of the programme
including the conceptual framework that maps out the linkages between inputs,
processes, outputs, and outcomes.
Step 2:
Define key issues to be addressed during the evaluation.
Step 3:
Define the methodological approach in terms of how you intend to carry out the
evaluation (i.e. study design, indicators and data sources).
Step 4:
Describe the implementation plan in terms of the main deliverables, who (individuals
and institutions that will be responsible for aspects of the evaluation), when (time
table for carrying out specific activities) and with what funds (budget).
Step 5:
Define format for presentation of findings and dissemination and utilization of

results.
29
References:
1.
2.
3.
4.
5.
6.
Peter H. Rossi et al, Evaluation: A Systematic Approach, Sage Publications, 1999.

Susan Adamchak, A Guide to Monitoring and Evaluating Adolescent Reproductive
Health Programmes, Tools Series, 5 June 2000.
Michael Q. Patton, Utilization Focused Evaluation, Sage Publications, 1978.
Andrew A. Fisher et al, Handbook for Family Planning Operations Research Design,
Population Council, New York, 1998.
Bertrand et al, Evaluating Family Planning Programmes, The Population Project,
September 1996.
The World Bank, Monitoring and Evaluation: Some Tools, Methods and Approaches,
2002.
See Appendix I for Tips on How to Plan a Useful Evaluation
30
MODULE 4: EVALUATION DESIGN

It is important that evaluations produce information that is authoritative and can stand external
scrutiny. Consequently, appropriate methods and tools for evaluation must be utilised to ensure
credibility of results. This module focuses on selected evaluation designs such as experimental
design, quasi-experimental design and sampling techniques.
Experimental Design
In experimental design, the subjects, namely the experimental group and the control group, are
randomly selected from a single population. This is otherwise known as random assignment of
cases (RA).
The experimental group and the control group are subjected to an initial
observation (O: the pre-tests O1 and O3 ). The experimental group then receives the intervention
(X). However, the control group does not receive this intervention. Subsequently, a second set
of measurement observations is made (O 2 and O4 ). It is assumed that since the experimental
group received the intervention (X) and the control group did not, O2 would be greater than O4.
(Experimental Group)
O1 X
O2
Pre-test
Post-test
O3
O4
(Control Group)
It should be noted that since cases for both the experimental group and the control group were
randomly selected, O1 would be equal to O3 on such key variables as age, sex, parity and
education.
This is a true experimental design. It enables a good control of validity. However in practice,
for ethical, programmatic and political reasons, it may be difficult to deny programme
interventions to one group while offering them to another. This type of design is more
appropriate for pilot testing a programme for eventual replication elsewhere.
Non Experimental Design
Post-test Only Design
X
Time
O1
31
Intervention (X) is introduced and sometime after the introduction, an attempt is made to
undertake some measurement (O 1 ). Considering that there is no control group, it is not possible
to make a comparison. O1 provides descriptive information and other data that could be useful
for planning an intervention. It is used for conducting diagnostic studies to determine why a
problem exists.
Pre Test and Post Test Design
Time
O1 X
O2
There is no control group for this type of design. However, there is an earlier measurement
observation (O 1 ) that makes it possible for the evaluator to assess changes overtime after an
intervention (O 2 ). If there is a significant change in the circumstances of the beneficiaries then
one could attribute those to the intervention. The findings could be influenced by threats to
validity, history, maturation, etc.
Static Group Comparison
(Experimental Group) Time (Control Group)
X O1
O2
In this design, the experimental group receives an interve ntion (X) followed by observation (O 1 ).
This measurement observation is compared against a second observation (O 2 ) from a control
group that did not receive the intervention. The design is not based on random sampling, which
means that there are differences between the two groups (not equal) and hence strong threats to
validity.
Quasi Experimental Design
We had indicated earlier on that due to ethical, programmatic and political considerations, it
might not always be possible to implement a true experimental design. Nonetheless, a similar
design could be applied to obtain almost identical results by using quasi experimental design
such as time series design.
Time Series Design
Time series design is similar to the nonexperimental, pretest | post-test design, except that it
has the advantage of repeated measurement observations before and after programme
intervention (X).
32
Experimental Group
O1
O2 X
O3
Time
O4
O5
O6
We may initially find that there is no difference between O1 and O2. Then it is observed that
there has been a sudden improvement in the circumstances of the beneficiary group, which is
observed by O3 and O4 . The increase is subsequently maintained in O5 and O6 . Therefore, it can
be concluded with some degree of confidence that the increases were probably due to the effects
of the intervention (X). This design is often used to perform trend analysis in terms of what
pertained before and after the intervention.
It would be interesting to apply this approach to assessing changes brought about by any of the
existing social sector programmes. For example to determine if, as a result of the distribution of
hampers under the SHARE Programme, the nutritional in take of the beneficiaries has indeed
improved.
Issues to Consider when Selecting an Evaluation Design
The following should be taken into consideration when selecting a design.
Ethical issues Try to avoid a design that could lead to the application of unethical
procedures, a violation of peoples rights and dignity, or a denial of services that would
otherwise be available.
The ideal situation is the use of experimental design with randomly selected subjects
based on a single population. If this is not possible, find a comparison group that is
nearly equivalent to the experimental group.
If a randomly assigned control group is not available or there is no similar comparison
group, you may consider using a time series design that can provide information on
trends before and after the programme intervention.
If it is not possible to use a times series design, try to obtain baseline (pre-test)
information that can be compared against post intervention information (pre-test post-test
design = O1 X O2 ).
The issue of validity should be considered at all times by asking such questions as: How
true are the measurements? Are there possible threats to validity? etc.
Another principle to bear in mind is that a good design does not stop the provision of
services.
Always use multiple data sources to obtain information, this is also known as data
triangulation.
33
Other Evaluation Tools and Methods

Rapid Appraisal Methods
Rapid Appraisal is a relatively quick and low-cost strategy to gather data from beneficiaries and
other stakeholders to feed into an evaluation report, in response to the information needs of
programme managers.
This method is flexible and easy to apply. However, the information gathered is largely
qualitative and it cannot be extrapolated. It is less valid and reliable than data collected from a
formal survey.
Here are some rapid appraisal methods:
Key informant interviews- a series of opened-ended questions posed to individuals who
have specific knowledge and experience of a topic/programme. Interviews are semistructured and in-depth.
Focus group discussion- a facilitated discussion with a carefully selected group.
Participants should not be more than 12.
Community group interview- a facilitated discussion based on a series of questions
opened to all community members.
Direct observation- often useful for collection data on the quality of service provision.
Mini-survey- a structured questionnaire that has a number of close-ended questions. This
mini-survey could be administered to 50-75 people. The selection of the interviewees
may be random or purposive (e.g. interviewing clients of a family planning clinic about
the quality of service provision).
Participatory Methods
This approach enables those who are directly affected by a programme, or have a stake in it, to
participate directly in assessing it. It enables them to have a sense of ownership of the
Monitoring and Evaluation findings and recommendations.
This method also allows the stakeholders to give their perspectives and impressions about the
usefulness or otherwise of the intervention.
The following are the most commonly used participatory tools:
Stakeholder analysis- useful for establishing key power brokers and relationships.
Participatory rural appraisal- it enables programme managers and local people to work
collaboratively on an intervention.
34
Beneficiary assessment- it involves regular consultation of beneficiaries and other

stakeholders.
Participatory monitoring and evaluation- it involves stakeholders in the planning and
implementation of an evaluation in all its phases such as identifying the problem,
collecting and analysing the information, and generating recommendations.
Sampling
A distinction must first be made between a total population and a sample of that population. A
population contains elements, each of which is a potential case. If an evaluation involves a small
population, all the elements can be studied. For example evaluating the health status of 50 people
living with HIV in a hospice in Arima. Since 50 people is a relatively small population, there is
no need to draw a sample, the entire population can be evaluated. On the other hand, it would be
costly and time consuming to evaluate the health status of the entire population of 50,000 people
living with HIV in Trinidad and Tobago. One may need to draw up a sample of elements/cases
within the 50,000 population to study.
Usually, the practice is to concentrate on a select few or a representative sub-set to interview. If
the sample were truly representative, the information obtained would be similar to the
information that would be obtained from the entire population.
There are two basic types of samples.
samples.
These are probability samples and non-probability
The Probability Sample

The essence of probability sampling is that each element of the larger population (couple, young,
old, male, female, etc) has a known probability of being selected. If each element has an equal
chance of being selected, it is known as self-weighting and the findings can be extrapolated to
the general population. The findings resulting from probability sampling are considered to be
truly representative.
There are several methods for drawing probability samples. The most common ones are as
follows:
Simple Random Sampling
In this sampling method, each element of the larger population is given a number and a table of
random numbers or a lottery technique is used to select elements, one at a time, until the desired
sample size is reached. This approach can be tedious. A list of all elements is called the sample
frame.
35
Systematic Sampling
This is a modification of simple random sampling which is less tedious and less time consuming.
The estimated number of elements in the larger population is divided by the desired sample size,
yielding a sampling interval. Let us call it n. Using a sample frame with the population
elements listed in an arbitrary order and selecting every nth case, starting with a randomly
selected number between one and n, the sample size is drawn.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Systematic drawn cases are 1 5 9 13 and 17
Stratified Sampling
Stratification can be used for either simple random sampling or systematic sampling to ensure
the desired representation of specific sub groups. For example, elements in the larger population
can be arranged by age, education, income, location, profession, political affiliation, etc.
Cluster Sampling
This method is used to simplify sampling by selecting clusters of elements, using simple random,
systematic, or stratified sampling techniques and then proceeding to study all the elements in
each of the sampled clusters. Usually the clusters are geographic units such as provinces,
districts, towns, villages, units or organisational units such as centres, clinics, or training groups.
NonProbability Sample
This is also known as convenience sampling. It refers to the selection of cases that are not based
on known probabilities. The selection could be accidental (choosing from whatever case that is
available) or purposive (selection from specific type cases). This type of sampling is not
representative of the larger population since there can be over selection or under selection of
cases. If it is too expensive to use probability-sampling technique, then non-probability
sampling may be the most appropriate method to use.
Sample Size
The size of the sample is determined by two main things, namely the availability of resources
and the proposed plan of analysis. For example, if you intend to analyse cross-tabulations of
variables, take into consideration that each category of an independent variable included in crosstabulation must contain at least 50 cases.
36
Confidence Level
In practice it is recommended that a study should aim at obtaining a 95% confidence level that
has a + or - 5% interval/error.
Example of sample sizes
Population = 500
Population = 8,000
Population=500,000
95%|+/-5%
99%|+/-5%
95%|+/-5%
99%|+/-5%
95%|+/-5%
99%|+/-5%
217
286
367
615
384
665
References:
1. Susan Adamchak, A Guide to Monitoring and Evaluating Adolescent Reproductive
Health Programmes, Tools Series, 5 June 2000
2. Michael Q. Patton, Utilization Focused Evaluation, Sage Publications, 1978
3. Andrew A. Fisher et al, Handbook for Family Planning Operations Research Design,
Population Council, New York, 1998
4. Bertrand et al, Evaluating Family Planning Programmes, The Population Project,
September 1996
37
MODULE 5: DATA COLLECTION AND ANALYSIS

Decisions affecting programmes and development of policies must be informed by the
availability and use of credible data. The basic rule of thumb is that evaluation should be based
on empirical evidence; therefore, there should be a well- laid plan for gathering and analysing
data.
There are various approaches that facilitate the collection of appropriate data. This module
focuses on strategies for collecting quantitative and qualitative data. In broad terms, quantitative
data tends to be precise and qualitative data tends to provide descriptive information.
Quantitative Data
Findings from quantitative data tend to have numerical values. Data collection instruments can
be based on any of the following approaches:
1. Sample surveys which can be based on interviews, using a standard questionnaire. All
respondents are asked the same set of questions.
2. Direct observation using service statistics and other programme documents.
3. Self-administered questionnaire Not ideal for a less educated population and tends to
have a low response rate.
4. Secondary data sources (official records, census, official statistics, etc).
Advantages
Quantitative data tends to be flexible, reliable and allows for international comparisons. The
most commonly used software to analyse data are SAS, SPSS and Epi Info.
When using a quantitative approach, keep the following in mind:
Use a simple language so that respondents will be able to answer the questions without
any difficulty.
Train interviewers and field supervisors prior to administering the questionnaire.
Pre code the responses to facilitate transfer of information and analysis.
Avoid embarrassing questions.
Pretest the questionnaire before administering it.
Ask respondents the same questions that have been tested.
Add information from qualitative interviews.
Qualitative Data
Qualitative data can be collected using the following key approaches:
38
1. In depth Interviews - Usually there is a guide or a set of questions to facilitate collection

of information from respondents. The guide helps to standardise the questions being
asked so that there is uniformity in analysing the responses.
2. Focus Group Discussions Respondents are brought together for open discussions on a
set of issues prepared in advance. There is a facilitator who helps to guide the
discussions as well as a rapporteur who takes notes. It is recommended that focus group
discussions involve 8-10 participants, however, at the outer limits there should be no less
than 5 and no more than 12.
3. Direct Observation This is often used to assess service delivery points to determine
quality of service provision. It requires highly skilled observers and analysts such as
ethnographers.
4. Case Studies these normally concentrate on a small number of cases, which are
examined in depth. Case studies can examine one moment in time and one event or
processes that evolve over long periods of time.
5. Content Analysis of written materials - this is useful for analysing training materials.
Software for qualitative data analysis: Atlas Ti and QSRN 6.
Advantages of Quantitative and Qualitative Data:
Quantitative
Data is consistent and provides a basis
for national and international
comparisons
It is cost effective for collecting data
from a large population
Provides standardised responses
Suitable for collecting data from people
who are less educated
It is ideal for a large sample size
It is less time consuming.
Qualitative
It makes it possible to collect information
from respondents whose views are based
on gut feelings
Helps to probe social and cultural
attitudes
Allows for probing for unintended results
Allows assessment of goals that are not
amenable to quantitative analysis. For
example: empowerment, self-esteem,
negotiation skills, etc.
Tips for Quantitative and Qualitative Data

To ensure high quality of data, prepare written guidelines for data collection. The guideline will
ensure some degree of standardization in the data collection process. Also, pilot-testing should
not be done in an area where the questionnaire will be administered.
39
Coding
It may be useful to develop a codebook as part of designing a questionnaire. A numerical or
symbolic code may be assigned. For example, to find out the level of education of respondents,
the responses may be coded as follows:
(1) None
(2) Primary
(3) Secondary
(4) Tertiary
Data Analysis
Once data is collected, either qualitative or quantitative, the next step is analysing it. There are
various techniques for analysing both qualitative and quantitative data.
Analysing Qualitative Data
Qualitative data is often presented in a narrative form. It is not always feasible to assign a code
or even a numerical character to qualitative data. Instead qualitative data can be coded as
categories (thematic coding) and presented as a narrative.
The following shows how qualitative data can be categorised and presented:
1. Case Studies Based on narratives or interpretations of respondents understanding of
the workings and benefits of an intervention.
2. Process Analysis - Visual depiction of a programmes processes and outcomes.
3. Causal Flow Charts Shows how things work.
4. A decision Tree Model This graphically outlines the realm of choices and priorities
that go into decision- making.
5. Taxonomy A visual representation/diagram showing respondents relate categories of
language and meaning.
Analysing Quantitative Data
This usually involves mathematical calculations through the application of statistics. The most
commonly used statistics are descriptive and inferential statistics.
Descriptive Statistics
Descriptive statistics is the first step in quantitative analysis. Descriptive statistics are used to
describe the general characteristics of a set of data. Descriptive statistics include frequencies,
counts, averages and percentages.
This method is used to analyse data from monitoring, process evaluation and outcome /impact
evaluation.
Frequencies
A frequency denotes a univariate (single variable) number of observations or occurrences. For
example, when you say that 100 out of 200 homeless people have been given shelter you are
40
stating a frequency. When the frequencies related to a single variable (homeless people) are
listed together, it is known as a frequency distribution. For example, of the 300 people listed as
homeless, 100 were given shelter, 105 were sent for a medical check up, 95 received training in
basic skills, etc. Data for a single variable (homeless people) can be further disaggregated suc h
as of the 100 homeless people who were given shelter, 50 were females and 50 males. Of the
100 homeless, 10 were young people of between 10-18 years of age. This is known as bivariate
and multivariate.
Bivariate and multivariate frequencies can be classified and presented in a table format. This
display of labelled rows and columns is known as cross tabulation (See example below).
Number of Children by Disability Type
District
Hearing Learning
Caroni Educational District
Other
Physical
Visual
All
225
1,210
29
82
418
1,964
North Eastern Educational District
47
16
32
104
Port of Spain Educational District
80
2,054
65
52
172
2,423
South Eastern Educational District
86
620
84
97
318
1205
St George East Educational District
165
1,853
174
68
257
2,517
St Patrick's Educational District
123
665
94
90
332
1,304
Tobago Educational District
10
400
15
435
Victoria Educational District
54
275
63
400
750
7,124
471
400
1607
10,352
All Districts
Percentages
Percentages are calculated by dividing the frequency in one category by the total number of
observations then multiplying by 100:
Variable
Shelter
Medical Check Up
Males
Females
Total
Frequency
100
100
50
50
300
Percent (%)
33.3
33.3
16.7
16.7
100
41
Inferential Statistics
Inferential statistics allow the evaluator to make inferences about the population from which the
sample was drawn, based on probabilities or stratified sampling. Testing for statistical
significance helps to ensure the differences observed in data, however small or large, were not
due to chance.
Preparing an Evaluation Report
See Step Nine of Appendix I.
Developing a Solutions Framework
Sample of a Solutions Framework (Rapid Assessment of the Mediation Centres):
Time Frame
Person/Organization
Responsible
i. The Community Mediation Act should be

revisited and amended to enable the
High
Centres to discharge their roles and social
responsibilities with legitimacy.
Aug-Oct 2003
Office of the Attorney

General
ii. The Intake forms should be standardised

across Centres and reviewed to include
High
broad ranges for age and income which
would facilitate a higher response rate.
Sep-Oct 2003
Directors
Mediation Centres
iii. The Evaluation forms which have been

developed should be used by all Centres
and all clients using the mediation service Medium
should be encouraged to complete them as
fully as possible.
Sep-Oct 2003
Directors Mediation Centres
Recommendations
Priority
Level
References:
1. Peter H. Rossi et al, Evaluation: A Systematic Approach, Sage Publications, 1999
2. Susan Adamchak, A Guide to Monitoring and Evaluating Adolescent Reproductive
Health Programmes, Tools Series, 5 June 2000
3. Michael Q. Patton, Utilization Focused Evaluation, Sage Publications, 1978
4. Andrew A. Fisher et al, Handbook for Family Planning Operations Research Design,
Population Council, New York, 1998
5. Bertrand et al, Evaluating Family Planning Programmes, The Population Project,
September 1996
42

M&E Training Manual

Uploaded by

Copyright:

Available Formats

M&E Training Manual

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

M&E Training Manual

Uploaded by

Copyright:

Available Formats

A Training Manual

What is the Purpose of the Training Workshop? .................................................................... 3

Origins of Programme Evaluation.......................................................................................... 5

The Evaluation Process ........................................................................................................ 19

Time Series Design................................................................................................................ 32

Ministry of Social Development/Monitoring and Evaluation Division September 2003

Ministry of Social Development/Monitoring and Evaluation Division September 2003

A Training Manual on Monitoring and Evaluation Concepts, Tools and

to help in determining whether or not a programme should be continued, improved,

Ministry of Social Development/Monitoring and Evaluation Division September 2003

What is the Purpose of the Training Workshop?

Ministry of Social Development/Monitoring and Evaluation Division September 2003

The Training Programme

Ministry of Social Development/Monitoring and Evaluation Division September 2003

MODULE 1: DEFINING THE CONCEPTS OF MONITORING AND EVALUATION

Ministry of Social Development/Monitoring and Evaluation Division September 2003

Ministry of Social Development/Monitoring and Evaluation Division September 2003

Key Evaluation Issues

Is the intervention still relevant in relation to the

Programme design and effectiveness

To what extent has the intervention met the stated

How cost-effective were the various activities that

How well did the intervention adhere to established

Did the intervention result in changing the

Did the programme result in changes that were not

Can the intervention survive when funding ceases?

Ministry of Social Development/Monitoring and Evaluation Division September 2003

Ministry of Social Development/Monitoring and Evaluation Division September 2003

Ministry of Social Development/Monitoring and Evaluation Division September 2003

MODULE 2: PROGRAMME PLANNING AND DESIGN

Ministry of Social Development/Monitoring and Evaluation Division September 2003

do a needs assessment, consideration should be given to choosing the most appropriate

Post Intervention Data

Conceptual Model as an Integral Part of Programme Design

Goals and objectives

Ministry of Social Development/Monitoring and Evaluation Division September 2003

Development of selfemployment capacity

Establishing a Programme Theory

What are the key components of a Logical Framework?

One Caring Adult

To facilitate the partnering of an adult mentor with an at risk youth in order to

To provide a source of guidance and support for a minimum of 30 at risk youth in yr 1.

1.1 Advertise for mentors

Defining Goals and Objectives

Ministry of Social Development/Monitoring and Evaluation Division September 2003

What are Objectives?

Ministry of Social Development/Monitoring and Evaluation Division September 2003

Ministry of Social Development/Monitoring and Evaluation Division September 2003

There should be a clear relationship between the indicator and the

Relates to the stability of the measurement process. The same

Ministry of Social Development/Monitoring and Evaluation Division September 2003

The indicator must be consistent and should represent what is

There should be ready access to sources of data.

The indicator does not require sophisticated methods of

The social planner/evaluator must clearly communicate what is

The cost of data collection in terms of both human and financial

The data that is generated must be reliable and replicable.

Ministry of Social Development/Monitoring and Evaluation Division September 2003

Ministry of Social Development/Monitoring and Evaluation Division September 2003