MONITORING AND EVALUATION

 

 

The evaluation data continues beyond collecting and analysing data to determine mean values or degrees of satisfaction. After analysing the results, value judgments should be made according to evaluation criteria. Influential factors that have affected the results should be fully analysed to provide useful recommendations and lessons learned. This process is called data interpretation, which is the final step in the evaluation study process,  following data collection and analysis. The interpretation process involves two steps: evaluating the project based on the Five Evaluation Criteria and drawing conclusions based on those judgments. The first step involves evaluating a project based on the five criteria and specifying the factors influencing the results. The evaluator must identify the factors that inhibited the achievement of the objective if the effectiveness is low. The evaluator must present specific evidence drawn from survey results to all stakeholders when explaining hin...

The last step of making an evaluation plan is to bring all the contents of planning together in an “Evaluation Grid,” a table of evaluation work plan (refer to table 2-2-7 showing an example). All components in the evaluation grid are interrelated to each other, and help us develop the most appropriate work plan for conducting an evaluation. In the process of creating the evaluation grid, we always have to keep in mind what is the most appropriate way to answer evaluation questions. The evaluation grid should be used flexibly. There are no rules in the ways of description, and a new column (e.g., sampling method, gender ratio, etc.) can be added when necessary. The important thing is that the way of answering evaluation questions is clearly specified and the effective methods within limited sources are identified. After completing the grid, it is also important to properly reflect the contents of the grid on the questionnaire or a question sheet to actually conduct surveys. In the case...

(1) Types of Data Collection Methods The main methods of collecting data are shown below. Some methods can be applied to collect both quantitative and qualitative data depending on the structure of questions and responses. For example, data can be quantified in a questionnaire survey with response options prepared (e.g., 80% of respondents are satisfied), whereas only qualitative data is available if it is structured with open-ended questions Collecting data takes time and money, and thus the availability of existing credible data is the first thing to be investigated. Each method has merits and demerits, and the evaluator has to carefully select the appropriate methods for data collection. Table 2-2-3 and 2-2-4 show the characteristics of each data collection method for reference. In the questionnaire survey or the interview, results may easily be influenced by the way questions are formulated. Even though a questionnaire survey is conducted for the whole population, or for key respon...

The next step is to identify what data should be collected from what sources in order to answer evaluation questions. The task of identifying the necessary data is a part of the process of finding out the specific indicators for measuring a phenomenon. For instance, an evaluator should decide a method of measurement: for instance, whether “the richness” should be measured by the annual income or the number of livestock in particular situation of the project. There usually are several ways of measuring a phenomenon, and thus the most appropriate measurement should be selected. (1) Types of Data There are two types of information or data to answer evaluation questions: 1) quantitative data; and 2) qualitative data. The different nature of each type of data should be realized because it affects data collection and analysis methods in an evaluation study. (Refer to 2-2-4 for quantitative analysis and qualitative analysis methods) i). Quantitative Data is the data expressed in a numerical v...

(1) Confirming Target Values or Setting New Criteria The assessment of a project’s performance in and after mid-term evaluation needs achievement criteria (or target values) in order to judge whether the objectives were achieved as expected. Target values described in the indicators column can be utilized as such criteria . In case no target values are specified, an evaluator needs to set up the criteria for determining both the level of achievement and what to compare that achievement too. For instance, to evaluate a water safety level, one can apply the criteria for water quality set by the WHO. Another example is to apply the national average rate of infectious diseases to the target region to measure the changes after a health project. The evaluator and stakeholders have to fully discuss what criteria to use as a basis for judgment. (Refer to Box 2: Several Ways of Deciding Target Values.) It is also necessary to consider what criteria to use in assessing efficiency. For instance, ...

1) Levels of Evaluation Questions and Five Evaluation Criteria i) What is an ‘Evaluation Question’? Evaluation questions represent “what one wants to know through evaluation.” For instance, the question of “whether a project was (or will be) meaningful” is an example of an evaluation question and a common question for project evaluation. More focused questions to verify the value of a project will be: “Were effects produced by the project?”  or  “Were resources efficiently used?”  It is possible to formulate more specific questions focusing on the effectiveness of a project. In an irrigation project, for instance, evaluators can ask such questions as: “Was the production volume of crops increased?”  or  “Were farmers’ incomes increased?”  Starting from a general question, more specific questions should be developed to make the evaluation study operational. By doing so, more concrete survey methods as well as the necessary data, are easily identified. Develo...

Evaluation is a practical tool for effectively managing a project. There are three phases in evaluation: assessment of the project performance; value judgment based on the Five Evaluation Criteria; and Making recommendations, drawing lessons learnt and providing feedback.  The evaluation study basically consists of the following six steps. Confirmation of the evaluation’s purpose Organization of information on a target project Making plans for the implementation of the evaluation Collection and analysis of data Interpretation of data and putting the results together Reporting evaluation results

Criteria such as: 1) usefulness,  2) impartiality and independence,  3) credibility, and  4) participation of partner countries,    All of these criteria should be satisfied in order to provide appropriate information worth being utilized; all of these are important for conducting a ‘Good Evaluation.’ (1) Usefulness Evaluation results should be clear and useful in order to be utilized in the decision-making of the organization. The objectives of the evaluation should be made clear, and for feedback to be effective, it is desirable that the evaluation should reflect the needs of envisioned users. Information regarding evaluation results is utilized in various stages during the project cycle and it is also utilized in the formulation of effective strategy, so it is essential that information is readily accessible, and implementation of the evaluation is just in time. (2) Impartiality and Independence Evaluation should be impartially conducted in a neutral set...

Feedback to the public is a process for JICA to fulfill its accountability which is one of the purposes of JICA’s evaluation. Accountability means more than a commitment to publishing evaluation results. It requires a system in which the ‘entrustee’ of such undertakings (JICA) gives a full account to the ‘entruster’ (taxpayers) on how it has implemented those undertakings in a responsible manner whereby the entruster can judge the entrustee’s performance. Also, since cooperation projects need to be carried out jointly with the countries receiving the aid, feedback for those concerned and the wider public of these countries is also important.   The accountability requirements include clear cooperation objectives, transparency in the organization’s decision making process and efficient use of inputs, and accurate measurement of the achievements resulting from the project.  Ensuring accountability demands the disclosure of evaluation information with quality that m...

Feedback for Project Management and Operation Feedback for project management and operation involves a process of using evaluation results as well as the lessons and recommendations obtained to improve the planning and implementation of projects. This type of feedback can be further classified into feedback to improve the decision-making process and feedback for the learning process of the concerned parties. i) Feedback to improve the decision-making process Feedback to improve the decision-making process involthe ves direct use of evaluation results in making decisions concerning the target project. In most cases, this process forms a part of the project management procedures by the department responsible for the project. For example, the results of the ex-ante evaluation serve as an important reference for deciding whether the project in question should be executed, while those of the mid-term evaluations are considered to decide whether to make a revision of the original project pla...

JICA regards the function of evaluation feedback as twofold: 1) feedback for project management and operation and 2) feedback for the public.

JICA’s current evaluation system involves four main parties: the Evaluation Study Committee, the Advisory Committee on Evaluation, the Office of Evaluation and Post Project Monitoring and the project implementation departments (departments and overseas offices responsible for project operation). The principal roles of these respective parties are as follows: i) Evaluation Study Committee The committee is headed by the JICA Vice President in charge of planning and evaluation and is composed of managing directors of related departments. The committee examines and discusses JICA’s basic evaluation policies as well as the methods for giving evaluation 10 feedback. Under this committee, an “Evaluation Study Working Group” has been set up to study, deliberate and report on related issues and problems. ii) Advisory Committee on Evaluation This committee is made up of external experts (academics, NGO members, journalists, etc.) well informed about issues concerning development aid and evaluati...

Development of JICA’s Evaluation System JICA has long been committed to improving its evaluation system. In July 1981, JICA set up the Evaluation Study Committee to deal with issues and challenges with its evaluation. The Committee has been leading JICA’s effort to develop new approaches and techniques for better evaluation. In April 1988, the Office of Evaluation was set up within the Planning Department, as a unit specializing in evaluation. In April 1990, the Office was reorganized as the Evaluation and Post Project Monitoring Division, and then put under direct supervision of the President in October 1996 for greater independence of evaluations. In January 2000, the Office of Evaluation and Post Project Monitoring was merged again with the Planning and Evaluation Department as a step to enhance the feedback of evaluation results for better project planning. In an effort to ensure objectivity and transparency in evaluation, an Advisory Committee on Evaluation composed of external ex...

 As is explained in the previous section, JICA’s Project Evaluation includes such activities as assessment of performance, value judgment based on Five Evaluation Criteria, and feedback of recommendation and lessons learned to potential decision makers. In order to implement these activities effectively, the following steps are taken as shown below. An evaluation study consists of 3 major steps: planning, doing and reporting the results. Table1-2-1 summarizes major activities of each step as well as some issues to be considered. A detailed explanation follows in the pages referred to in the table. Making an evaluation plan is a process for determining what and how to implement the evaluation inline with its purpose. An evaluation study is usually limited in its budget and time, and therefore an effective and efficient way to conduct the study should be well developed. The major steps of evaluation planning are described below. These steps are interrelated with each other,...

Evaluation is a practical tool for effectively managing a project. Evaluation provides all stakeholders throughout all levels and stages of project management with information about the process. Such information includes what the needs of a target society are, whether a project is being carried out as planned, how a project is affecting a target society, or what the factors that affect project management are. JICA conducts ex-ante, monitoring, mid-term, terminal and ex-post evaluation throughout the operating cycle of a project, aiming at effective and efficient project implementation.  In ex-ante evaluation, the appropriateness and relevance of project strategy as well as its efficiency and effectiveness are examined. In addition, adequate indicators, target values and means of measurement (means of verification) are developed. Once a project starts, monitoring is conducted with those indicators and measurements, and some modification of activities and strategy are made ...

Evaluation is the act of making a value judgment backed up by evidence. Also, as explained in the previous section, the proper feedback of recommendations and lessons learned to the future operation is essential. It is not satisfactory only to conclude that “objectives are achieved” or rate a project according to some scale at the end of evaluation studies. There are three phases in JICA’s evaluation: 1) assessing the performance of a project; 2) making a value judgment on the project using Five Evaluation Criteria; and 3) making recommendations, drawing from the lessons learned, and feeding them back to the next stage. This section explains the basic concepts of each framework. (1) Assessing the Performance of a Project: Results, Implementation Process, and Causal Relationships Three kinds of assessment are indispensable for analyzing the actual situation and the performance of the project. Measurement of results is to see what is achieved in a project and to what extent (o...

(1) Usefulness Evaluation results should be clear and useful to be utilized in the organisation's decision-making. The objectives of the evaluation should be made clear, and for feedback to be effective, it is desirable that the evaluation should reflect the needs of envisioned users. Information regarding evaluation results is utilized in various stages during the project cycle, and it is also utilized in the formulation of an effective strategy, so it is essential that information is readily accessible. Implementation of the evaluation is just in time. (2) Impartiality and Independence Evaluation should be impartially conducted in a neutral setting. To ensure the credibility of the evaluation information, it is indispensable to assure impartiality in providing standardized analysis of evaluation results. It is also important to gather a broad range of information from various stakeholders as well as from the particular persons and organizations concerned. In addition, the guarant...

Evaluation can be classified by evaluators as follows: i) Evaluation by third parties (External Evaluation). In order to improve the quality and objectivity of its evaluation, JICA entrusts a certain portion of its evaluation studies to external third parties that were not involved in the planning and implementation of the projects to be evaluated. For the same reason, JICA also includes those that have high expertise in the targeted fields for evaluation, such as universities, research institutions, academics, consultants, etc. ii) Evaluation by JICA (Internal Evaluation) In order to derive lessons and recommendations that meet the actual condition or needs of recipient countries, this evaluation is conducted mainly by JICA with the knowledge of those systems and other things that surround a project or an issue. JICA also promotes the review of such internal evaluation results by third parties (academics, journalists, NGOs, etc.) with expertise in development assistance and familiarit...