Skip to main content

Framework of Evaluation

Evaluation is the act of making a value judgment backed up by evidence. Also, as explained in the previous section, the proper feedback of recommendations and lessons learned to the future operation is essential. It is not satisfactory only to conclude that “objectives are achieved” or rate a project according to some scale at the end of evaluation studies.

There are three phases in JICA’s evaluation: 1) assessing the performance of a project; 2) making a value judgment on the project using Five Evaluation Criteria; and 3) making recommendations, drawing from the lessons learned, and feeding them back to the next stage. This section explains the basic concepts of each framework.

(1) Assessing the Performance of a Project: Results, Implementation Process, and Causal Relationships Three kinds of assessment are indispensable for analyzing the actual situation and the performance of the project. Measurement of results is to see what is achieved in a project and to what extent (or to see the validity of plans and target values in the case of ex-ante evaluation). Examination of implementation process is to analyze what is happening in the process towards its achievement and how it affects the performance (or to see whether the process is well planned in the case of ex-ante evaluation). Lastly, examination of causal relationships is to examine whether the project performance resulted from project intervention by looking at causal relationships between the project and whatever effects (or to see the validity of project designs in the case of ex-ante evaluation).

i) Measurement of Results (or Assessment of Indicators and Target Values in Ex-ante Evaluation) The measurement of results is for the purpose of understanding what the project has achieved and for the purpose of verifying whether it has been achieved as expected in mid-term, terminal and ex-post evaluation. The achievement level of objectives (e.g., project purpose and overall goal), outputs, and inputs are measured at the time of evaluation, and those results are compared with what has been planned. In ex-ante evaluation, the evaluator examines the appropriateness of the contents of plans. It is also necessary to see the validity of the target values in comparison with the baseline data.

ii) Examination of Implementation Process The examination of implementation process is for the purpose of investigating the whole process of activities and what is happening during project implementation. In evaluation after mid-term, the evaluator examines the dynamism within a project by focusing on such aspects as whether its activities are carried out as planned, whether a project is properly managed, whether those concerned with a project get along with each other, or how the beneficiaries perceive the situation. Also, any factors in the implementation process that may have influenced the achievement of its objectives need to be examined.

In ex-ante evaluation, the contents of project plans are examined with attention to the implementation process. This process includes the relevance of activities to produce intended outputs and objectives, the appropriateness of the management system, and the relationships with the beneficiary and relevant organizations.

Information about the implementation process often becomes supporting evidence for efficiency and effectiveness, and thus may be useful for analysis of the hindering or contributing factors of project’s success. In the case of ex-ante evaluation, the information can be utilized to avoid obstacles in advance, and to re-design the original plans or the planning of a similar project in the case of mid-term, terminal and ex-post evaluation. 

iii) Examination of Causal Relationships The examination of causal relationships is for the purpose of investigating whether project purpose and overall goal are or will be achieved as a result of project implementation. A project is merely one intervention for a whole society and thus, there always are influences caused by other factors. Even though some effects are observed as expected, there might be other influential factors that are not related to the project’s implementation. The evaluator should rule out other possible explanations in order to determine whether the project, and not other factors, caused the effects.

In order to examine causal relationships between effects and a project, different approaches from the ones used for performance assessment will be adopted. Those approaches include the method of comparing a target group (with a project) with a control group (without a project), and seeing changes before and after a project.

It is indispensable to examine the project performance in those three ways. In order to improve project operation, some hindering factors should be identified especially through an examination of the implementation process and causal relationships. In the case that some problem is identified in the implementation process, a management issue might be at the problem’s root. Or, when the causal linkage has not properly been constructed, the responsibility might be attributed to the planning itself.

Also, “monitoring” plays an important role with respect to evaluation. Monitoring is routine work and provides information for re-designing a project’s plan if necessary. The monitoring information contains quite a few data on “results” and “implementation process” and thus, it is an important data source for evaluation study. If monitoring is not carried out appropriately, necessary data and information for evaluation might be lacking.

(2) Value Judgment Based on Five Evaluation Criteria Evaluation is undertaken for the purpose of making a value judgment based on the achievement of results. JICA adopted Five Evaluation Criteria for conducting an evaluation (mainly project evaluation), which was proposed by the Development Assistance Committee (DAC) at the Organization for Economic Cooperation and Development (OECD) in 1991. These five criteria, as shown below, are meant to be used for evaluating development assistance activities from a comprehensive range of criteria. 

i) Relevance A criterion for considering the validity and necessity of a project regarding whether the expected effects of a project (or project purpose and overall goal) meet with the needs of target beneficiaries; whether a project intervention is appropriate as a solution for problems concerned; whether the contents of a project is consistent with policies; whether project strategies and approaches are relevant, and whether a project is justified to be implemented with public funds of ODA. 

 ii) Effectiveness A criterion for considering whether the implementation of project has benefited (or will benefit) the intended beneficiaries or the target society. 

 iii) Efficiency A criterion for considering how economic resource/inputs are converted to results. The main focus is on the relationship between project cost and effects. 

iv) Impact A criterion for considering the effects of the project with an eye on the longer term effects including direct or indirect, positive or negative, intended or unintended. 

 v) Sustainability A criterion for considering whether produced effects continue after the termination of the assistance. Five Evaluation Criteria are used to evaluate the value of conducting a project from a comprehensive point of view. The evaluator examines the effects of a project as well as the appropriateness of the volume of resources for producing the effects (i.e. efficiency). The value of conducting a project would possibly decrease if it costs more than necessary even though it produced enormous effects, or if effects were limited despite the fact that a large scale of inputs were used. Also, in order to judge the validity of an aid intervention, the relevance of strategies is examined as well as the sustainability of the effects after the termination of the assistance. Thus, evaluating a project using the five criteria makes it 22 possible to see the value of a project from different viewpoints, and as a result, various factors that influence the success or the failure of the project can be specified. When evaluating projects, whether to assess performance or to foresee the future, perspectives will differ depending on the timing of evaluation study in operating cycle. Focus of each criterion also will vary among projects. However, JICA decided to basically cover all five criteria in any type of evaluation study. 

 (3) Make Recommendations, Learn Lessons and Feedback Results of an evaluation are meaningful only when they are utilized. JICA recognizes the evaluation as a practical management tool, and therefore specific recommendations should be made and lessons should be drawn from the evaluation results. The recommendations and lessons learned have to be fed back to relevant departments of JICA and related organizations so that they will be utilized for the improvement of the target project or future operations. 

It is indispensable to specify the hindering or contributing factors influencing a project when making such recommendations and understanding what has been learned. For instance, when the project is found to be not effective, the hindering factors should be identified by examining both the implementation process and the causal relationships, and based on those factors, recommendations or lessons learned should be proposed. As another example, if it turned out that there was a problem in the arrangement of staff in the implementing agency, some recommendations to change the situation would be needed. 

In the case that evaluation results tell us the need for another set of outputs to achieve intended effects, recommendations to re-design the outputs should be proposed. In summary, there are three phases in evaluation: 1) assessing performance; 2) making a value judgment about the project; and 3) making recommendations and drawing lessons learned by analyzing the hindering or contributing factors, and feeding them back. It is not possible to conduct useful evaluation without any of these phases. 23 



 

Labels:

Comments

Post a Comment

Planning, Monitoring and Evaluation

Monitoring Definition

Monitoring - Definition  Monitoring is the systematic and continuous collection, analysis, and use of information for quality control and decision-making related to the projects. It involves the ongoing and routine review of projects to verify whether they are developing according to plan and budgetary requirements and whether adjustments may be needed to achieve intended goals.  Purpose of Monitoring  The primary aim of monitoring is to improve the quality of projects during implementation. Through monitoring, ineffective efforts are identified and rectified, and remarkably effective actions are appreciated.  Monitoring enables us to answer these questions and then make important decisions to improve the quality of projects and their implementation:  Are project inputs (finance, personnel, materials, etc.) available on time and in the right quantities and quality?  Are activities being implemented according to the project design and on schedule  Ar...
1 comment
Read more

Definition (Planning)

Planning can be defined as the process of setting goals, developing strategies, outlining implementation arrangements, and allocating resources to achieve those goals. It is important to note that planning involves looking at several different processes: Identifying the vision, goals, or objectives to be achieved. Formulating the strategies needed to complete the vision and goals Determining and allocating the resources (financial and other) required to achieve the vision and goals Outlining implementation arrangements, which include the arrangements for monitoring and evaluating progress toward achieving the vision and goals The expression “failing to plan is planning to fail.” While it is not always true that those who fail to plan will eventually fail in their endeavors, there is strong evidence to suggest that having a plan leads to greater effectiveness and efficiency. Not having a plan—whether for an office, program or project—is in some ways similar to attempting to build a hous...
Post a Comment
Read more

Objectives of Project Evaluation

It is impor tant to evaluate the outcomes that a project achieves and to feedback the evaluation results, lessons, and recommendations obtained for more effective and efficient implementation of development assistance. The harsh economic and fiscal situations at home have generated a strong call to the organization for more effective and efficient implementation and to ensure accountability. The enhancement of evaluation has drawn attention as one of a major improvement measures. In addition, there are changes in the political landscape such as the adoption of public sector evaluation by ministries and the reorganization of agencies into Independent Administrative Institutions (IAI) that ask for improvements of the evaluation system.  Evaluation is a tool for judging as objectively as possible the relevance and effectiveness of organization’s cooperation activities at four different stages during the project cycle: ex-ante, mid-term, terminal, and ex-post. The primary objectivethe...
Post a Comment
Read more

Database-Based Monitoring/Tracking

In this approach to monitoring, achievements in terms of numbers (of inputs, activities and outputs) are collected and stored in the project/central databases regularly and presented in an identified frequency against the contractual targets in order to establish whether the project . is leading towards achieving its targets. This approach to monitoring authenticates the progress of the project toward the intended targets. It is important to remember the following when conducting database-based monitoring: The reports drawn on the overall target population, but not sampled; Mostly captures quantities only; Not field-based. Usually performed in the capital office; Types-and-scope-of-monitoring, the officer/staff are responsible for the final products; however, database staff must coordinate necessary data gathering, storing, and analysis. M&E practices two types of tracking under this approach to monitoring in general: Indicator Tracking: Tracking contractual indicators is cruc...
Post a Comment
Read more

Step 4: Creating the Vision of the future

Based on the problem analysis, stakeholders should engage in a process of formulating solutions. This exercise may simply involve rewording the problems and their causes into positive statements or objectives. However, stakeholders should first engage in a visioning process before rewording the problems. The aim of this process is to visualize what the future would look like if the problems were resolved. The benefits of doing a visioning process before rewording the problems include the following: Visioning brings energy to the group. Rather than immediately beginning another detailed process of working on each problem, groups can be energized by thinking positively about what the future would look like if these problems were solved. This exercise encourages creativity and helps ensure that the process is not too analytical and methodical. TIP  Visioning process to have prior knowledge or understanding of the results chain or logical framework model. In fact, in the initial stages...
1 comment
Read more

Types and Scope of Monitoring

  Types and Scope of Monitoring  While there are several types of monitoring, M&E mainly focuses on performance monitoring, that is, the monitoring of project inputs, activities/processes, outputs, and outcomes in the field.              Inputs: checking that the resources needed for the intervention are mobilized as planned. Example Indicators: The tools are available in sufficient quality and quantity to enable an efficient CFW process The funds are available on time to the money vendors to enable timely cash distribution to the beneficiaries Activities/Processes: checking that project activities are carried out as planned; assessing the implementation method. Example Indicators:     A complaints response mechanism is operational in the areas of interventio % of selected households meets the targeting criteria  % of targeted households receiving the correct amount  % households reporting good or fair s...
Post a Comment
Read more

Planning Process

This chapter provides step-by-step guidance on how to undertake planning for results. It focuses on the tasks involved in planning for desired results and includes  considerations  for operationalizing results. As noted earlier, monitoring and evaluation are closely related to planning. Therefore, in planning, it is essential to consider not only intended results   but also how results, and the process of achieving them, will be monitored and evaluated. In particular, planning needs to ensure that planned  gives initiatives  are evaluation-ready. Planning can be done in many different ways. This chapter is designed to make the persons involved in planning more comfortable with the main steps involved in preparing a plan that can be implemented, monitored, and evaluated. The actions and approaches recommended generally apply to all planning processes, whether for a global, regional, or country program; a project; or ...
Post a Comment
Read more

Step 5: Creating Dradft Results Map

This step provides guidance on how to create a draft results map using what is commonly referred to as a ‘results mapping technique’.  Developing the draft results map can be time- consuming but is extremely worthwhile. The fundamental question that stakeholders in the planning session should answer is “What must be in place for us to achieve the vision and objectives that we have developed in the particular problem area?” Creating a set of positive results A good starting point in creating a results map is to take each major problem identified on the trunk of the problem tree and reword it as the immediate positive result with longer-term positive results or effects. For example, if the problem were stated as “low public confidence and involvement in governance” the immediate positive result could be “greater public confidence and involvement in governance.” This could lead to longer term positive results such as “wider citizen participation in elections, particularly by women, in...
Post a Comment
Read more

Types of Project Evaluation

The Ministry of Foreign Affairs “Report on Improvement of ODA Evaluation System,” released in March 2000, classified ODA evaluation into three levels: policy-level, program-level, and project-level, as shown in Figure 1-1-2. The report called for enhancement of policy- and program-level evaluation. The Ministry of Foreign Affairs conducts policy-level evaluations such as Country Assistance Program and Sector-specific Initiatives, as well as program-level evaluation by sectors and schemes. JICA conducts program- and project-level evaluations. How to evaluate JICA’s management cycle is illustrated in Figure 1-1-3. There are two cycles: the “small cycle” (project cycle) and the “large cycle” (program cycle)-The information source is from JICA. 
1 comment
Read more

Finalizing the result frame work

At this stage of the process, stakeholders should be ready to begin converting the results map into a results framework. In many situations, a smaller group of stakeholders is engaged in this undertaking. However, the wider group can participate in preparing a rough draft of the framework, using simple techniques and without going into the details and mechanics of RBM terminologies CREATING THE DRAFT RESULT FRAMEWORK Table 5 provides a starting point for converting the results map into a draft framework for  programme and project documents. It shows how to translate some general terms and questions used in the planning session into the common programming language. The table can be used to produce an initial draft of the results framework with all or most stakeholders. It can be particularly helpful at the project level or in situations involving a diverse group of stakeholders. FORMULATING STRONG RESULTS AND INDICATORS Having a smaller group of persons with greater familiarity wit...
Post a Comment
Read more