Evaluation Synthesis: Evidences from Central Project Evaluations of Transitional Development Assistance Projects
As a bridge between humanitarian aid and longer-term development cooperation, transition assistance is becoming more and more important, especially against the backdrop of increased conflicts worldwide.
Consequently, the Evaluation Unit of GIZ had commissioned the CEval GmbH to conduct an evaluation synthesis of transitional development assistance (TDA) projects that have been implemented in the period from January 2013 to April 2021.
Subject to this evaluation synthesis had been nine central project evaluations (CPE) of TDA projects and one CPE of a regional EU co-financed project with similar thematic priorities, implemented in four countries. Thereby, the evaluation covered diverse countries and regions: South Sudan, Somalia, Haiti, Ukraine, Iraq, Turkey, Lebanon, and Jordan. In addition to the CPEs, the synthesis had been based on interviews with project leaders and, as possible, with the respective evaluators. The aim was to identify TDA related and context-specific factors of success and failure of the projects and to derive recommendations for action. Methodologically, the synthesis was based on a mixed-method approach, consisting of a combination of qualitative content analysis and QCA (fuzzy set qualitative comparative analysis).
Preparation of an AMPA overview for 20 multilateral organizations as well as two general overviews
The Annual Multilateral Performance Assessment (AMPA) is an instrument of the Swiss Agency for Development and Cooperation (SDC) for the annual performance assessment of its priority multilateral organizations. The aim is to assess these organizations as neutrally and independently as possible, with little resource input on the basis of existing and accessible information. AMPA is based on 1) MOPAN reports (external source), 2) the CCM Annual Reports (SDC assessment) and 3) self-assessments of multilateral organizations (e.g. scorecards).
As part of a consulting evaluation assignment, CEval has worked with the client to develop concrete measures to optimize the performance assessment.
Development of a systematic set of social inequality indicators in the context of German development cooperation
In the context of of the Sustainable Development (SDG 2030) goals, the United Nations have placed the reduction of social inequalities at the center of international development politics. Against this backdrop, altering social inequalities has gained importance in the field of German development cooperation. The evaluation of the impact of developmental projects on social inequalities requires a set of specific and valid indicators.
CEval has compiled a set of social inequality indicators and developed a systematic approach to categorize the indicators by vertical and horizontal dimensions of inequality, sectors and operational level. CEval has developed the adaption of relevant indicators to the context of german development cooperation and the projects of GIZ.
Meta Evaluation of CoE Project and Programme Evaluations (2016-2019)
In the framework of its Quality Assurance of CoE Decentralised Evaluation Reports, the Council of Europe had mandated the CEval GmbH to perform an independent quality check of a sample of 32 CoE decentralised evaluation reports prepared by various Major Administrative Entities for the period 2016-2019. The overarching objective of this stocktaking is to support the establishment of an Evaluation Framework and the designing of a Quality Assurance System for the CoE.
Accordingly, the quality check informs about the extent to which the decentralised evaluations fulfil widely accepted international standards in terms of quality and gives recommendations on what aspects to improve; provides an overall assessment of the major strengths and weaknesses of the decentralised evaluation reports; identifies key findings, recommendations and lessons learnt to improve the performance of CoE institutional capacity-building; and finally provides recommendations for the establishment of a framework for quality assurance. The assessment has been conducted based on an analysis grid including all relevant evaluation criteria and indicators, with a detailed guidance, in order to assure inter-rater reliability and thus to provide for maximum objectivity of the results.
Meta-evaluation and synthesis of Christoffel Blindenmission evaluation reports
On behalf of the Christoffel Blindenmission (CBM), CEval conducted an evaluation and synthesis of 24 evaluation reports from 2016 and 2017. With its mandate to transform the lives of disabled people in the worlds poorest countries the project focus of CBM has been an exhaustive collaboration with as well as the promotion of local partners in developing countries and conflict areas.
The primary purpose of the evaluation and synthesis was to conduct a general assessment of the evaluation reports quality and summarize findings of those reports that pass the quality control. In a first step, to ensure homogeneity of data, the quality of reports was assessed systematically by creating a quality control analysis grid. Reports that fulfil minimum quality standards were then subjected to an in-depth analysis based on the evaluation questions of CBM. Main interest of the synthesis was to drew up general learnings along CBMs mandate areas and examine the actual contribution of joint projects to changes for CBM partners and clients. Lastly, by employing an online survey directed to end users of project evaluations, the evaluation inquired the usage and usefulness of the evaluation reports for CBM staff members in seven world regions.
Meta-Evaluation of Plan International evaluation reports
Plan International is working for many years now in Africa, Asia and the Americas to promote child rights and lift children out of poverty. The German National Office of Plan International was founded in 1989, and has been contributing to funding development projects and programs as well as funding and commissioning external evaluations. Plan International Germany decided to implement a meta-evaluation of evaluation reports conducted between 2013 and 2017 covering all geographical working areas.
The primary objectives of the meta-evaluation were to assess the strengths and weaknesses by analyzing the quality of the evaluation reports, to identify the success and risk factors that may affect their quality and to give recommendations. A particular focus was laid upon the influence of the type of evaluation (i.e. by whom it is coordinated and implemented) and the available resources and timeframe.
The meta-evaluation was based on the analysis of the evaluation reports. The document analysis was structured in two phases. While in the first phase (pre-rating) the completeness of the evaluation reports was assessed, in a second phase (in-depth analysis), a more comprehensive set of quality assessment criteria was applied on a sample of reports, to allow for a more detailed assessment of the validity of their findings and usefulness to their addressees. Comprehensive analysis tools were used to analyze the completeness of the reports as well as the validity of findings and the usefulness of evaluation reports.
Evaluation of APPEAR projects (programme periods I and II)
The Austrian Partnership Programme in Higher Education and Research for Development (APPEAR) is conceived, guided and financed by the Austrian Development Agency (ADA) to implement its “Higher Education and Scientific Cooperation” strategy in support of Higher Education and Research for Development on an academic institutional level. The funded projects are implemented by the Austrian Agency for International Mobility and Cooperation in Education, Science and Research (OeAD GmbH). APPEARs overall objective is to strengthen the institutional capacities in higher education, research and management through Academic Partnerships with Austrian higher education institutions and through scholarships. The partnerships aim to improve the general standards in higher education, research and management by means of knowledge sharing and designing innovative projects. The achievement of these objectives should finally contribute to effective and sustainable reduction of poverty in the partner countries.
The overarching goal of the evaluation was to assess the ongoing and completed projects of the APPEAR periods I and II. Thereby, it provided insights about the projects adherence to the APPEAR guidelines and basic principles during their implementation, and about the extent to which they were to achieve their objectives and results. Furthermore, the projects were assessed according to the OECD/DAC criteria Relevance, Effectiveness, Efficiency, Impact and Sustainability. The objective of the evaluation was to identify the strengths, weaknesses and challenges of the APPEAR projects and present conclusions, lessons learnt and recommendations for the remaining or future projects.
The evaluation made use of a variety of methods. Primary data was collected during project visits to Ethiopia and Uganda, interviews and focus group discussions with project partners in Austria as well as virtual interviews with staff and key stakeholders of two projects in Nicaragua. The data collection instruments also comprised semi-standardized surveys, programme and project documentations and previous programme evaluations.
Evaluation of CAPAQUA
CAPAQUA (Development of Educational and Research CAPacity in Eastern Africa for the Sustainable Management of AQUAatic Ecosystems) aims at enhancing the capacity of Eastern African Higher Education, Science and Technology (HEST) institutions. CAPAQUA is jointly implemented by the University of Natural Resources and Life Sciences (BOKU), Vienna, the IHE Delft Institute for Water Education, the Egerton University (EGU), the Addis Ababa University (AAU), the Bahir Dar University and the Ethiopian Institute for Agricultural Research, National Fish and Other Aquatic Lives Research Center (EIAR-NFALRC). CAPAQUAs overall objective is to foster sustainable management of aquatic ecosystems and their resources in order to improve livelihoods in Eastern Africa and to contribute towards the achievement of the Sustainable Development Goals.
The main purpose of the evaluation was to identify successes as well as weaknesses of the project, and to provide recommendations for the project partners to improve the relevance, effectiveness, efficiency and sustainability of possible future CAPAQUA projects. The evaluation was also analysing the extent to which the cross-cutting issue gender mainstreaming was applied and identified the major factors influencing the achievement or non-achievement of the expected project results and objectives. In order to assess the project, the evaluation focussed on the project period from October 2015 to September 2018.
A rich set of mainly qualitative data was collected by means of in-depth interviews and focus group discussions with different stakeholders, programme participants, graduates as well as staff. Programme and project documentations provided secondary data and was complemented by primary data collected during focus group discussions. The geographic focus of the evaluation was the Eastern African region, with emphasis on Uganda, Kenya and Ethiopia, where empirical data was collected during short field visits. Additionally, Austrian institutions were visited and stakeholders interviewed.
Development and implementation of an organization-wide results-based monitoring and evaluation system
The objective of the consultancy was to develop an organization-wide results-based monitoring and evaluation (M&E) system, and implement it together with the program staff. Due to the complexity of the results-model that had to be captured, comprising a total of 17 education and support programs, community development programs and services as well as the overarching organizational goals, the assignment had to be research-based. With this scope it is the first wholistic results-based M&E-system on organizational level in the field of Development Cooperation.
According to the system complexity and the large number of involved stakeholders (management, program staff, community actors, civil society organizations, German Federal Ministry for Economic Cooperation and Development), the assignment followed a multi-stage, participatory approach. Starting point was an inventory of the indicators, data collection instruments and analysis methods that were already used by Engagement Globals programs and services. Based on reconstructed and validated results-models, in the next step it was checked to what extent the effectiveness of the programs and services were covered therewith, and which information gaps still existed. Thereby, particular focus was laid upon the identification of the programs contributions to the achievement of the organizational goals of Engagement Global. In the following, the instruments were further developed together with the program staff and practically tested. The results of this development process was finally documented in a comprehensive M&E-concept and integrated into the organization-wide quality assurance system, accompanied with appropriate staff trainings.
Mid-Term Evaluation of the ASEAN Secretariat Post 2015 – Institutional Strengthening and Capacity Development” project (Jakarta, Indonesia)
The ASEAN Secretariat (ASEC) and the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) are jointly implementing the “ASEAN Secretariat Post 2015 & Organizational Development and Capacity Development” project funded by the Federal Foreign Office of Germany. The evaluated project phase began in June 2016 and, building on previous capacity development activities, supported the professionalization of the strategic administrative departments and staff units of the ASEAN Secretariat. It also worked with specialized departments that contribute to strengthening the ASEAN Economic Community. In order to assess the progress of the project phase, CEval was commissioned to conduct an interim evaluation based on the OECD-DAC criteria and extended, project-specific key questions. Taking into account the institutional framework of the ASEC and the diversity of stakeholders, the design of the contribution analysis and a multi-method approach was chosen.
For this purpose, guided interviews with key figures and semi-standardized online surveys of workshop participants were conducted. Project documentation and monitoring data supplemented empirical data and enabled triangulation. The aim of the interim evaluation was to gain insights into the current progress of the project and to enable learning for further project activities and future phases.