Evaluation of a Capacity Development Project on Child Protection Systems in Ethiopia and India
Worldwide across all socio-economic backgrounds, cultures and religions, children are exposed to a wide spectrum of risk factors in their daily lives. Considering the complexity and interconnectedness of underlying causes, functioning child protection systems have proven to be key to address risk factors and prevent violence, exploitation and abuse. Kindernothilfe, jointly with Karl Kübel Stiftung, Child Fund and terre des hommes, initiated the project titled Development and implementation of institutional child safeguarding policies as well as networking with local child protection systems through at least 240 organizations in nine countries in Asia and Africa in 2014.
As the project ended in December 2018, the consortium commissioned the Center for Evaluation in August 2018 to conduct an evaluation to understand the impact of the project as well as retrieve learnings and recommendations for future interventions of similar character. A crucial part of the assignment was to retrospectively develop the Theory of Change (ToC) of the project in a participatory manner, integrating consortium members, national trainers as well as the target group and final beneficiaries in the process. The ToC guided the evaluation while being further validated during the course of the evaluation. Field visits took place in Ethiopia and South India in October 2018. In total, six partner organizations had been visited and interviews and focus group discussions had been hold. Children are included in the evaluation by making use of participatory research tools, such as Picture Talk and Smiley-face scales. Each field visit ended with a one-day workshop to which all participating organizations of the respective region had been invited. During the workshop, preliminary results and the ToC were presented, discussed and validated. Finally, the various data were analyzed, triangulated and the results presented to the consortium partners.
International Program for Development Evaluation Training (IPDET)
The International Program for Development Evaluation Training (IPDET) is an executive training program in development evaluation. From 2018 to 2024, it was managed jointly by the Center for Continuing Education at the University of Bern (Switzerland), the Center for Evaluation at Saarland University (Germany), and the Independent Evaluation Group of the World Bank. Since 2021, IPDET has been a key implementing partner of the Global Evaluation Initiative (GEI).
IPDET aims to provide managers and practitioners in the field of evaluation with the tools required to evaluate policies, programs and projects at local, regional, national, and global levels, as well as to commission, manage, and especially use those evaluations for decision-making. Further information on IPDET can be found on the website: https://ipdet.org/.
Development of an M&E framework, M&E support and facilitation of learning & exchange formats for the initiative “Improved livelihoods perspectives for young people in rural East Africa”
The German Federal Ministry for Economic Cooperation and Development (BMZ), Deutsche Stiftung Weltbevölkerung, Siemens Stiftung and Hanns R. Neumann Stiftung collectively intend to improve upon the prospects of youths in rural areas of East Africa. Within the joint initiative of Improved livelihoods perspectives for young people in rural East Africa they empower youths to actively shape developmental processes and to have their voices heard in the respective policy processes. Further, they intend to improve youths livelihoods through enhanced income generation and health conditions. Through this, they also aim to contribute to jointly fight root causes of migration. The initiative is implemented by three local implementing partners in Uganda, namely Action for Health Uganda, Hanns R. Neumann Stiftung Africa Ltd. and WHAVE Solutions.
The objective of this assignment was threefold: To develop an M&E Framework for the program, to support its monitoring and evaluation through the implementation of three assessments (baseline, mid-term and final assessment), and to facilitate learning and exchange during its pilot phase. For achieving these objectives, a participatory approach was employed, focusing on a strong stakeholder involvement from the planning stage through data collection to results interpretation. During the assessments a multi-method approach was applied, based on a quantitative quasi-experimental and a qualitative case-study design for achieving a comprehensive picture of the programs impact and sustainability, and eventually validating its intervention logic. While the quasi-experimental design comprises the collection of target and comparison group data from approx. 1,000 pax in total, the case-study design included interviews and Focus Group Discussions with key informants from implementing partners, target groups, local authorities and sector experts.
Evaluation Synthesis: Evidences from Central Project Evaluations of Transitional Development Assistance Projects
As a bridge between humanitarian aid and longer-term development cooperation, transition assistance is becoming more and more important, especially against the backdrop of increased conflicts worldwide.
Consequently, the Evaluation Unit of GIZ had commissioned the CEval GmbH to conduct an evaluation synthesis of transitional development assistance (TDA) projects that have been implemented in the period from January 2013 to April 2021.
Subject to this evaluation synthesis had been nine central project evaluations (CPE) of TDA projects and one CPE of a regional EU co-financed project with similar thematic priorities, implemented in four countries. Thereby, the evaluation covered diverse countries and regions: South Sudan, Somalia, Haiti, Ukraine, Iraq, Turkey, Lebanon, and Jordan. In addition to the CPEs, the synthesis had been based on interviews with project leaders and, as possible, with the respective evaluators. The aim was to identify TDA related and context-specific factors of success and failure of the projects and to derive recommendations for action. Methodologically, the synthesis was based on a mixed-method approach, consisting of a combination of qualitative content analysis and QCA (fuzzy set qualitative comparative analysis).
Preparation of an AMPA overview for 20 multilateral organizations as well as two general overviews
The Annual Multilateral Performance Assessment (AMPA) is an instrument of the Swiss Agency for Development and Cooperation (SDC) for the annual performance assessment of its priority multilateral organizations. The aim is to assess these organizations as neutrally and independently as possible, with little resource input on the basis of existing and accessible information. AMPA is based on 1) MOPAN reports (external source), 2) the CCM Annual Reports (SDC assessment) and 3) self-assessments of multilateral organizations (e.g. scorecards).
As part of a consulting evaluation assignment, CEval has worked with the client to develop concrete measures to optimize the performance assessment.
Development of a systematic set of social inequality indicators in the context of German development cooperation
In the context of of the Sustainable Development (SDG 2030) goals, the United Nations have placed the reduction of social inequalities at the center of international development politics. Against this backdrop, altering social inequalities has gained importance in the field of German development cooperation. The evaluation of the impact of developmental projects on social inequalities requires a set of specific and valid indicators.
CEval has compiled a set of social inequality indicators and developed a systematic approach to categorize the indicators by vertical and horizontal dimensions of inequality, sectors and operational level. CEval has developed the adaption of relevant indicators to the context of german development cooperation and the projects of GIZ.
Meta-Evaluation of ADA Project and Programme Evaluations 2016-2018
Subject of the meta-evaluation were ADA evaluations of development cooperation projects and programmes implemented between 2016 and 2018. The objective was to assess the quality and usefulness of ADAs (internal and external) evaluations and to deliver recommendations on how to improve the ADA evaluation tool for the design and implementation of future evaluations.
A document analysis of the evaluation reports was at the heart of the meta-evaluation. Based on a detailed analysis protocol the quality of said reports was assessed. The use of such a protocol, which covered all relevant analysis dimensions, indicators and rating scales, guaranteed the transparency and reliability of the meta-evaluation results. In order to evaluate the usefulness for and use of the evaluations under study, their target audience (e.g. programme managers) were interviewed via guided individual and group interviews and in order to achieve the greatest possible representativeness of the results a semi-standardized online survey.
Meta-evaluation and synthesis of Christoffel Blindenmission evaluation reports
On behalf of the Christoffel Blindenmission (CBM), CEval conducted an evaluation and synthesis of 24 evaluation reports from 2016 and 2017. With its mandate to transform the lives of disabled people in the worlds poorest countries the project focus of CBM has been an exhaustive collaboration with as well as the promotion of local partners in developing countries and conflict areas.
The primary purpose of the evaluation and synthesis was to conduct a general assessment of the evaluation reports quality and summarize findings of those reports that pass the quality control. In a first step, to ensure homogeneity of data, the quality of reports was assessed systematically by creating a quality control analysis grid. Reports that fulfil minimum quality standards were then subjected to an in-depth analysis based on the evaluation questions of CBM. Main interest of the synthesis was to drew up general learnings along CBMs mandate areas and examine the actual contribution of joint projects to changes for CBM partners and clients. Lastly, by employing an online survey directed to end users of project evaluations, the evaluation inquired the usage and usefulness of the evaluation reports for CBM staff members in seven world regions.
Meta-Evaluation of Plan International evaluation reports
Plan International is working for many years now in Africa, Asia and the Americas to promote child rights and lift children out of poverty. The German National Office of Plan International was founded in 1989, and has been contributing to funding development projects and programs as well as funding and commissioning external evaluations. Plan International Germany decided to implement a meta-evaluation of evaluation reports conducted between 2013 and 2017 covering all geographical working areas.
The primary objectives of the meta-evaluation were to assess the strengths and weaknesses by analyzing the quality of the evaluation reports, to identify the success and risk factors that may affect their quality and to give recommendations. A particular focus was laid upon the influence of the type of evaluation (i.e. by whom it is coordinated and implemented) and the available resources and timeframe.
The meta-evaluation was based on the analysis of the evaluation reports. The document analysis was structured in two phases. While in the first phase (pre-rating) the completeness of the evaluation reports was assessed, in a second phase (in-depth analysis), a more comprehensive set of quality assessment criteria was applied on a sample of reports, to allow for a more detailed assessment of the validity of their findings and usefulness to their addressees. Comprehensive analysis tools were used to analyze the completeness of the reports as well as the validity of findings and the usefulness of evaluation reports.
Evaluation of APPEAR projects (programme periods I and II)
The Austrian Partnership Programme in Higher Education and Research for Development (APPEAR) is conceived, guided and financed by the Austrian Development Agency (ADA) to implement its “Higher Education and Scientific Cooperation” strategy in support of Higher Education and Research for Development on an academic institutional level. The funded projects are implemented by the Austrian Agency for International Mobility and Cooperation in Education, Science and Research (OeAD GmbH). APPEARs overall objective is to strengthen the institutional capacities in higher education, research and management through Academic Partnerships with Austrian higher education institutions and through scholarships. The partnerships aim to improve the general standards in higher education, research and management by means of knowledge sharing and designing innovative projects. The achievement of these objectives should finally contribute to effective and sustainable reduction of poverty in the partner countries.
The overarching goal of the evaluation was to assess the ongoing and completed projects of the APPEAR periods I and II. Thereby, it provided insights about the projects adherence to the APPEAR guidelines and basic principles during their implementation, and about the extent to which they were to achieve their objectives and results. Furthermore, the projects were assessed according to the OECD/DAC criteria Relevance, Effectiveness, Efficiency, Impact and Sustainability. The objective of the evaluation was to identify the strengths, weaknesses and challenges of the APPEAR projects and present conclusions, lessons learnt and recommendations for the remaining or future projects.
The evaluation made use of a variety of methods. Primary data was collected during project visits to Ethiopia and Uganda, interviews and focus group discussions with project partners in Austria as well as virtual interviews with staff and key stakeholders of two projects in Nicaragua. The data collection instruments also comprised semi-standardized surveys, programme and project documentations and previous programme evaluations.