Meta Evaluation of CoE Project and Programme Evaluations (2016-2019)
In the framework of its Quality Assurance of CoE Decentralised Evaluation Reports, the Council of Europe had mandated the CEval GmbH to perform an independent quality check of a sample of 32 CoE decentralised evaluation reports prepared by various Major Administrative Entities for the period 2016-2019. The overarching objective of this stocktaking is to support the establishment of an Evaluation Framework and the designing of a Quality Assurance System for the CoE.
Accordingly, the quality check informs about the extent to which the decentralised evaluations fulfil widely accepted international standards in terms of quality and gives recommendations on what aspects to improve; provides an overall assessment of the major strengths and weaknesses of the decentralised evaluation reports; identifies key findings, recommendations and lessons learnt to improve the performance of CoE institutional capacity-building; and finally provides recommendations for the establishment of a framework for quality assurance. The assessment has been conducted based on an analysis grid including all relevant evaluation criteria and indicators, with a detailed guidance, in order to assure inter-rater reliability and thus to provide for maximum objectivity of the results.
Sector evaluation of climate projects within the framework of the BMZ Special Initiative on Climate of the Hanns Seidel Foundation e.V. 2019
The Hanns Seidel Foundation (HSS) has been receiving funds from the BMZ Special Initiative on Climate since 2011 and has carried out project activities in the fields of climate, environment and energy in numerous countries since then. The measures in this area are intended to make a contribution to environmental protection and nature conservation as a whole, to combating and adapting to climate change, to sustainable energy supply and resource efficiency, and to sustainable and environmentally conscious agriculture and forestry. A basic strategy was developed from the context of Hanns Seidel Foundation’s current work abroad in the fields of climate, environment and energy and its attempts to find solutions to these topics at an international level. This strategy was evaluated by CEval GmbH using three projects in India, Bolivia and Namibia as examples.
From a methodological point of view, the evaluation included a theory-based part, which consisted of the reconstruction of the Theories of Change (ToC) of the three projects. In addition, the three selected projects were evaluated at project level, for which field phases in all three countries and interviews with partners and target groups have been conducted. In addition, the evaluation included a synthesis part which referred to the climate strategy of the Hanns Seidel Foundation in general and in which the results of the evaluations of the three projects were bundled and supplemented with information from the headquarters of the HSS in Munich.
Rapid Appraisal of two Moringa Investments
Moringa SCA SICAR is a private equity fund that has been established as a response to growing interest of the Financial and the Development sector on investments that provide for a financial, environmental and a social return at the same time. Moringa focuses on investments in agroforestry in African and Latin American Developing and Least Developed Countries. It applies agroforestry land use systems on a commercial basis with a nucleus commercial investment approach that integrates large numbers of neighboring smallholder farmers and processors into its business model through contractual arrangements. The Agroforestry Technical Assistance Facility (ATAF) was created by Moringa Partnership to provide technical assistance in relation to investments of the Fund with the goal to amplify and upscale positive environmental and social impacts triggered through Moringa investments.
CEval GmbH has conducted Baseline Rapid Appraisals for two Moringa Investments, focused specifically on the livelihood situation of the farmers and other relevant target groups, such as factory workers. Methodologically, the Rapid Appraisals followed a mixed-methods approach, which included a small survey with farmers for each Rapid Appraisal, as well as interviews with the relevant stakeholders. Rapid Appraisals were conducted for Moringa investments in Benin and Togo.
Evaluation of two school rehabilitationprojects in Nepal
During the two massive earthquakes in Nepal in 2015, several thousand schools got destroyed. Together with the consortium partners Helvetas and Swiss Red Cross, Caritas Switzerland built and rehabilitated 36 schools in the hard to reach district of Sindhupalchok. The Project encompassed trainings regarding disaster risk reduction, operation and maintainance, WASH (water, sanitation, hygiene) and social interaction with drop out students.
The final evaluation was based on three of the Core Humanitarian Standards (appropriateness and relevance, effectiveness and timeliness, preparedness and resilience) in order to draw lessons-learnt and give recommendations for further humanitarian outreaches.
The evaluation utilized different methods: next to interviews with the leadership of the project, representatives of national authorities, team members, head of schools and mayors; Focus Group Discussions gave voice to school management committees and teachers. A comprehensive survey among hundreds of students, parents and teachers enabled a in-depth quantitative analysis.
Central project evaluation of the project Competitive African Cashew Value Chains for Pro-Poor Growth
The cashew value chain offers African producing countries interesting opportunities to create jobs, increase business and economic income and improve food security. Up to the time of the evaluation, however, the potential had hardly been exploited. At 300 to 550 kilograms of raw nuts per hectare, the yields of African producers were far below the yields in India and Vietnam and the processing capacities and labor productivity of the farms were still too low, which meant that only a small proportion of the raw nuts were processed in the African producing countries. The “Competitive African Cashew Value Chains for Pro-Poor Growth” project of the Deutsche Gesellschaft für internationale Zusammenarbeit GmbH supported the expansion and intensification of the cashew value chain in the six countries Benin, Burkina Faso, Côte d’Ivoire, Ghana, Mozambique and Sierra Leone.
The project was based on four levels of action: (i) improving the supply of high-performance seeds/seedlings through research and educating producers through training and consulting, (ii) advising processors of cashew nuts and by-products to improve process efficiency, (iii) intensifying business relations and exchange between all participants in the value chain, and (iv) promoting national and regional framework conditions by advising the relevant ministries and intensifying regional exchange between all actors.
CEval GmbH was commissioned by GIZ with the central project evaluation. The evaluation was aligned with the OECD DAC criteria and thus examined the relevance, effectiveness, efficiency, impact, sustainability and coherence of project implementation. Methodologically, the evaluation was based on the available quantitative secondary data of the project and expanded on this by conducting guided interviews and focus group discussions with project managers, stakeholders and target group representatives. Due to the corona pandemic prevailing at the time of the evaluation, the project was implemented as a remote evaluation. The evaluators from CEval GmbH worked together with local consultants in the African countries, who carried out the data collection in the field.
Scientific advice to the GIZ Evaluation Unit on the development of aggregated impact indicators.
The Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH contributes in many ways to solving regional and global problems. These include mitigating the effects of climate change, fighting poverty and hunger, or creating equal access to education and employment for men and women. To record the contributions made by its projects, GIZ has a number of instruments and procedures for impact measurement and evaluation. However, these tools and procedures are mainly geared toward reporting to the BMZ and generating evidence to improve individual interventions (PEVs) or to further develop the strategic direction of GIZ in the respective sector (USEs). At the level of the overall organization, however, there had been a lack of comparable approaches for presenting the contributions of GIZ as a whole, e.g. to the public or parliament (esp. EEZs).
In order to make GIZ capable of providing information at this level as well in the future, the Monitoring and Evaluation Unit initiated a process in 2014 to develop aggregated indicators that allow statements to be made about the joint contribution of as many projects as possible in the various sectors in which GIZ is active.
Evaluation of a Capacity Development Project on Child Protection Systems in Ethiopia and India
Worldwide across all socio-economic backgrounds, cultures and religions, children are exposed to a wide spectrum of risk factors in their daily lives. Considering the complexity and interconnectedness of underlying causes, functioning child protection systems have proven to be key to address risk factors and prevent violence, exploitation and abuse. Kindernothilfe, jointly with Karl Kübel Stiftung, Child Fund and terre des hommes, initiated the project titled Development and implementation of institutional child safeguarding policies as well as networking with local child protection systems through at least 240 organizations in nine countries in Asia and Africa in 2014.
As the project ended in December 2018, the consortium commissioned the Center for Evaluation in August 2018 to conduct an evaluation to understand the impact of the project as well as retrieve learnings and recommendations for future interventions of similar character. A crucial part of the assignment was to retrospectively develop the Theory of Change (ToC) of the project in a participatory manner, integrating consortium members, national trainers as well as the target group and final beneficiaries in the process. The ToC guided the evaluation while being further validated during the course of the evaluation. Field visits took place in Ethiopia and South India in October 2018. In total, six partner organizations had been visited and interviews and focus group discussions had been hold. Children are included in the evaluation by making use of participatory research tools, such as Picture Talk and Smiley-face scales. Each field visit ended with a one-day workshop to which all participating organizations of the respective region had been invited. During the workshop, preliminary results and the ToC were presented, discussed and validated. Finally, the various data were analyzed, triangulated and the results presented to the consortium partners.
International Program for Development Evaluation Training (IPDET)
The International Program for Development Evaluation Training (IPDET) is an executive training program in development evaluation. Since 2018, it is managed jointly by the Center for Continuing Education at the University of Bern (Switzerland), the Center for Evaluation at Saarland University (Germany) and the Independent Evaluation Group of the World Bank.
IPDET aims to provide managers and practitioners in the field of evaluation with the tools required to evaluate policies, programs and projects at local, regional, national, and global levels, as well as to commission, manage and especially use those evaluations for decision-making. Further information on this year’s IPDET can be found on the website: https://ipdet.org/.
Development of an M&E framework, M&E support and facilitation of learning & exchange formats for the initiative “Improved livelihoods perspectives for young people in rural East Africa”
The German Federal Ministry for Economic Cooperation and Development (BMZ), Deutsche Stiftung Weltbevölkerung, Siemens Stiftung and Hanns R. Neumann Stiftung collectively intend to improve upon the prospects of youths in rural areas of East Africa. Within the joint initiative of Improved livelihoods perspectives for young people in rural East Africa they empower youths to actively shape developmental processes and to have their voices heard in the respective policy processes. Further, they intend to improve youths livelihoods through enhanced income generation and health conditions. Through this, they also aim to contribute to jointly fight root causes of migration. The initiative is implemented by three local implementing partners in Uganda, namely Action for Health Uganda, Hanns R. Neumann Stiftung Africa Ltd. and WHAVE Solutions.
The objective of this assignment was threefold: To develop an M&E Framework for the program, to support its monitoring and evaluation through the implementation of three assessments (baseline, mid-term and final assessment), and to facilitate learning and exchange during its pilot phase. For achieving these objectives, a participatory approach was employed, focusing on a strong stakeholder involvement from the planning stage through data collection to results interpretation. During the assessments a multi-method approach was applied, based on a quantitative quasi-experimental and a qualitative case-study design for achieving a comprehensive picture of the programs impact and sustainability, and eventually validating its intervention logic. While the quasi-experimental design comprises the collection of target and comparison group data from approx. 1,000 pax in total, the case-study design included interviews and Focus Group Discussions with key informants from implementing partners, target groups, local authorities and sector experts.
Results oriented monitoring of the technical cooperation measure COPLAN II and evaluation of the measure’s contribution towards a better alignment of development cooperation and foreign trade facilitation on a systemic level
Climate Change and the uninterrupted decline of fossil fuels affect different regions of the world in different ways. Many nations have thus formulated ambitious goals: Mexico for instance is determined to cover one third of its energy production by means of renewable energies by 2024. Despite such ambitions however environmental conservation has not been marked by major success to this date. This is partly due to a lack of dispersion of modern environmental technology. Against this background the GIZ Project Cooperation Platform Latin America North II (COPLAN II) scrutinizes central deficits that hinder international investment and cooperation such as a lack of professional expertise regarding modern environmental technology in small and medium enterprises (SMEs).
Within the scope of the present assignment CEval together with iSPO Institute supported COPLAN II in its monitoring and evaluation efforts. Starting point for any results oriented monitoring was typically an impact model with respective indicators and assumptions. Changes within the mechanisms of cooperation are however rarely or marginally observed under the monitoring component. This development of cooperation systems or the suitability of the relevant funding instruments are also examined as part of the evaluation component.
All in all, next to monitoring tasks and the compilation of results of individual measures the assignment at hand intended to also generate insights into COPLAN IIs contribution towards a better alignment of development cooperation and foreign trade facilitation as well as public-private sector cooperation on a systemic level.
