Development of an operational guidebook for integrating and using impact evaluation-based M&E approaches in the field of climate change adaption
Climate change adaption projects require methodologically elaborated Monitoring and Evaluation (M&E) Systems that are not limited to measuring Project Outputs but also Need to include outcomes and further Impacts as well as their attribution to the measures undertaken.
GIZ engaged the Center for Evaluation (CEval) for technical support to develop an operationally viable impact evaluation and provided examples for their applicability in adaptation projects. Results were presented in a guidebook for integrating and using impact evaluation-based M&E approaches in the field of climate change Adaptation. Target Groups of the Framework are ministries, donors, and other related institutions that are looking for the underlying Drivers of Adaptation and evidence-based findings to improve the design of future climate change adaptation projects.
External Evaluation of the Netherlands Fellowship Programme (NFP II) and Netherlands Initiative for Capacity Development in Higher Education (NICHE II)
On behalf of the Dutch Ministry of Foreign Affairs (MFA) the CEval implemented an external Evaluation of the Dutch Fellowship Programme (NFP II) and Netherlands Initiative for Capacity Development in Higher Education (NICHE II). Financed by the MFA and administered by the Netherlands Organisation for International Cooperation in Higher Education ‘EP-Nuffic’, these programs offer capacity development in higher education for developing partner countries.
The overall aim of NFP and NICHE is to help reduce quantitative and qualitative shortage of trained professional staff in developing countries and to build sustainable capacity within the poverty reduction Framework. The NFP provides Fellowships to mid-career professionals to participate in Trainings and postgraduate courses in the Netherlands. NICHE is a capacity building programme to improve the quality, relevance and gender sensitiveness of post-secondary education in NICHE-countries.
The evaluation covered the period of 2002-2015 while a focus was on the latest project phase since 2012 which put an additional emphasis on economic diplomacy and closer alignment with the four priority themes of the Dutch development cooperation strategy (water, sexual reproductive health and rights, security and rule of law and food security). The purpose of the evaluation was to account for the Netherlands funding provided for the two programs under the current project cycle and to gain lessons for future policy development and implementation. Within the scope of the evaluation, both programs were analysed regarding the OECD/DAC criteria of relevance, efficiency, effectiveness, impact and sustainability. Recommendations for possible optimization of the program were developed on the basis of the findings. The evaluation was conducted in cooperation with ACE Europe, a Belgian consultancy.
Program Evaluation “Support in Strengthening Regional Cooperation and Integration in the Field of Quality Infrastructure (QI) in SAARC”
With this regional Project, PTB Germany aims to reduce the technical barriers to trade by fostering cooperation among the SAARC countries (India, Pakistan, Bangladesh, Nepal, Buthan, Maldives, Afghanistan and Sri Lanka) in the field of metrology, accreditation and standardization. The Project targeted professional and Management staff of national QI-institutions as well as ministries relevant to the QI-System. Amongst others, measures lige Trainings and awareness raising Events have been taken and consultancy has been provided.
The evaluation combined qualitative and quantitative methods (mixed method design). A document Analysis and qualitative interviews with different stakeholders from all SAARC countries during a field visit to Bangladesh, Bhutan and India, and an online Survey of former metrology and accreditation Training participants in awareness raising Events have been carried out.
By statistical data and qualitative Content Analysis, the evaluation aimed to provide empirical evidence about the relevance, effectiveness, efficiency, impact and sustainability of the Project (DAC criteria) and the success factores for the observed results and change processes (Capacity Works criteria: strategy, cooperation, steering structure, processes, learning and innovation).
Thereby, it contributed to the further development of the Project approach and ist revision in view of future activities by PTB (in the region). The evaluation results had also been used as a Basis to decide on a follow-up project and on potentially necessary adaptions of its design.
Evaluation of the Projects “Sustainable Municipal Development through Partnership Projects” and “Support Program for Municipal Climate Protection and Climate Change Adaption Projects”
As part of Engagement Global – Service für Entwicklungsinitiativen (Service for Development Initiatives) the Service Agency Communities in One World in Germany ist the competence Center for local development cooperation. Founded in 2001 the Service Agency offers a platform for dialogue for municipalities in Germany and the South and supports development activities of local actors by providing them with Training, Information and advisory Services.
The two programs “Sustainable Municipal Development through Partnership Projects – Nachhaltige Kommunalentwicklung durch Partnerschaftsprojekte (Nakopa)” and “Support Program for Municipal Climate Protection and Climate Change Adaption Projects – Förderprogramm für kommunale Klimaschutz- und Klimaanpassungsprojekte (FKKP)” are core activities of the Service Agency. While Nakopa supports existing and new municipal partnership Projects in selected relevant fields of development cooperation, FKKP explicitly addresses municipal activities in the field of climate protection and climate change adaptation.
After the first Project cycle of three years has been completed, the current Evaluation aims at evaluating These two programs with respect to processual aspects of project Administration as well as Impacts of both the funded partnership Projects and the Support programs Nakopa and FKKP. Leading questions of this evaluation are guided by the OECD-DAC evaluation criteria and had been answered using the approach of contribution Analysis and documentary and empirical data. Hence, the evaluation was based on Monitoring data and Project documents on the one Hand and empirical data gathered through extensive partly standardized interviews and partly standardized online surweys with participating communities on the other hand.
Compliation of methodological approaches and experiences for impact Evaluations of Training and capacity building measures in the public sector
In cooperation with the Costa Rican institute CICAP, CEval counselled the GIZ program CADESAN in the Andean region. The regional program supports the development of capacity building measures on decentralization and modernization in the public sector.
The consultancy aimed at providing GIZ and its partners with an analysis and compilation of possible methodological approaches to evaluate the impacts of capacity building and training measures. To that end, various approaches and evaluation designs, their applicability in the respective realm as well as international experiences and practices were analyzed. The results were discussed with GIZ and its partners during a training designed for that purpose.
Project Evaluation of GIZ’s Sector Project Inclusion of persons with Disabilities
As part of the project evaluation, the extent to which the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) sector programme Inclusion of Persons with Disabilities was able to achieve developmental impacts was examined. Based on the OECD-DAC criteria, expert interviews and focus group discussions were conducted and M&E data from the sector programme and other documents were systematically analysed. This contributed to accountability towards the German Federal Ministry for Economic Cooperation and Development (BMZ), the publication of a short report met the requirements of the International Aid Transparency Initiative (IATI) and laid the foundation for planning a follow-up measure.
Evaluation of the Initiative “Schulen – Partner der Zukunft” (PASCH)
The initiative Schools Partners of the future (PASCH), founded in 2008, is part of the central initiatives of the German foreign cultural and educational policy within the current legislature period. PASCH is a worldwide network currently including around 1800 partner schools, which are linked to Germany and especially teach German. The initiative concentrates diversified measures, which are supposed to contribute to establish lively and long-term relations to Germany, and encourage schools, teachers and students to exchange their knowledge and ideas as well as to work together. PASCH is coordinated by the German Federal Foreign Office and is implemented by the Central Office for Schools Abroad, the Goethe Institute, the Pedagogic Exchange Service, the Conference of Education Ministers and the German Academic Exchange Service.
Within the scope of the planned evaluation, PASCH will be analyzed regarding its relevance, efficiency, effectivity, impact and sustainability. It will also be analyzed to what extent the existing offers, processes and structures are suitable instruments for achieving the objectives. Recommendations for possible optimization of the measures, processes and structures and the creation of synergies will be derived on the basis of these results.
The evaluation is divided into two phases: During the first phase, document analyses as well as guided personal and telephone interviews with stakeholders in Germany and selected partner countries (Poland, Turkey and China) will be conducted. The focus of the second phase is then on quantitative online surveys with target groups in China, Turkey and Poland as well as on qualitative and quantitative data collection in another case study country.
Evaluation of the project ‘Media in Libya: Stability through reconciliation’
The object of this evaluation was the project ‘Media in Libya: Stability through reconciliation’ (MLSR) of the DW Akademie. MLSR was a follow-up of the precursor project Media in Libya: Stability through Structure. The overall goal of MLSR was to enable a process of public discourse and eventually reconciliation across a fragmented Libyan society prerequisites for development of democratic reform and constitutional process in the country. By promoting [the] right to access to information to the people of Libya and [the] freedom of expression in Libya (DW Akademie, 2017, p. 4). The approach of MLSR is fourfold (i.e., four activity clusters): (1) supporting Libyan institutions/authorities efforts in the field of media governance, (2) developing a Libyan Media Network, (3) training local freelance journalists from all regions of Libya with a special focus on ethnic minorities and gender, and (4) providing grants for Libyan media projects.
The evaluation of MLSR had been considered as a planned impact evaluation, which needs valid qualitative/quantitative data on the situation at the beginning of MLSR (i.e., baseline). The second stage of evaluation (i.e., interim evaluation) started in January 2019. Comparing the results of baseline and interim evaluation, the evaluators had been able to identify and assess the preliminary effects of MLSR. Moreover, the evaluators may give first recommendations on how to improve MLSR (e.g., identification of [un]successful project strategies, alternative actions). The third stage of evaluation started shortly after the end of MLSR (i.e., January 2021). The main goal of the final evaluation was to assess the effectiveness of MLSR (i.e., achievements in relevant targeted areas).
Methodologically, a robust mixed-method approach was pursued against the background of the very volatile and difficult framework conditions in Libya: Secondary analysis of existing documents/data, content analysis of short written and in-depth oral interviews and statistical analysis of quantitative data from a standardized questionnaire survey.
Evaluation of the Funding Activities of the Foundation Environment and Development North Rhine-Westphalia
The Foundation Environment and Development North Rhine-Westphalia was founded 2001 and promotes mainly volunteer environmental and development work of civil society groups. The objective of the foundation is to support the environmental and development work in North Rhine-Westphalia (NRW) in order to establish this work in the consciousness and in the practical involvement of the population permanently and at the state of the art.
For this purpose about 1,100 projects of non-profit organisations in NRW were supported with a volume of EUR 47.5 million so far. In addition, the Foundation carries out its own projects, to also promote the implementation of Agenda 21 of the United Nations Conference on Environment and Development in Rio de Janeiro in in 1994.
Within the evaluation, the results and effects of the funding activities of the Foundation were recorded and assessed. Furthermore, the evaluation was intended to find out information on potential inhibitory and / or promoting factors in the funding activities of the Foundation with respect to the foundation goals. Based on these findings recommendations on priorities of the Foundation with regard to content, actors, target groups and the methods and formats will be derived.
CEval applied a mixed method approach to collect relevant data. Therefore, relevant documents and accessible data of the foundation and of funded projects will be analysed. This will be combined with data based on qualitative guideline-based interviews with several stakeholders and standardised online surveys of current and completed projects.
Assessing Effectiveness and Impact of the Extractive Industries Transparency Initiative (EITI)
The Extractive Industries Transparency Initiative (EITI) is a global standard to promote open and accountable management of natural resources. It seeks to strengthen government and company systems, inform public debate, and enhance trust. In each implementing country it is supported by a coalition of governments, companies and civil society working together.
The GIZ has mandated an evaluation of the initiative based on the following objectives:
- Evaluate the outcomes, effectiveness and impact (positive and negative, intended and unintended) of EITI against the backdrop of the EITI MDTF Results Framework, and to shed light on the links between transparency, accountability and, possibly, wider development outcomes.
- Assess EITIs theory of change and the EITI MDTF Results Framework in order to develop an improved generic EITI Results Model and assist selected MSGs to develop specific national EITI Results Models and to derive impact-oriented indicators from it.
These objectives and the complexity of EITIs context required a tailor-made design. The methodology combined
- macro-quantitative statistical analyses
- in-depth case studies
- participatory approaches to operationalization, data collection and analysis of findings.
Under the coordination of LBN Strategies and in collaboration with further consultants, the Center for Evaluation took the lead in the study component Approaches and Methodology.
