Meta-evaluation of MFA’s Decentralised Programme and Project Evaluations in 2017-2020
Together with a consortium of Particip and Niras, CEval was again commissioned by the Ministry of Foreign Affairs of Finland (MFA) to conduct a meta-evaluation of decentralised evaluations of development cooperation programmes and projects covering 2017-2020. The purpose was to build upon and amend the findings of the previous meta-evaluation to generate more learnings on the use and usefulness of decentralised evaluations, investigate the developments since the previous meta-evaluation, and to see if the quality of evaluations has further increased. Thus, this meta-evaluation is responsible for two ‘firsts’ in MFA’s meta-evaluation tradition. It is the first to replicate the methodology of the previous meta-evaluation allowing for sound comparison of results. Furthermore, it is the first to take into consideration the perspective of primary users of evaluations to assess the actual use and usefulness of decentralised evaluations. New features were also added and analysis tools amended to accommodate changes and developments since the previous meta-evaluation. A multi-method approach of qualitative and quantitative analysis was conducted on both secondary and primary data to generate evidence-based findings, conclusion and recommendations. This meta-evaluation consists of three components:
Component 1 – Meta-evaluation of 80 decentralised evaluations to assess the methodological quality of the evaluation reports and corresponding ToRs. The recent meta-evaluation was also compared to the previous one to identify differences and trends and identify gaps regarding MFA’s evaluation capacity.
Component 2 – Content assessment of 72 evaluation reports, that passed minimal quality standards, to assess the overall quality of the interventions. The assessment included quality of interventions according to the OECD DAC criteria, consideration of cross-cutting objectives (CCOs) i.e. gender equality, non-discrimination and climate sustainability, human rights-based approach (HRBA) and Finnish policy priority areas (PPAs). A qualitative synthesis of evaluators’ lessons and recommendations to improve quality of Finnish development cooperation was also performed.
Component 3 – Primary data of 26 interviewed MFA commissioners at embassy and headquarters level, and 119 survey responses from MFA commissioners and intervention implementers were collected to assess the actual use and usefulness of decentralized evaluations from their perspectives. Facilitating and hampering factors for usefulness, as wells as interviewees’ lessons and recommendations were also identified from this data.
Central Project Evaluation: Sector Programme Sustainability Standards and Public-Private Responsibility
The sector project “Sustainability Standards and Public-Private Responsibility” advised the German Federal Ministry for Economic Cooperation and Development on the creation of a favorable regulatory framework in Germany for the implementation of sustainability standards in global supply chains. Further, the project developed several online platforms with information on corporate due diligence and sustainability labels addressing small and medium-sized enterprises (KMU-Kompass), public procurers (kompass-nachhaltigkeit.de) and private consumers (siegelklarheit.de).
The evaluation was conducted according to the specifications and guidelines for central project evaluations of GIZ, which include a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). The CEval team applied a mixed-methods approach combining qualitative and quantitative instruments for data collection and analysis. Data collection featured semi-structured interviews, focus group discussions and an online survey. Causal relations between project activities and potential impacts, were assessed following a contribution analysis approach.
Central Project Evaluation: Sector Programme Marine Conservation
As a member of the United Nations, Germany is committed to international goals for marine conservation, such as SDG 14 “Life under Water” or the Convention on Biological Diversity (UN CBD) of 1992. The Sector Programme Marine Conservation advised the German Federal Ministry for Economic Cooperation and Development (BMZ) on its role in (inter)national policy processes on marine conservation, on the orientation and further development of the project portfolio relevant to marine protection, on setting new political priorities and on public relations. The objective of the sector project was to support the BMZ in shaping policy and implementation processes that serve marine conservation by providing technical input and analyses.
The evaluation was aligned with the specifications and guidelines for central project evaluations of GIZ, which include a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). With regards to data collection, semi-structured interviews and focus group discussions were applied. Causal relations between project activities and potential impacts, were assessed following a contribution analysis approach.
Meta-Evaluation of evaluative studies
Evaluative Studies (ES) in GIZ are commissioned and managed by operative units of projects and programs. They suit particular knowledge requirements of those operative units and are characterized by flexibility regarding the evaluation object, evaluation criteria and applied methods. In contrast to central project evaluations, ES are not directly managed by the GIZ evaluation unit. However, the evaluation unit does provide a so called evaluation support service to project managers in charge with the management of ES if requested.
In 2022 the GIZ evaluation unit commissioned a meta-evaluation of a sample of 27 ES conducted between 2020 and 2022. The purpose of this meta evaluation was to deliver information on the methodological quality and usefulness of ES in GIZ as well as to understand why ES are commissioned (context) and how they are implemented (methodological approach).
The meta-evaluation is based on a systematic review of the evaluation reports and the ToR. The review had been implemented applying a comprehensive meta-evaluation grid containing a range of items to be rated by the reviewer. In addition semi-structured interviews had been conducted with operational staff in charge with the management of ES to gain insight regarding their perspective on the usefulness of the ES’ findings and regarding their experience with the evaluation support service.
Evaluation of the project “International Sustainability Academy (ISA)”
The International Sustainability Academy (ISA) is a project of the Schutzgemeinschaft Deutscher Wald Hamburg e.V. (SDW). ISA offers a 9-month fellowship program, which is funded by the Federal Ministry for Economic Cooperation and Development (BMZ), with the aim of supporting and strengthening international cooperation in the implementation of the Sustainable Development Goals (SDGs) in developing countries. The scholarship program is open to interested individuals who are involved in activities relating to the SDGs and coming from countries included in the Development Assistance Committee (DAC) list. The first phase of the fellowship includes an eight-month stay in Hamburg, wherein the fellows are given capacity-building trainings, networking opportunities, individual consultation, and support for the (further) development of their scientific or business projects or products. The fellowship ends with the second phase, an additional funded month in their home country, during which focus is on the completion and implementation of their individual projects. ISA started in September 2019.
The CEval GmbH was commissioned by SDW to conduct a mid-term evaluation of the second funding phase from 2022 to 2024. The goal of the evaluation was to draw up an interim assessment and identify potential for optimizing the design of the fellowship program. The evaluation had been based on the OECD DAC criteria and addresses questions of relevance, coherence, effectiveness, impact, efficiency, and sustainability of the fellowship program.
Methodologically, a mixed-method approach was employed for this evaluation, combining qualitative and quantitative data collection and analytical procedures. Qualitative methods include a review of secondary data and project documents, semi-structured interviews and focus group discussions of stakeholders, and participant observation of the application selection process. Furthermore, quantitative online surveys of current ISA fellows and alumni had been conducted to capture their experiences and personal perspectives of the fellowship program.
Measurement of customer satisfaction with AIZ services for GIZ projects, follow-up survey 2022
To ensure continuous quality control, the Academy for International Cooperation (AIZ) has mandated CEval GmbH to conduct a customer satisfaction survey concerning AIZ services for GIZ projects. The aim is to determine the level of satisfaction of AIZ customers with their services in the field of technical vocational education and training.
For that purpose, a semi-standardized telephone survey was implemented among GIZ internal clients of AIZ services on matters of communication, consulting and services, work style and development orientation. Based on the findings, strengths and weaknesses of the HCD measures implemented by AIZ as well as resulting potential for improvement had been identified.
Development of a Monitoring- and Evaluation Concept for the Deutsche Stiftung für Internationale Rechtliche Zusammenarbeit e.V. [German Foundation for International Juridical Cooperation]
The German Foundation for International Juridical Cooperation e.V. (IRZ) supports partner countries in the development and consolidation of constitutional and market-economy structures on behalf of and with funds from the Federal Ministry of Justice and Consumer Protection (BMJV) since 1992. Among the approximately 25 partner states are new EU member states and accession candidates as well as other states in Southeast Europe, Central Asia, North Africa and the Middle East. The foundation supports these states in particular in the protection of human rights as well as in the areas of rule of law and justice, civil and commercial law, public law, criminal justice and European law.
The CEval GmbH supported IRZ in the development of a comprehensive monitoring and evaluation concept for the Foundation’s program line financed by the BMJV. The service included the operationalization of the target system of IRZ by means of suitable indicators and the development of a data collection plan. This included conducting an inventory of IRZ’s existing instruments harmonizing them with the new M&E concept. Within the framework of the implementation of the M&E concept, CEval GmbH had carried out a three-day capacity building training for the IRZ staff and accompany the test implementation of the monitoring instruments. Finally, the results of the development process had been documented in a handbook with hints for the practical implementation of the M&E concept.
Measurement of Customer Satisfaction with AIZ Services for GIZ Projects, Follow-up Survey 2020
To ensure continuous quality control, the Academy for International Cooperation (AIZ) had mandated CEval GmbH for the second time in a row to evaluate customer satisfaction with AIZ services for GIZ projects. For this purpose, a total of eleven GIZ internal clients had been interviewed by telephone between December 2020 and January 2021 and the data had been analyzed using descriptive statistics.
The evaluation design was divided into four key areas: communication, consulting and services, functioning and development Orientation. Based on the findings, both strengths and weaknesses of the HCD measures implemented by AIZ as well as resulting potential for improvement had been identified. Finally, the data had been compared with that from the previous year with regard to potential trends.
M&E support of two WaSH projects in Ethiopia and Kenya.
Knorr-Bremse Global Care e.V. (KBGC) is planning to implement two WASH projects in Ethiopia and Kenya. The common goal of the two projects is to reduce mortality and morbidity rates from water-related diseases. For this purpose, a series of measures adapted to the framework conditions in the respective implementation regions is planned, such as improving access to clean water, increasing the knowledge and skills of the actors involved, rehabilitating sanitary facilities, disseminating more energy-efficient stoves, health education and, finally, changing attitudes and behaviour among the target groups. These activities are intended to contribute to the improvement of the general living conditions of the target groups.
CEval GmbH is responsible for the scientific monitoring and evaluation of the two projects with the aim of assessing their effectiveness. The implementation will be carried out in close cooperation with national evaluators in both countries, while CEval GmbH is responsible for the preparation of the evaluation design and methodological concept, the training of those responsible for monitoring as well as accompaniment of the entire process.
Mid-Term Evaluation of the „Palestinian-German Science Bridge” (PGSB)
The program Palestinian-German Science Bridge is a pilot project financed by the German Federal Ministry of Education and Research (BMBF), which is jointly implemented by Forschungszentrum Jülich and the Palestinian Academy for Science and Technology (PALAST). It aims at establishing a joint education and research cooperation between Forschungszentrum Jülich and the Palestinian partner universities. To this end, the program combines measures focusing on student and researcher mobility, scientific exchange among faculty staff, the support of research collaborations and the reintegration of young Palestinian researchers into the Palestinian academic sector.
Forschungszentrum Jülich mandated CEval GmbH to counsel the Forschungszentrum with regard to the conceptualization of a mid-term evaluation of the program and to assist the Forschungszentrum in implementing it. The purpose of the evaluation was to assess the short- and medium-term results of the program as well as the prospects for long-term goal achievement. The objective, further, was to identify potentials for improvement in relation to the program.
Methodologically, the evaluation was based on a mix of qualitative and quantitative data collection and analysis methods. Data collection combines a document and secondary data analysis with guided interviews and semi-standardized online surveys among the project stakeholders. During data analysis, descriptive statistics and qualitative content analysis were applied. Due to the COVID-19 pandemic, the evaluation was planned as a remote evaluation.