Meta-evaluation of MFA’s Decentralised Programme and Project Evaluations in 2017-2020
Together with a consortium of Particip and Niras, CEval was again commissioned by the Ministry of Foreign Affairs of Finland (MFA) to conduct a meta-evaluation of decentralised evaluations of development cooperation programmes and projects covering 2017-2020. The purpose was to build upon and amend the findings of the previous meta-evaluation to generate more learnings on the use and usefulness of decentralised evaluations, investigate the developments since the previous meta-evaluation, and to see if the quality of evaluations has further increased. Thus, this meta-evaluation is responsible for two ‘firsts’ in MFA’s meta-evaluation tradition. It is the first to replicate the methodology of the previous meta-evaluation allowing for sound comparison of results. Furthermore, it is the first to take into consideration the perspective of primary users of evaluations to assess the actual use and usefulness of decentralised evaluations. New features were also added and analysis tools amended to accommodate changes and developments since the previous meta-evaluation. A multi-method approach of qualitative and quantitative analysis was conducted on both secondary and primary data to generate evidence-based findings, conclusion and recommendations. This meta-evaluation consists of three components:
Component 1 – Meta-evaluation of 80 decentralised evaluations to assess the methodological quality of the evaluation reports and corresponding ToRs. The recent meta-evaluation was also compared to the previous one to identify differences and trends and identify gaps regarding MFA’s evaluation capacity.
Component 2 – Content assessment of 72 evaluation reports, that passed minimal quality standards, to assess the overall quality of the interventions. The assessment included quality of interventions according to the OECD DAC criteria, consideration of cross-cutting objectives (CCOs) i.e. gender equality, non-discrimination and climate sustainability, human rights-based approach (HRBA) and Finnish policy priority areas (PPAs). A qualitative synthesis of evaluators’ lessons and recommendations to improve quality of Finnish development cooperation was also performed.
Component 3 – Primary data of 26 interviewed MFA commissioners at embassy and headquarters level, and 119 survey responses from MFA commissioners and intervention implementers were collected to assess the actual use and usefulness of decentralized evaluations from their perspectives. Facilitating and hampering factors for usefulness, as wells as interviewees’ lessons and recommendations were also identified from this data.
Central Project Evaluation: Sector Programme Sustainability Standards and Public-Private Responsibility
The sector project “Sustainability Standards and Public-Private Responsibility” advised the German Federal Ministry for Economic Cooperation and Development on the creation of a favorable regulatory framework in Germany for the implementation of sustainability standards in global supply chains. Further, the project developed several online platforms with information on corporate due diligence and sustainability labels addressing small and medium-sized enterprises (KMU-Kompass), public procurers (kompass-nachhaltigkeit.de) and private consumers (siegelklarheit.de).
The evaluation was conducted according to the specifications and guidelines for central project evaluations of GIZ, which include a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). The CEval team applied a mixed-methods approach combining qualitative and quantitative instruments for data collection and analysis. Data collection featured semi-structured interviews, focus group discussions and an online survey. Causal relations between project activities and potential impacts, were assessed following a contribution analysis approach.
Central Project Evaluation: Sector Programme Marine Conservation
As a member of the United Nations, Germany is committed to international goals for marine conservation, such as SDG 14 “Life under Water” or the Convention on Biological Diversity (UN CBD) of 1992. The Sector Programme Marine Conservation advised the German Federal Ministry for Economic Cooperation and Development (BMZ) on its role in (inter)national policy processes on marine conservation, on the orientation and further development of the project portfolio relevant to marine protection, on setting new political priorities and on public relations. The objective of the sector project was to support the BMZ in shaping policy and implementation processes that serve marine conservation by providing technical input and analyses.
The evaluation was aligned with the specifications and guidelines for central project evaluations of GIZ, which include a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). With regards to data collection, semi-structured interviews and focus group discussions were applied. Causal relations between project activities and potential impacts, were assessed following a contribution analysis approach.
Meta-Evaluation of evaluative studies
Evaluative Studies (ES) in GIZ are commissioned and managed by operative units of projects and programs. They suit particular knowledge requirements of those operative units and are characterized by flexibility regarding the evaluation object, evaluation criteria and applied methods. In contrast to central project evaluations, ES are not directly managed by the GIZ evaluation unit. However, the evaluation unit does provide a so called evaluation support service to project managers in charge with the management of ES if requested.
In 2022 the GIZ evaluation unit commissioned a meta-evaluation of a sample of 27 ES conducted between 2020 and 2022. The purpose of this meta evaluation was to deliver information on the methodological quality and usefulness of ES in GIZ as well as to understand why ES are commissioned (context) and how they are implemented (methodological approach).
The meta-evaluation is based on a systematic review of the evaluation reports and the ToR. The review had been implemented applying a comprehensive meta-evaluation grid containing a range of items to be rated by the reviewer. In addition semi-structured interviews had been conducted with operational staff in charge with the management of ES to gain insight regarding their perspective on the usefulness of the ES’ findings and regarding their experience with the evaluation support service.
Evaluation of the project “International Sustainability Academy (ISA)”
The International Sustainability Academy (ISA) is a project of the Schutzgemeinschaft Deutscher Wald Hamburg e.V. (SDW). ISA offers a 9-month fellowship program, which is funded by the Federal Ministry for Economic Cooperation and Development (BMZ), with the aim to support and strengthen international cooperation in the implementation of the Sustainable Development Goals (SDGs) in developing countries. The scholarship program is open to interested individuals who are involved in activities relating to the SDGs and coming from countries included in the Development Assistance Committee (DAC) list. The first phase of the fellowship includes an eight-month stay in Hamburg wherein the fellows are given capacity-building training, networking opportunities, individual consultation, and support for the (further) development of their scientific or business projects or products. The fellowship ends with the second phase, an additional funded month in their home country, during which focus is on the completion and implementation of their individual projects. ISA started in September 2019.
The CEval GmbH was commissioned by SDW to conduct a mid-term evaluation of the second funding phase from 2022 to 2024. The goal of the evaluation was to draw up interim assessment and identify potential for optimizing the design of the fellowship program. The evaluation had been based on the OECD DAC criteria and addresses questions of relevance, coherence, effectiveness, impact, efficiency, and sustainability of the fellowship program.
Methodologically, a mixed-method approach was employed for this evaluation, combining qualitative and quantitative data collection and analytical procedures. Qualitative methods include a review of secondary data and project documents, semi-structured interviews and focus group discussions of stakeholders, and participant observation of the application selection process. Furthermore, quantitative online surveys of current ISA fellows and alumni had been conducted to capture their experiences and personal perspectives of the fellowship program.
Measurement of customer satisfaction with AIZ services for GIZ projects, follow-up survey 2022
To ensure continuous quality control, the Academy for International Cooperation (AIZ) has mandated CEval GmbH to conduct a customer satisfaction survey concerning AIZ services for GIZ projects. The aim is to determine the level of satisfaction of AIZ customers with their services in the field of technical vocational education and training.
For that purpose, a semi-standardized telephone survey was implemented among GIZ internal clients of AIZ services on matters of communication, consulting and services, work style and development orientation. Based on the findings, strengths and weaknesses of the HCD measures implemented by AIZ as well as resulting potential for improvement had been identified.
Joint departmental strategic evaluation of the AA and BMZ engagement in Iraq
Within the framework of its crisis engagement in Iraq, the Federal Government intends to further embed monitoring and evaluation as instruments of learning and quality assurance, accountability and strategic steering. In this context, a joint evaluation of the Iraq engagement of the Federal Foreign Office (AA) and the Federal Ministry for Economic Cooperation and Development (BMZ) was conducted. This was intended to serve as a pilot project and, if possible, as a model for joint evaluations of both ministries in relevant fields of action and at their interfaces. The main objective of this joint evaluation was joint institutional learning. The AA component of the evaluation was realised by a consortium under the leadership of the GFA Consulting Group, in which CEval GmbH was involved; the BMZ component by the German Evaluation Institute (DEval). Both evaluation teams worked closely together in a so called Arbeitsgemeinschaft.
The aim of the evaluation was to conduct an independent and comprehensive analysis and assessment of the engagement of the two ministries in order to gain and use insights for (joint) steering. The evaluation provided insights into the relevance, effectiveness and overarching impacts, efficiency, sustainability and connectivity as well as coordination, coherence and complementarity of the Iraq engagement of both ministries. Furthermore, the evaluation provided recommendations for optimising the overall engagement and individual measures, structures and processes in order to adjust the current portfolio if necessary. Finally, learning experiences from the AA-BMZ engagement in Iraq were identified for similar engagement in other crisis contexts. The evaluation also served as an accompanying or final performance review of the country portfolio in accordance with the Federal Budget Code (Bundeshaushaltsordnung).
Methodologically, the evaluation was based on a broad mix of qualitative and quantitative data collection and evaluation methods. Due to pandemic-related travel restrictions, the data collection was realised or managed remotely by the consortium in both Germany and Iraq.
Development of a Monitoring- and Evaluation Concept for the Deutsche Stiftung für Internationale Rechtliche Zusammenarbeit e.V. [German Foundation for International Juridical Cooperation]
The German Foundation for International Juridical Cooperation e.V. (IRZ) supports partner countries in the development and consolidation of constitutional and market-economy structures on behalf of and with funds from the Federal Ministry of Justice and Consumer Protection (BMJV) since 1992. Among the approximately 25 partner states are new EU member states and accession candidates as well as other states in Southeast Europe, Central Asia, North Africa and the Middle East. The foundation supports these states in particular in the protection of human rights as well as in the areas of rule of law and justice, civil and commercial law, public law, criminal justice and European law.
The CEval GmbH supported IRZ in the development of a comprehensive monitoring and evaluation concept for the Foundation’s program line financed by the BMJV. The service included the operationalization of the target system of IRZ by means of suitable indicators and the development of a data collection plan. This included conducting an inventory of IRZ’s existing instruments harmonizing them with the new M&E concept. Within the framework of the implementation of the M&E concept, CEval GmbH had carried out a three-day capacity building training for the IRZ staff and accompany the test implementation of the monitoring instruments. Finally, the results of the development process had been documented in a handbook with hints for the practical implementation of the M&E concept.
Measurement of Customer Satisfaction with AIZ Services for GIZ Projects, Follow-up Survey 2020
To ensure continuous quality control, the Academy for International Cooperation (AIZ) had mandated CEval GmbH for the second time in a row to evaluate customer satisfaction with AIZ services for GIZ projects. For this purpose, a total of eleven GIZ internal clients had been interviewed by telephone between December 2020 and January 2021 and the data had been analyzed using descriptive statistics.
The evaluation design was divided into four key areas: communication, consulting and services, functioning and development Orientation. Based on the findings, both strengths and weaknesses of the HCD measures implemented by AIZ as well as resulting potential for improvement had been identified. Finally, the data had been compared with that from the previous year with regard to potential trends.
M&E support of two WaSH projects in Ethiopia and Kenya.
Knorr-Bremse Global Care e.V. (KBGC) is planning to implement two WASH projects in Ethiopia and Kenya. The common goal of the two projects is to reduce mortality and morbidity rates from water-related diseases. For this purpose, a series of measures adapted to the framework conditions in the respective implementation regions is planned, such as improving access to clean water, increasing the knowledge and skills of the actors involved, rehabilitating sanitary facilities, disseminating more energy-efficient stoves, health education and, finally, changing attitudes and behaviour among the target groups. These activities are intended to contribute to the improvement of the general living conditions of the target groups.
CEval GmbH is responsible for the scientific monitoring and evaluation of the two projects with the aim of assessing their effectiveness. The implementation will be carried out in close cooperation with national evaluators in both countries, while CEval GmbH is responsible for the preparation of the evaluation design and methodological concept, the training of those responsible for monitoring as well as accompaniment of the entire process.
