Evaluation of the Diakonie Katastrophenhilfe Programme Indonesia 2018-2023
The DKH Indonesia programme was launched in the immediate aftermath of three large scale natural disasters occurring in Lombok (earthquake), Central Sulawesi (earthquake and tsunami) and the Sunda Strait (tsunami) in 2018. The programme was composed of eight individual projects implemented by two Indonesian partner organisations in five provinces and featured immediate emergency response measures, recovery assistance and Disaster Risk Reduction activities with a focus on inclusion of people with disability and prioritisation of most vulnerable groups. CEval was commissioned to provide an independent assessment of the programme based on the OECD-DAC criteria.
The evaluation was based on extensive stakeholder participation as well as on a mixed-methods and theory-based evaluation approach. The evaluation team developed a comprehensive Theory of Change (ToC) at programme level and a detailed evaluation matrix. A desk-based systematic review of project documents was complemented by a three-week on-site study in Indonesia. The systematic review was based on a standardised assessment grid developed from the evaluation matrix and applied to aggregate monitoring data and relevant information from project documents. During the on-site study, the evaluation team conducted 29 interviews and 29 focus group discussions with programme and project staff, local authorities, and beneficiaries. Primary data was analysed applying qualitative content analysis. Findings were presented in a comprehensive evaluation report and during several workshops.
System Evaluation of the Evaluation System of the ‘International Cooperation’ of the Physikalisch-Technische Bundesanstalt
In 2021, CEval GmbH was assigned by the Physikalisch-Technische Bundesanstalt (PTB), the National Metrology Institute, with a system evaluation. The overall objective was a strategic improvement of the PTB’s evaluation system. Relevant points of the analysis grid and the evaluation questions were:
1. Evaluation policy, goals, strategies and criteria of evaluation.
2. Planning and procedure of evaluations, control processes, forms and scope of evaluations, financial resources, relation to the total budget
3. Standards of evaluation, quality assurance system, impact orientation, methodological quality
4. Anchoring of evaluation in the organization, independence of evaluation, credibility of evaluation
5. Personnel capacity, qualification, evaluators, further training, networking with the professional community
6. Utilization of evaluation results (processes of utilization, products of evaluation, dissemination of results, knowledge management system).
Methodologically, various data sources, such as documents and evaluation reports, were included and 28 guided qualitative interviews, as well as a group discussion, were conducted.
Pilot DC-Program Evaluation Sustainable Economic Development – Promotion of Youth Employment and Vocational Training in Kenya
In the future, evaluations of development cooperation (DC) programs should provide policy-relevant findings, especially on the establishment and coordination of DC projects and their interaction in a sector or region. In order to develop an appropriate format for DC program evaluations, experience was gathered during the pilot evaluation of the DC program “Sustainable Economic Development – Promotion of Youth Employment and Vocational Training in Kenya” and subsequently systematically processed.
The evaluation used a theory-based approach to empirically test assumptions about causal relationships between the services provided, the goals of the individual projects, and the overall goal of the program. The results are based on individual and group interviews with program and project managers, partner organizations and target group representatives. Furthermore, the program and project reporting was subjected to a qualitative content analysis.
Developments in fair trade and fair procurement in Germany 2015-2022. Follow-up to the study “Does fair trade change society?”
In 2015, leading stakeholders of the German fair trade movement commissioned a study on the impact of fair trade in Germany, which was carried out by CEval GmbH. The study investigated the impact of fair trade in four societal areas, i.e., civil society, consumers, retailers and producers, and politics and the public sector. Since 2015 the fairtrade sector as well as the framework conditions in which it operated have changed. This includes, in particular, the environment for a fair public procurement.
Against that backdrop, Fairtrade Germany, Engagement Global – Service für Entwicklungsinitiativen (Service for Development Initiatives) with its Servicestelle Kommunen in der Einen Welt (SKEW – Service Agency Communities in One World) and in cooperation with Forum Fairer Handel (Forum Fair Trade) and Weltladen-Dachverband (umbrella organization of German world shops) mandated CEval GmbH with the implementation of an updated and extended version of the 2015 study. The objective was to investigate the impact of fair trade in Germany between 2016 and 2022 under the changed framework conditions and taking into account the increased role of the German municipalities for fair trade and fair public procurement. Based on the findings, recommendations on how to further increase the effectiveness of fair trade and how to further support a fair public procurement have been put forth.
Methodologically, the study built on the approach developed for the 2015 study and relied on qualitative contribution analysis. On the basis of a so-called theory of change and its underlying hypotheses, the study investigated the changes that are intended by the fair trade community in the various areas of observation as well as the contributions that have been made by the fair trade community to these changes. In doing so, a mix of qualitative and quantitative methods of data collection and analysis was applied. The starting point was an extensive desk study of existing literature and secondary data, which was complemented by empirical data collected by means of guided key informant interviews, (focus) group discussions and a semi-standardized online survey.
Central Project Evaluation: Sector Programme Soil Protection, Combating Desertification, Sustainable Land Management
The Sector Programme Soil Conservation, Combating Desertification, Sustainable Land Management advised the Ministry for Economic Cooperation and Development (BMZ) on the topics of sustainable land management, soil conservation and soil related climate impacts. In addition, the project tested and implemented approaches for the utilization of soil conservation measures for climate protection with partners in India and Kenya. This included the implementation of a carbon certification system in Kenya and studies on the climate impact of sustainable land management methods in India. The experience and knowledge gained from these studies and pilots were to be disseminated at national and international levels.
The evaluation was carried out according to the specifications and guidelines for central project evaluations of the GIZ, which included a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). During a ten-day on-site mission in Kenya, the evaluation team conducted interviews and focus group discussions with stakeholders and beneficiaries of the carbon certification scheme. The data collection in India was implemented remotely (online interviews). A contribution analysis approach was applied to examine causal relationships between project activities and potential impacts.
Meta-evaluation of MFA’s Decentralised Programme and Project Evaluations in 2017-2020
Together with a consortium of Particip and Niras, CEval was again commissioned by the Ministry of Foreign Affairs of Finland (MFA) to conduct a meta-evaluation of decentralised evaluations of development cooperation programmes and projects covering 2017-2020. The purpose was to build upon and amend the findings of the previous meta-evaluation to generate more learnings on the use and usefulness of decentralised evaluations, investigate the developments since the previous meta-evaluation, and to see if the quality of evaluations has further increased. Thus, this meta-evaluation is responsible for two ‘firsts’ in MFA’s meta-evaluation tradition. It is the first to replicate the methodology of the previous meta-evaluation allowing for sound comparison of results. Furthermore, it is the first to take into consideration the perspective of primary users of evaluations to assess the actual use and usefulness of decentralised evaluations. New features were also added and analysis tools amended to accommodate changes and developments since the previous meta-evaluation. A multi-method approach of qualitative and quantitative analysis was conducted on both secondary and primary data to generate evidence-based findings, conclusion and recommendations. This meta-evaluation consists of three components:
Component 1 – Meta-evaluation of 80 decentralised evaluations to assess the methodological quality of the evaluation reports and corresponding ToRs. The recent meta-evaluation was also compared to the previous one to identify differences and trends and identify gaps regarding MFA’s evaluation capacity.
Component 2 – Content assessment of 72 evaluation reports, that passed minimal quality standards, to assess the overall quality of the interventions. The assessment included quality of interventions according to the OECD DAC criteria, consideration of cross-cutting objectives (CCOs) i.e. gender equality, non-discrimination and climate sustainability, human rights-based approach (HRBA) and Finnish policy priority areas (PPAs). A qualitative synthesis of evaluators’ lessons and recommendations to improve quality of Finnish development cooperation was also performed.
Component 3 – Primary data of 26 interviewed MFA commissioners at embassy and headquarters level, and 119 survey responses from MFA commissioners and intervention implementers were collected to assess the actual use and usefulness of decentralized evaluations from their perspectives. Facilitating and hampering factors for usefulness, as wells as interviewees’ lessons and recommendations were also identified from this data.
Central Project Evaluation: Sector Programme Sustainability Standards and Public-Private Responsibility
The sector project “Sustainability Standards and Public-Private Responsibility” advised the German Federal Ministry for Economic Cooperation and Development on the creation of a favorable regulatory framework in Germany for the implementation of sustainability standards in global supply chains. Further, the project developed several online platforms with information on corporate due diligence and sustainability labels addressing small and medium-sized enterprises (KMU-Kompass), public procurers (kompass-nachhaltigkeit.de) and private consumers (siegelklarheit.de).
The evaluation was conducted according to the specifications and guidelines for central project evaluations of GIZ, which include a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). The CEval team applied a mixed-methods approach combining qualitative and quantitative instruments for data collection and analysis. Data collection featured semi-structured interviews, focus group discussions and an online survey. Causal relations between project activities and potential impacts, were assessed following a contribution analysis approach.
Central Project Evaluation: Sector Programme Marine Conservation
As a member of the United Nations, Germany is committed to international goals for marine conservation, such as SDG 14 “Life under Water” or the Convention on Biological Diversity (UN CBD) of 1992. The Sector Programme Marine Conservation advised the German Federal Ministry for Economic Cooperation and Development (BMZ) on its role in (inter)national policy processes on marine conservation, on the orientation and further development of the project portfolio relevant to marine protection, on setting new political priorities and on public relations. The objective of the sector project was to support the BMZ in shaping policy and implementation processes that serve marine conservation by providing technical input and analyses.
The evaluation was aligned with the specifications and guidelines for central project evaluations of GIZ, which include a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). With regards to data collection, semi-structured interviews and focus group discussions were applied. Causal relations between project activities and potential impacts, were assessed following a contribution analysis approach.
Meta-Evaluation of evaluative studies
Evaluative Studies (ES) in GIZ are commissioned and managed by operative units of projects and programs. They suit particular knowledge requirements of those operative units and are characterized by flexibility regarding the evaluation object, evaluation criteria and applied methods. In contrast to central project evaluations, ES are not directly managed by the GIZ evaluation unit. However, the evaluation unit does provide a so called evaluation support service to project managers in charge with the management of ES if requested.
In 2022 the GIZ evaluation unit commissioned a meta-evaluation of a sample of 27 ES conducted between 2020 and 2022. The purpose of this meta evaluation was to deliver information on the methodological quality and usefulness of ES in GIZ as well as to understand why ES are commissioned (context) and how they are implemented (methodological approach).
The meta-evaluation is based on a systematic review of the evaluation reports and the ToR. The review had been implemented applying a comprehensive meta-evaluation grid containing a range of items to be rated by the reviewer. In addition semi-structured interviews had been conducted with operational staff in charge with the management of ES to gain insight regarding their perspective on the usefulness of the ES’ findings and regarding their experience with the evaluation support service.
Evaluation of the project “International Sustainability Academy (ISA)”
The International Sustainability Academy (ISA) is a project of the Schutzgemeinschaft Deutscher Wald Hamburg e.V. (SDW). ISA offers a 9-month fellowship program, which is funded by the Federal Ministry for Economic Cooperation and Development (BMZ), with the aim to support and strengthen international cooperation in the implementation of the Sustainable Development Goals (SDGs) in developing countries. The scholarship program is open to interested individuals who are involved in activities relating to the SDGs and coming from countries included in the Development Assistance Committee (DAC) list. The first phase of the fellowship includes an eight-month stay in Hamburg wherein the fellows are given capacity-building training, networking opportunities, individual consultation, and support for the (further) development of their scientific or business projects or products. The fellowship ends with the second phase, an additional funded month in their home country, during which focus is on the completion and implementation of their individual projects. ISA started in September 2019.
The CEval GmbH was commissioned by SDW to conduct a mid-term evaluation of the second funding phase from 2022 to 2024. The goal of the evaluation was to draw up interim assessment and identify potential for optimizing the design of the fellowship program. The evaluation had been based on the OECD DAC criteria and addresses questions of relevance, coherence, effectiveness, impact, efficiency, and sustainability of the fellowship program.
Methodologically, a mixed-method approach was employed for this evaluation, combining qualitative and quantitative data collection and analytical procedures. Qualitative methods include a review of secondary data and project documents, semi-structured interviews and focus group discussions of stakeholders, and participant observation of the application selection process. Furthermore, quantitative online surveys of current ISA fellows and alumni had been conducted to capture their experiences and personal perspectives of the fellowship program.
