Central Project Evaluation: Agribusiness Facilities for Africa

The project Agribusiness Facilities for Africa (ABF) serves as a pan-African knowledge and advisory hub on agribusiness trainings and has worked with about 30 countries across the continent. With the objective of improving the effectiveness of private and public agribusiness support measures and supporting organisations for micro, small and medium-sized enterprises in Africa, ABF develops training concepts for farmers and processors to meet their partners’ specific needs. The Farmer Business School (FBS) training teaches farmers and producers principles of agriculture as a business and promises to strengthen the management skills of MSMEs for better and diversified incomes and nutrition. Other training modules approaches offered by ABF include, Processor Business School (PBS), Cooperative Business School (CBS) and the more recently developed Aquaculture Business School (ABS ).
The evaluation was conducted according to the specifications and guidelines for central project evaluations of GIZ, which include a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). The evaluation team applies qualitative methods for data collection and analysis. Semi-structured interviews and focus groups discussions (FGDs) are conducted for primary data collection. During a 3 week field study in Uganda, Zambia and Malawi, the evaluation team will visit partner projects of ABF to assess the implementation and the impact of FBS, CBS and ABS teaching approaches on the ground. Qualitative content analysis will be applied based on written transcripts of the interviews, FGDs and the vast body of project documents. Causal relations between project activities and potential impacts, are assessed following a contribution analysis approach.

Evaluation of the Diakonie Katastrophenhilfe Programme Indonesia 2018-2023

The DKH Indonesia programme was launched in the immediate aftermath of three large scale natural disasters occurring in Lombok (earthquake), Central Sulawesi (earthquake and tsunami) and the Sunda Strait (tsunami) in 2018. The programme was composed of eight individual projects implemented by two Indonesian partner organisations in five provinces and featured immediate emergency response measures, recovery assistance and Disaster Risk Reduction activities with a focus on inclusion of people with disability and prioritisation of most vulnerable groups. CEval was commissioned to provide an independent assessment of the programme based on the OECD-DAC criteria.

The evaluation was based on extensive stakeholder participation as well as on a mixed-methods and theory-based evaluation approach. The evaluation team developed a comprehensive Theory of Change (ToC) at programme level and a detailed evaluation matrix. A desk-based systematic review of project documents was complemented by a three-week on-site study in Indonesia. The systematic review was based on a standardised assessment grid developed from the evaluation matrix and applied to aggregate monitoring data and relevant information from project documents. During the on-site study, the evaluation team conducted 29 interviews and 29 focus group discussions with programme and project staff, local authorities, and beneficiaries. Primary data was analysed applying qualitative content analysis. Findings were presented in a comprehensive evaluation report and during several workshops.

Evaluation of the EU Home Affairs Funds: Mid-term evaluations of the Internal Security Fund (ISF) and the Instrument for Financial Support for Border Management and Visa Policy (BMVI)

Under the responsibility of the Directorate-General for Migration and Home Affairs (DG HOME), the European Commission (EU COM) provides the Internal Security Fund (ISF; managed by the Federal Criminal Police Office (BKA)), the Border Management and Visa Instrument (BMVI, managed by the Federal Police) and the Asylum, Migration and Integration Fund (AMIF, managed by the Federal Office for Migration and Refugees (BAMF)). Together, these form the so-called “Home Affairs Funds”.

The member states are obliged to prepare mid-term and final evaluations for the EU Commission on the implementation of the measures financed by the individual funds and the degree to which the objectives of the national programmes supported by the Home Affairs Funds have been achieved. The main users of the evaluations are the EU (DG Home), the fund-managing authorities BKA, BPOL and BAMF and their monitoring committees.

The aim of the evaluations is to assess the results and outcomes of the programmes and measures co-financed by the Home Affairs Fund based on the criteria of effectiveness, efficiency, relevance, coherence and EU added value. The coherence between the individual funds is of particular interest. In addition to assessing success, the evaluations should also develop recommendations for improving the quality of the design and implementation of the national programmes and analyse the coherence and use of synergy potential between the individual funds.

After the Gesellschaft für Internationale Zusammenarbeit (GIZ) carried out the mid-term evaluation for the ISF on behalf of the BKA in 2017, it has now been commissioned again to carry out the final evaluation of the ISF for the 2014-2020 funding period and the mid-term evaluation of the ISF, the BMVI and the AMIF for the 2021-2027 funding period.

In the course of this, GIZ commissioned CEval GmbH to carry out the
> Final evaluation of the ISF, security sub-sector and border sub-sector, 2014-2021 funding period,
> Mid-term evaluation of the ISF, current funding period 2021-2027 and
> Mid-term evaluation of the BMVI, current funding period 2021-2027
to be carried out.

Various survey procedures are used to triangulate sources and methods:
> Research of all relevant programme and project documents as well as monitoring data,
> Semi-standardised online survey with project participants and their implementation partners as well as other potentially eligible applicants and
> Guided qualitative interviews with programme and project managers.

System Evaluation of the Evaluation System of the ‘International Cooperation’ of the Physikalisch-Technische Bundesanstalt

In 2021, CEval GmbH was assigned by the Physikalisch-Technische Bundesanstalt (PTB), the National Metrology Institute, with a system evaluation. The overall objective was a strategic improvement of the PTB’s evaluation system. Relevant points of the analysis grid and the evaluation questions were:
1. Evaluation policy, goals, strategies and criteria of evaluation.
2. Planning and procedure of evaluations, control processes, forms and scope of evaluations, financial resources, relation to the total budget
3. Standards of evaluation, quality assurance system, impact orientation, methodological quality
4. Anchoring of evaluation in the organization, independence of evaluation, credibility of evaluation
5. Personnel capacity, qualification, evaluators, further training, networking with the professional community
6. Utilization of evaluation results (processes of utilization, products of evaluation, dissemination of results, knowledge management system).

Methodologically, various data sources, such as documents and evaluation reports, were included and 28 guided qualitative interviews, as well as a group discussion, were conducted.

Pilot DC-Program Evaluation Sustainable Economic Development – Promotion of Youth Employment and Vocational Training in Kenya

In the future, evaluations of development cooperation (DC) programs should provide policy-relevant findings, especially on the establishment and coordination of DC projects and their interaction in a sector or region. In order to develop an appropriate format for DC program evaluations, experience was gathered during the pilot evaluation of the DC program “Sustainable Economic Development – Promotion of Youth Employment and Vocational Training in Kenya” and subsequently systematically processed.

The evaluation used a theory-based approach to empirically test assumptions about causal relationships between the services provided, the goals of the individual projects, and the overall goal of the program. The results are based on individual and group interviews with program and project managers, partner organizations and target group representatives. Furthermore, the program and project reporting was subjected to a qualitative content analysis.

Developments in fair trade and fair procurement in Germany 2015-2022. Follow-up to the study “Does fair trade change society?”

In 2015, leading stakeholders of the German fair trade movement commissioned a study on the impact of fair trade in Germany, which was carried out by CEval GmbH. The study investigated the impact of fair trade in four societal areas, i.e., civil society, consumers, retailers and producers, and politics and the public sector. Since 2015 the fairtrade sector as well as the framework conditions in which it operated have changed. This includes, in particular, the environment for a fair public procurement.
Against that backdrop, Fairtrade Germany, Engagement Global – Service für Entwicklungsinitiativen (Service for Development Initiatives) with its Servicestelle Kommunen in der Einen Welt (SKEW – Service Agency Communities in One World) and in cooperation with Forum Fairer Handel (Forum Fair Trade) and Weltladen-Dachverband (umbrella organization of German world shops) mandated CEval GmbH with the implementation of an updated and extended version of the 2015 study. The objective was to investigate the impact of fair trade in Germany between 2016 and 2022 under the changed framework conditions and taking into account the increased role of the German municipalities for fair trade and fair public procurement. Based on the findings, recommendations on how to further increase the effectiveness of fair trade and how to further support a fair public procurement have been put forth.
Methodologically, the study built on the approach developed for the 2015 study and relied on qualitative contribution analysis. On the basis of a so-called theory of change and its underlying hypotheses, the study investigated the changes that are intended by the fair trade community in the various areas of observation as well as the contributions that have been made by the fair trade community to these changes. In doing so, a mix of qualitative and quantitative methods of data collection and analysis was applied. The starting point was an extensive desk study of existing literature and secondary data, which was complemented by empirical data collected by means of guided key informant interviews, (focus) group discussions and a semi-standardized online survey.

Central Project Evaluation: Sector Programme Soil Protection, Combating Desertification, Sustainable Land Management

The Sector Programme Soil Conservation, Combating Desertification, Sustainable Land Management advised the Ministry for Economic Cooperation and Development (BMZ) on the topics of sustainable land management, soil conservation and soil related climate impacts. In addition, the project tested and implemented approaches for the utilization of soil conservation measures for climate protection with partners in India and Kenya. This included the implementation of a carbon certification system in Kenya and studies on the climate impact of sustainable land management methods in India. The experience and knowledge gained from these studies and pilots were to be disseminated at national and international levels.

The evaluation was carried out according to the specifications and guidelines for central project evaluations of the GIZ, which included a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). During a ten-day on-site mission in Kenya, the evaluation team conducted interviews and focus group discussions with stakeholders and beneficiaries of the carbon certification scheme. The data collection in India was implemented remotely (online interviews). A contribution analysis approach was applied to examine causal relationships between project activities and potential impacts.

Meta-evaluation of MFA’s Decentralised Programme and Project Evaluations in 2017-2020

Together with a consortium of Particip and Niras, CEval was again commissioned by the Ministry of Foreign Affairs of Finland (MFA) to conduct a meta-evaluation of decentralised evaluations of development cooperation programmes and projects covering 2017-2020. The purpose was to build upon and amend the findings of the previous meta-evaluation to generate more learnings on the use and usefulness of decentralised evaluations, investigate the developments since the previous meta-evaluation, and to see if the quality of evaluations has further increased. Thus, this meta-evaluation is responsible for two ‘firsts’ in MFA’s meta-evaluation tradition. It is the first to replicate the methodology of the previous meta-evaluation allowing for sound comparison of results. Furthermore, it is the first to take into consideration the perspective of primary users of evaluations to assess the actual use and usefulness of decentralised evaluations. New features were also added and analysis tools amended to accommodate changes and developments since the previous meta-evaluation. A multi-method approach of qualitative and quantitative analysis was conducted on both secondary and primary data to generate evidence-based findings, conclusion and recommendations. This meta-evaluation consists of three components:

Component 1 – Meta-evaluation of 80 decentralised evaluations to assess the methodological quality of the evaluation reports and corresponding ToRs. The recent meta-evaluation was also compared to the previous one to identify differences and trends and identify gaps regarding MFA’s evaluation capacity.

Component 2 – Content assessment of 72 evaluation reports, that passed minimal quality standards, to assess the overall quality of the interventions. The assessment included quality of interventions according to the OECD DAC criteria, consideration of cross-cutting objectives (CCOs) i.e. gender equality, non-discrimination and climate sustainability, human rights-based approach (HRBA) and Finnish policy priority areas (PPAs). A qualitative synthesis of evaluators’ lessons and recommendations to improve quality of Finnish development cooperation was also performed.

Component 3 – Primary data of 26 interviewed MFA commissioners at embassy and headquarters level, and 119 survey responses from MFA commissioners and intervention implementers were collected to assess the actual use and usefulness of decentralized evaluations from their perspectives. Facilitating and hampering factors for usefulness, as wells as interviewees’ lessons and recommendations were also identified from this data.

Central Project Evaluation: Sector Programme Sustainability Standards and Public-Private Responsibility

The sector project “Sustainability Standards and Public-Private Responsibility” advised the German Federal Ministry for Economic Cooperation and Development on the creation of a favorable regulatory framework in Germany for the implementation of sustainability standards in global supply chains. Further, the project developed several online platforms with information on corporate due diligence and sustainability labels addressing small and medium-sized enterprises (KMU-Kompass), public procurers (kompass-nachhaltigkeit.de) and private consumers (siegelklarheit.de).

The evaluation was conducted according to the specifications and guidelines for central project evaluations of GIZ, which include a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). The CEval team applied a mixed-methods approach combining qualitative and quantitative instruments for data collection and analysis. Data collection featured semi-structured interviews, focus group discussions and an online survey. Causal relations between project activities and potential impacts, were assessed following a contribution analysis approach.

Central Project Evaluation: Sector Programme Marine Conservation

As a member of the United Nations, Germany is committed to international goals for marine conservation, such as SDG 14 “Life under Water” or the Convention on Biological Diversity (UN CBD) of 1992. The Sector Programme Marine Conservation advised the German Federal Ministry for Economic Cooperation and Development (BMZ) on its role in (inter)national policy processes on marine conservation, on the orientation and further development of the project portfolio relevant to marine protection, on setting new political priorities and on public relations. The objective of the sector project was to support the BMZ in shaping policy and implementation processes that serve marine conservation by providing technical input and analyses.

The evaluation was aligned with the specifications and guidelines for central project evaluations of GIZ, which include a standardized analysis grid (evaluation matrix) based on the OECD-DAC criteria (relevance, coherence, efficiency, effectiveness, impact, sustainability). With regards to data collection, semi-structured interviews and focus group discussions were applied. Causal relations between project activities and potential impacts, were assessed following a contribution analysis approach.