Central project evaluation of the project Competitive African Cashew Value Chains for Pro-Poor Growth
The cashew value chain offers African producing countries interesting opportunities to create jobs, increase business and economic income and improve food security. Up to the time of the evaluation, however, the potential had hardly been exploited. At 300 to 550 kilograms of raw nuts per hectare, the yields of African producers were far below the yields in India and Vietnam and the processing capacities and labor productivity of the farms were still too low, which meant that only a small proportion of the raw nuts were processed in the African producing countries. The “Competitive African Cashew Value Chains for Pro-Poor Growth” project of the Deutsche Gesellschaft für internationale Zusammenarbeit GmbH supported the expansion and intensification of the cashew value chain in the six countries Benin, Burkina Faso, Côte d’Ivoire, Ghana, Mozambique and Sierra Leone.
The project was based on four levels of action: (i) improving the supply of high-performance seeds/seedlings through research and educating producers through training and consulting, (ii) advising processors of cashew nuts and by-products to improve process efficiency, (iii) intensifying business relations and exchange between all participants in the value chain, and (iv) promoting national and regional framework conditions by advising the relevant ministries and intensifying regional exchange between all actors.
CEval GmbH was commissioned by GIZ with the central project evaluation. The evaluation was aligned with the OECD DAC criteria and thus examined the relevance, effectiveness, efficiency, impact, sustainability and coherence of project implementation. Methodologically, the evaluation was based on the available quantitative secondary data of the project and expanded on this by conducting guided interviews and focus group discussions with project managers, stakeholders and target group representatives. Due to the corona pandemic prevailing at the time of the evaluation, the project was implemented as a remote evaluation. The evaluators from CEval GmbH worked together with local consultants in the African countries, who carried out the data collection in the field.
Scientific advice to the GIZ Evaluation Unit on the development of aggregated impact indicators.
The Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH contributes in many ways to solving regional and global problems. These include mitigating the effects of climate change, fighting poverty and hunger, or creating equal access to education and employment for men and women. To record the contributions made by its projects, GIZ has a number of instruments and procedures for impact measurement and evaluation. However, these tools and procedures are mainly geared toward reporting to the BMZ and generating evidence to improve individual interventions (PEVs) or to further develop the strategic direction of GIZ in the respective sector (USEs). At the level of the overall organization, however, there had been a lack of comparable approaches for presenting the contributions of GIZ as a whole, e.g. to the public or parliament (esp. EEZs).
In order to make GIZ capable of providing information at this level as well in the future, the Monitoring and Evaluation Unit initiated a process in 2014 to develop aggregated indicators that allow statements to be made about the joint contribution of as many projects as possible in the various sectors in which GIZ is active.
Integration of the Programme “Service Agency Communities in One World” (SKEW) in the M&E-System of Engagement Global
Engagement Global gGmbH was established in 2012 as a central institution to promote and coordinate the development policy commitment of civil society, municipal and private sector actors as well as for development policy information and education work. SKEW a program of Engagement Global focuses in particular on the promotion of municipal development cooperation. In order to be able to monitor both the appropriate use of the funds and above all their effectiveness, Engagement Global has created a comprehensive monitoring and evaluation (M&E) system for all departments and programs. Since SKEW was excluded in the development of the monitoring system at that time, CEval and ISPO were commissioned to integrate SKEW into the organization-wide monitoring system.
As with the design of the company-wide M&E system, first the theory of change and indicators were reviewed or developed. Then the objectives and indicators were integrated into the strategical claims of Engagement Global. In order to complete the creation of a monitoring system for the SKEW, a data collection plan were created and the associated data collection instruments had been (further) developed.
Ex-post assessment of the Strategic Fairtrade Funding Programme (SFFP), Phase II (2010-2013)
One of the key objectives of the Swiss Department on Trade Promotion is to support fair trading conditions with developing countries to which private labels and standards substantially contribute. Accordingly, as part of a donor consortium, the State Secretariat for Economic Affairs (SECO) contributed to the Strategic Fairtrade Funding Programme (SFFP) in Phase I from 2008 to 2009 and Phase II from 2010 to 2013. Main purpose of the harmonized funding was to enable disadvantaged producers to establish stronger linkages to consumers, to gain improved market access and to be empowered for combating poverty.
In 2014/2015 an evaluation was already carried out which examined Phase II of the funding programme. Further research on Fairtrade conducted include, among others, a follow-up impact study at the producer-level, covering six product and context settings, implemented by the Center for Evaluation (CEval GmbH) in 2017/2018. Four years after the SFFP ended, it was now of interest for SECO to assess the level of achievement of the strategic indicators set in the Logical Framework of the SFFP as well as SECOs contribution to the same.
The objective of the ex-post assessment was hence threefold: (i) to valorize existing information on the effectiveness and impact of SECOs support to the Fairtrade system, (ii) to complement findings from previous studies with new data on relevance and sustainability of the program and (iii) to distill key findings, recommendations and lessons learnt in view of the SECOs future engagement in Fairtrade. To this end, CEval examined relevant secondary data and collected additional primary data from stakeholders at FT headquarters, three FT regional networks and FT USA.
Results oriented monitoring of the technical cooperation measure COPLAN II and evaluation of the measure’s contribution towards a better alignment of development cooperation and foreign trade facilitation on a systemic level
Climate Change and the uninterrupted decline of fossil fuels affect different regions of the world in different ways. Many nations have thus formulated ambitious goals: Mexico for instance is determined to cover one third of its energy production by means of renewable energies by 2024. Despite such ambitions however environmental conservation has not been marked by major success to this date. This is partly due to a lack of dispersion of modern environmental technology. Against this background the GIZ Project Cooperation Platform Latin America North II (COPLAN II) scrutinizes central deficits that hinder international investment and cooperation such as a lack of professional expertise regarding modern environmental technology in small and medium enterprises (SMEs).
Within the scope of the present assignment CEval together with iSPO Institute supported COPLAN II in its monitoring and evaluation efforts. Starting point for any results oriented monitoring was typically an impact model with respective indicators and assumptions. Changes within the mechanisms of cooperation are however rarely or marginally observed under the monitoring component. This development of cooperation systems or the suitability of the relevant funding instruments are also examined as part of the evaluation component.
All in all, next to monitoring tasks and the compilation of results of individual measures the assignment at hand intended to also generate insights into COPLAN IIs contribution towards a better alignment of development cooperation and foreign trade facilitation as well as public-private sector cooperation on a systemic level.
Meta-Evaluation of ADA Project and Programme Evaluations 2016-2018
Subject of the meta-evaluation were ADA evaluations of development cooperation projects and programmes implemented between 2016 and 2018. The objective was to assess the quality and usefulness of ADAs (internal and external) evaluations and to deliver recommendations on how to improve the ADA evaluation tool for the design and implementation of future evaluations.
A document analysis of the evaluation reports was at the heart of the meta-evaluation. Based on a detailed analysis protocol the quality of said reports was assessed. The use of such a protocol, which covered all relevant analysis dimensions, indicators and rating scales, guaranteed the transparency and reliability of the meta-evaluation results. In order to evaluate the usefulness for and use of the evaluations under study, their target audience (e.g. programme managers) were interviewed via guided individual and group interviews and in order to achieve the greatest possible representativeness of the results a semi-standardized online survey.
Meta-evaluation and synthesis of Christoffel Blindenmission evaluation reports
On behalf of the Christoffel Blindenmission (CBM), CEval conducted an evaluation and synthesis of 24 evaluation reports from 2016 and 2017. With its mandate to transform the lives of disabled people in the worlds poorest countries the project focus of CBM has been an exhaustive collaboration with as well as the promotion of local partners in developing countries and conflict areas.
The primary purpose of the evaluation and synthesis was to conduct a general assessment of the evaluation reports quality and summarize findings of those reports that pass the quality control. In a first step, to ensure homogeneity of data, the quality of reports was assessed systematically by creating a quality control analysis grid. Reports that fulfil minimum quality standards were then subjected to an in-depth analysis based on the evaluation questions of CBM. Main interest of the synthesis was to drew up general learnings along CBMs mandate areas and examine the actual contribution of joint projects to changes for CBM partners and clients. Lastly, by employing an online survey directed to end users of project evaluations, the evaluation inquired the usage and usefulness of the evaluation reports for CBM staff members in seven world regions.
Meta-Evaluation of Plan International evaluation reports
Plan International is working for many years now in Africa, Asia and the Americas to promote child rights and lift children out of poverty. The German National Office of Plan International was founded in 1989, and has been contributing to funding development projects and programs as well as funding and commissioning external evaluations. Plan International Germany decided to implement a meta-evaluation of evaluation reports conducted between 2013 and 2017 covering all geographical working areas.
The primary objectives of the meta-evaluation were to assess the strengths and weaknesses by analyzing the quality of the evaluation reports, to identify the success and risk factors that may affect their quality and to give recommendations. A particular focus was laid upon the influence of the type of evaluation (i.e. by whom it is coordinated and implemented) and the available resources and timeframe.
The meta-evaluation was based on the analysis of the evaluation reports. The document analysis was structured in two phases. While in the first phase (pre-rating) the completeness of the evaluation reports was assessed, in a second phase (in-depth analysis), a more comprehensive set of quality assessment criteria was applied on a sample of reports, to allow for a more detailed assessment of the validity of their findings and usefulness to their addressees. Comprehensive analysis tools were used to analyze the completeness of the reports as well as the validity of findings and the usefulness of evaluation reports.
Evaluation of APPEAR projects (programme periods I and II)
The Austrian Partnership Programme in Higher Education and Research for Development (APPEAR) is conceived, guided and financed by the Austrian Development Agency (ADA) to implement its “Higher Education and Scientific Cooperation” strategy in support of Higher Education and Research for Development on an academic institutional level. The funded projects are implemented by the Austrian Agency for International Mobility and Cooperation in Education, Science and Research (OeAD GmbH). APPEARs overall objective is to strengthen the institutional capacities in higher education, research and management through Academic Partnerships with Austrian higher education institutions and through scholarships. The partnerships aim to improve the general standards in higher education, research and management by means of knowledge sharing and designing innovative projects. The achievement of these objectives should finally contribute to effective and sustainable reduction of poverty in the partner countries.
The overarching goal of the evaluation was to assess the ongoing and completed projects of the APPEAR periods I and II. Thereby, it provided insights about the projects adherence to the APPEAR guidelines and basic principles during their implementation, and about the extent to which they were to achieve their objectives and results. Furthermore, the projects were assessed according to the OECD/DAC criteria Relevance, Effectiveness, Efficiency, Impact and Sustainability. The objective of the evaluation was to identify the strengths, weaknesses and challenges of the APPEAR projects and present conclusions, lessons learnt and recommendations for the remaining or future projects.
The evaluation made use of a variety of methods. Primary data was collected during project visits to Ethiopia and Uganda, interviews and focus group discussions with project partners in Austria as well as virtual interviews with staff and key stakeholders of two projects in Nicaragua. The data collection instruments also comprised semi-standardized surveys, programme and project documentations and previous programme evaluations.
Evaluation of CAPAQUA
CAPAQUA (Development of Educational and Research CAPacity in Eastern Africa for the Sustainable Management of AQUAatic Ecosystems) aims at enhancing the capacity of Eastern African Higher Education, Science and Technology (HEST) institutions. CAPAQUA is jointly implemented by the University of Natural Resources and Life Sciences (BOKU), Vienna, the IHE Delft Institute for Water Education, the Egerton University (EGU), the Addis Ababa University (AAU), the Bahir Dar University and the Ethiopian Institute for Agricultural Research, National Fish and Other Aquatic Lives Research Center (EIAR-NFALRC). CAPAQUAs overall objective is to foster sustainable management of aquatic ecosystems and their resources in order to improve livelihoods in Eastern Africa and to contribute towards the achievement of the Sustainable Development Goals.
The main purpose of the evaluation was to identify successes as well as weaknesses of the project, and to provide recommendations for the project partners to improve the relevance, effectiveness, efficiency and sustainability of possible future CAPAQUA projects. The evaluation was also analysing the extent to which the cross-cutting issue gender mainstreaming was applied and identified the major factors influencing the achievement or non-achievement of the expected project results and objectives. In order to assess the project, the evaluation focussed on the project period from October 2015 to September 2018.
A rich set of mainly qualitative data was collected by means of in-depth interviews and focus group discussions with different stakeholders, programme participants, graduates as well as staff. Programme and project documentations provided secondary data and was complemented by primary data collected during focus group discussions. The geographic focus of the evaluation was the Eastern African region, with emphasis on Uganda, Kenya and Ethiopia, where empirical data was collected during short field visits. Additionally, Austrian institutions were visited and stakeholders interviewed.
