Evaluation of two school rehabilitationprojects in Nepal
During the two massive earthquakes in Nepal in 2015, several thousand schools got destroyed. Together with the consortium partners Helvetas and Swiss Red Cross, Caritas Switzerland built and rehabilitated 36 schools in the hard to reach district of Sindhupalchok. The Project encompassed trainings regarding disaster risk reduction, operation and maintainance, WASH (water, sanitation, hygiene) and social interaction with drop out students.
The final evaluation was based on three of the Core Humanitarian Standards (appropriateness and relevance, effectiveness and timeliness, preparedness and resilience) in order to draw lessons-learnt and give recommendations for further humanitarian outreaches.
The evaluation utilized different methods: next to interviews with the leadership of the project, representatives of national authorities, team members, head of schools and mayors; Focus Group Discussions gave voice to school management committees and teachers. A comprehensive survey among hundreds of students, parents and teachers enabled a in-depth quantitative analysis.
Central project evaluation of the project Competitive African Cashew Value Chains for Pro-Poor Growth
The cashew value chain offers African producing countries interesting opportunities to create jobs, increase business and economic income and improve food security. Up to the time of the evaluation, however, the potential had hardly been exploited. At 300 to 550 kilograms of raw nuts per hectare, the yields of African producers were far below the yields in India and Vietnam and the processing capacities and labor productivity of the farms were still too low, which meant that only a small proportion of the raw nuts were processed in the African producing countries. The “Competitive African Cashew Value Chains for Pro-Poor Growth” project of the Deutsche Gesellschaft für internationale Zusammenarbeit GmbH supported the expansion and intensification of the cashew value chain in the six countries Benin, Burkina Faso, Côte d’Ivoire, Ghana, Mozambique and Sierra Leone.
The project was based on four levels of action: (i) improving the supply of high-performance seeds/seedlings through research and educating producers through training and consulting, (ii) advising processors of cashew nuts and by-products to improve process efficiency, (iii) intensifying business relations and exchange between all participants in the value chain, and (iv) promoting national and regional framework conditions by advising the relevant ministries and intensifying regional exchange between all actors.
CEval GmbH was commissioned by GIZ with the central project evaluation. The evaluation was aligned with the OECD DAC criteria and thus examined the relevance, effectiveness, efficiency, impact, sustainability and coherence of project implementation. Methodologically, the evaluation was based on the available quantitative secondary data of the project and expanded on this by conducting guided interviews and focus group discussions with project managers, stakeholders and target group representatives. Due to the corona pandemic prevailing at the time of the evaluation, the project was implemented as a remote evaluation. The evaluators from CEval GmbH worked together with local consultants in the African countries, who carried out the data collection in the field.
Development and implementation of an organization-wide results-based monitoring and evaluation system
The objective of the consultancy was to develop an organization-wide results-based monitoring and evaluation (M&E) system, and implement it together with the program staff. Due to the complexity of the results-model that had to be captured, comprising a total of 17 education and support programs, community development programs and services as well as the overarching organizational goals, the assignment had to be research-based. With this scope it is the first wholistic results-based M&E-system on organizational level in the field of Development Cooperation.
According to the system complexity and the large number of involved stakeholders (management, program staff, community actors, civil society organizations, German Federal Ministry for Economic Cooperation and Development), the assignment followed a multi-stage, participatory approach. Starting point was an inventory of the indicators, data collection instruments and analysis methods that were already used by Engagement Globals programs and services. Based on reconstructed and validated results-models, in the next step it was checked to what extent the effectiveness of the programs and services were covered therewith, and which information gaps still existed. Thereby, particular focus was laid upon the identification of the programs contributions to the achievement of the organizational goals of Engagement Global. In the following, the instruments were further developed together with the program staff and practically tested. The results of this development process was finally documented in a comprehensive M&E-concept and integrated into the organization-wide quality assurance system, accompanied with appropriate staff trainings.
Mid-Term Evaluation of the ASEAN Secretariat Post 2015 – Institutional Strengthening and Capacity Development” project (Jakarta, Indonesia)
The ASEAN Secretariat (ASEC) and the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) are jointly implementing the “ASEAN Secretariat Post 2015 & Organizational Development and Capacity Development” project funded by the Federal Foreign Office of Germany. The evaluated project phase began in June 2016 and, building on previous capacity development activities, supported the professionalization of the strategic administrative departments and staff units of the ASEAN Secretariat. It also worked with specialized departments that contribute to strengthening the ASEAN Economic Community. In order to assess the progress of the project phase, CEval was commissioned to conduct an interim evaluation based on the OECD-DAC criteria and extended, project-specific key questions. Taking into account the institutional framework of the ASEC and the diversity of stakeholders, the design of the contribution analysis and a multi-method approach was chosen.
For this purpose, guided interviews with key figures and semi-standardized online surveys of workshop participants were conducted. Project documentation and monitoring data supplemented empirical data and enabled triangulation. The aim of the interim evaluation was to gain insights into the current progress of the project and to enable learning for further project activities and future phases.
Analysis of the 2018 Forum Fairer Handel consumer survey
Since 2007 Forum Fairer Handel has repeatedly mandated the implementation of representative surveys on Fair Trade among the German population. In 2018 the most recent survey has been conducted. Apart from various aspects of Fair Trade consumption the survey also covered questions on the attitudes of the interviewees concerning the political claims of Fair Trade.
Forum Fairer Handel has mandated CEval GmbH with the analysis of the survey data. Descriptive statistics have been applied during the analysis. Furthermore, in relation to some questions the differences in responses between groups have been checked. Whenever data from previous surveys was available, developments and trends over time have also been demonstrated.
Integration of the Programme “Service Agency Communities in One World” (SKEW) in the M&E-System of Engagement Global
Engagement Global gGmbH was established in 2012 as a central institution to promote and coordinate the development policy commitment of civil society, municipal and private sector actors as well as for development policy information and education work. SKEW a program of Engagement Global focuses in particular on the promotion of municipal development cooperation. In order to be able to monitor both the appropriate use of the funds and above all their effectiveness, Engagement Global has created a comprehensive monitoring and evaluation (M&E) system for all departments and programs. Since SKEW was excluded in the development of the monitoring system at that time, CEval and ISPO were commissioned to integrate SKEW into the organization-wide monitoring system.
As with the design of the company-wide M&E system, first the theory of change and indicators were reviewed or developed. Then the objectives and indicators were integrated into the strategical claims of Engagement Global. In order to complete the creation of a monitoring system for the SKEW, a data collection plan were created and the associated data collection instruments had been (further) developed.
Ex-post assessment of the Strategic Fairtrade Funding Programme (SFFP), Phase II (2010-2013)
One of the key objectives of the Swiss Department on Trade Promotion is to support fair trading conditions with developing countries to which private labels and standards substantially contribute. Accordingly, as part of a donor consortium, the State Secretariat for Economic Affairs (SECO) contributed to the Strategic Fairtrade Funding Programme (SFFP) in Phase I from 2008 to 2009 and Phase II from 2010 to 2013. Main purpose of the harmonized funding was to enable disadvantaged producers to establish stronger linkages to consumers, to gain improved market access and to be empowered for combating poverty.
In 2014/2015 an evaluation was already carried out which examined Phase II of the funding programme. Further research on Fairtrade conducted include, among others, a follow-up impact study at the producer-level, covering six product and context settings, implemented by the Center for Evaluation (CEval GmbH) in 2017/2018. Four years after the SFFP ended, it was now of interest for SECO to assess the level of achievement of the strategic indicators set in the Logical Framework of the SFFP as well as SECOs contribution to the same.
The objective of the ex-post assessment was hence threefold: (i) to valorize existing information on the effectiveness and impact of SECOs support to the Fairtrade system, (ii) to complement findings from previous studies with new data on relevance and sustainability of the program and (iii) to distill key findings, recommendations and lessons learnt in view of the SECOs future engagement in Fairtrade. To this end, CEval examined relevant secondary data and collected additional primary data from stakeholders at FT headquarters, three FT regional networks and FT USA.
Results oriented monitoring of the technical cooperation measure COPLAN II and evaluation of the measure’s contribution towards a better alignment of development cooperation and foreign trade facilitation on a systemic level
Climate Change and the uninterrupted decline of fossil fuels affect different regions of the world in different ways. Many nations have thus formulated ambitious goals: Mexico for instance is determined to cover one third of its energy production by means of renewable energies by 2024. Despite such ambitions however environmental conservation has not been marked by major success to this date. This is partly due to a lack of dispersion of modern environmental technology. Against this background the GIZ Project Cooperation Platform Latin America North II (COPLAN II) scrutinizes central deficits that hinder international investment and cooperation such as a lack of professional expertise regarding modern environmental technology in small and medium enterprises (SMEs).
Within the scope of the present assignment CEval together with iSPO Institute supported COPLAN II in its monitoring and evaluation efforts. Starting point for any results oriented monitoring was typically an impact model with respective indicators and assumptions. Changes within the mechanisms of cooperation are however rarely or marginally observed under the monitoring component. This development of cooperation systems or the suitability of the relevant funding instruments are also examined as part of the evaluation component.
All in all, next to monitoring tasks and the compilation of results of individual measures the assignment at hand intended to also generate insights into COPLAN IIs contribution towards a better alignment of development cooperation and foreign trade facilitation as well as public-private sector cooperation on a systemic level.
Meta-Evaluation of ADA Project and Programme Evaluations 2016-2018
Subject of the meta-evaluation were ADA evaluations of development cooperation projects and programmes implemented between 2016 and 2018. The objective was to assess the quality and usefulness of ADAs (internal and external) evaluations and to deliver recommendations on how to improve the ADA evaluation tool for the design and implementation of future evaluations.
A document analysis of the evaluation reports was at the heart of the meta-evaluation. Based on a detailed analysis protocol the quality of said reports was assessed. The use of such a protocol, which covered all relevant analysis dimensions, indicators and rating scales, guaranteed the transparency and reliability of the meta-evaluation results. In order to evaluate the usefulness for and use of the evaluations under study, their target audience (e.g. programme managers) were interviewed via guided individual and group interviews and in order to achieve the greatest possible representativeness of the results a semi-standardized online survey.
Meta-evaluation and synthesis of Christoffel Blindenmission evaluation reports
On behalf of the Christoffel Blindenmission (CBM), CEval conducted an evaluation and synthesis of 24 evaluation reports from 2016 and 2017. With its mandate to transform the lives of disabled people in the worlds poorest countries the project focus of CBM has been an exhaustive collaboration with as well as the promotion of local partners in developing countries and conflict areas.
The primary purpose of the evaluation and synthesis was to conduct a general assessment of the evaluation reports quality and summarize findings of those reports that pass the quality control. In a first step, to ensure homogeneity of data, the quality of reports was assessed systematically by creating a quality control analysis grid. Reports that fulfil minimum quality standards were then subjected to an in-depth analysis based on the evaluation questions of CBM. Main interest of the synthesis was to drew up general learnings along CBMs mandate areas and examine the actual contribution of joint projects to changes for CBM partners and clients. Lastly, by employing an online survey directed to end users of project evaluations, the evaluation inquired the usage and usefulness of the evaluation reports for CBM staff members in seven world regions.
