Sector evaluation of climate projects within the framework of the BMZ Special Initiative on Climate of the Hanns Seidel Foundation e.V. 2019
The Hanns Seidel Foundation (HSS) has been receiving funds from the BMZ Special Initiative on Climate since 2011 and has carried out project activities in the fields of climate, environment and energy in numerous countries since then. The measures in this area are intended to make a contribution to environmental protection and nature conservation as a whole, to combating and adapting to climate change, to sustainable energy supply and resource efficiency, and to sustainable and environmentally conscious agriculture and forestry. A basic strategy was developed from the context of Hanns Seidel Foundation’s current work abroad in the fields of climate, environment and energy and its attempts to find solutions to these topics at an international level. This strategy was evaluated by CEval GmbH using three projects in India, Bolivia and Namibia as examples.
From a methodological point of view, the evaluation included a theory-based part, which consisted of the reconstruction of the Theories of Change (ToC) of the three projects. In addition, the three selected projects were evaluated at project level, for which field phases in all three countries and interviews with partners and target groups have been conducted. In addition, the evaluation included a synthesis part which referred to the climate strategy of the Hanns Seidel Foundation in general and in which the results of the evaluations of the three projects were bundled and supplemented with information from the headquarters of the HSS in Munich.
Rapid Appraisal of two Moringa Investments
Moringa SCA SICAR is a private equity fund that has been established as a response to growing interest of the Financial and the Development sector on investments that provide for a financial, environmental and a social return at the same time. Moringa focuses on investments in agroforestry in African and Latin American Developing and Least Developed Countries. It applies agroforestry land use systems on a commercial basis with a nucleus commercial investment approach that integrates large numbers of neighboring smallholder farmers and processors into its business model through contractual arrangements. The Agroforestry Technical Assistance Facility (ATAF) was created by Moringa Partnership to provide technical assistance in relation to investments of the Fund with the goal to amplify and upscale positive environmental and social impacts triggered through Moringa investments.
CEval GmbH has conducted Baseline Rapid Appraisals for two Moringa Investments, focused specifically on the livelihood situation of the farmers and other relevant target groups, such as factory workers. Methodologically, the Rapid Appraisals followed a mixed-methods approach, which included a small survey with farmers for each Rapid Appraisal, as well as interviews with the relevant stakeholders. Rapid Appraisals were conducted for Moringa investments in Benin and Togo.
Evaluation of two school rehabilitationprojects in Nepal
During the two massive earthquakes in Nepal in 2015, several thousand schools got destroyed. Together with the consortium partners Helvetas and Swiss Red Cross, Caritas Switzerland built and rehabilitated 36 schools in the hard to reach district of Sindhupalchok. The Project encompassed trainings regarding disaster risk reduction, operation and maintainance, WASH (water, sanitation, hygiene) and social interaction with drop out students.
The final evaluation was based on three of the Core Humanitarian Standards (appropriateness and relevance, effectiveness and timeliness, preparedness and resilience) in order to draw lessons-learnt and give recommendations for further humanitarian outreaches.
The evaluation utilized different methods: next to interviews with the leadership of the project, representatives of national authorities, team members, head of schools and mayors; Focus Group Discussions gave voice to school management committees and teachers. A comprehensive survey among hundreds of students, parents and teachers enabled a in-depth quantitative analysis.
Central project evaluation of the project Competitive African Cashew Value Chains for Pro-Poor Growth
The cashew value chain offers African producing countries interesting opportunities to create jobs, increase business and economic income and improve food security. Up to the time of the evaluation, however, the potential had hardly been exploited. At 300 to 550 kilograms of raw nuts per hectare, the yields of African producers were far below the yields in India and Vietnam and the processing capacities and labor productivity of the farms were still too low, which meant that only a small proportion of the raw nuts were processed in the African producing countries. The “Competitive African Cashew Value Chains for Pro-Poor Growth” project of the Deutsche Gesellschaft für internationale Zusammenarbeit GmbH supported the expansion and intensification of the cashew value chain in the six countries Benin, Burkina Faso, Côte d’Ivoire, Ghana, Mozambique and Sierra Leone.
The project was based on four levels of action: (i) improving the supply of high-performance seeds/seedlings through research and educating producers through training and consulting, (ii) advising processors of cashew nuts and by-products to improve process efficiency, (iii) intensifying business relations and exchange between all participants in the value chain, and (iv) promoting national and regional framework conditions by advising the relevant ministries and intensifying regional exchange between all actors.
CEval GmbH was commissioned by GIZ with the central project evaluation. The evaluation was aligned with the OECD DAC criteria and thus examined the relevance, effectiveness, efficiency, impact, sustainability and coherence of project implementation. Methodologically, the evaluation was based on the available quantitative secondary data of the project and expanded on this by conducting guided interviews and focus group discussions with project managers, stakeholders and target group representatives. Due to the corona pandemic prevailing at the time of the evaluation, the project was implemented as a remote evaluation. The evaluators from CEval GmbH worked together with local consultants in the African countries, who carried out the data collection in the field.
Development and implementation of an organization-wide results-based monitoring and evaluation system
The objective of the consultancy was to develop an organization-wide results-based monitoring and evaluation (M&E) system, and implement it together with the program staff. Due to the complexity of the results-model that had to be captured, comprising a total of 17 education and support programs, community development programs and services as well as the overarching organizational goals, the assignment had to be research-based. With this scope it is the first wholistic results-based M&E-system on organizational level in the field of Development Cooperation.
According to the system complexity and the large number of involved stakeholders (management, program staff, community actors, civil society organizations, German Federal Ministry for Economic Cooperation and Development), the assignment followed a multi-stage, participatory approach. Starting point was an inventory of the indicators, data collection instruments and analysis methods that were already used by Engagement Globals programs and services. Based on reconstructed and validated results-models, in the next step it was checked to what extent the effectiveness of the programs and services were covered therewith, and which information gaps still existed. Thereby, particular focus was laid upon the identification of the programs contributions to the achievement of the organizational goals of Engagement Global. In the following, the instruments were further developed together with the program staff and practically tested. The results of this development process was finally documented in a comprehensive M&E-concept and integrated into the organization-wide quality assurance system, accompanied with appropriate staff trainings.
Mid-Term Evaluation of the ASEAN Secretariat Post 2015 – Institutional Strengthening and Capacity Development” project (Jakarta, Indonesia)
The ASEAN Secretariat (ASEC) and the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) are jointly implementing the “ASEAN Secretariat Post 2015 & Organizational Development and Capacity Development” project funded by the Federal Foreign Office of Germany. The evaluated project phase began in June 2016 and, building on previous capacity development activities, supported the professionalization of the strategic administrative departments and staff units of the ASEAN Secretariat. It also worked with specialized departments that contribute to strengthening the ASEAN Economic Community. In order to assess the progress of the project phase, CEval was commissioned to conduct an interim evaluation based on the OECD-DAC criteria and extended, project-specific key questions. Taking into account the institutional framework of the ASEC and the diversity of stakeholders, the design of the contribution analysis and a multi-method approach was chosen.
For this purpose, guided interviews with key figures and semi-standardized online surveys of workshop participants were conducted. Project documentation and monitoring data supplemented empirical data and enabled triangulation. The aim of the interim evaluation was to gain insights into the current progress of the project and to enable learning for further project activities and future phases.
Analysis of the 2018 Forum Fairer Handel consumer survey
Since 2007 Forum Fairer Handel has repeatedly mandated the implementation of representative surveys on Fair Trade among the German population. In 2018 the most recent survey has been conducted. Apart from various aspects of Fair Trade consumption the survey also covered questions on the attitudes of the interviewees concerning the political claims of Fair Trade.
Forum Fairer Handel has mandated CEval GmbH with the analysis of the survey data. Descriptive statistics have been applied during the analysis. Furthermore, in relation to some questions the differences in responses between groups have been checked. Whenever data from previous surveys was available, developments and trends over time have also been demonstrated.
Integration of the Programme “Service Agency Communities in One World” (SKEW) in the M&E-System of Engagement Global
Engagement Global gGmbH was established in 2012 as a central institution to promote and coordinate the development policy commitment of civil society, municipal and private sector actors as well as for development policy information and education work. SKEW a program of Engagement Global focuses in particular on the promotion of municipal development cooperation. In order to be able to monitor both the appropriate use of the funds and above all their effectiveness, Engagement Global has created a comprehensive monitoring and evaluation (M&E) system for all departments and programs. Since SKEW was excluded in the development of the monitoring system at that time, CEval and ISPO were commissioned to integrate SKEW into the organization-wide monitoring system.
As with the design of the company-wide M&E system, first the theory of change and indicators were reviewed or developed. Then the objectives and indicators were integrated into the strategical claims of Engagement Global. In order to complete the creation of a monitoring system for the SKEW, a data collection plan were created and the associated data collection instruments had been (further) developed.
Ex-post assessment of the Strategic Fairtrade Funding Programme (SFFP), Phase II (2010-2013)
One of the key objectives of the Swiss Department on Trade Promotion is to support fair trading conditions with developing countries to which private labels and standards substantially contribute. Accordingly, as part of a donor consortium, the State Secretariat for Economic Affairs (SECO) contributed to the Strategic Fairtrade Funding Programme (SFFP) in Phase I from 2008 to 2009 and Phase II from 2010 to 2013. Main purpose of the harmonized funding was to enable disadvantaged producers to establish stronger linkages to consumers, to gain improved market access and to be empowered for combating poverty.
In 2014/2015 an evaluation was already carried out which examined Phase II of the funding programme. Further research on Fairtrade conducted include, among others, a follow-up impact study at the producer-level, covering six product and context settings, implemented by the Center for Evaluation (CEval GmbH) in 2017/2018. Four years after the SFFP ended, it was now of interest for SECO to assess the level of achievement of the strategic indicators set in the Logical Framework of the SFFP as well as SECOs contribution to the same.
The objective of the ex-post assessment was hence threefold: (i) to valorize existing information on the effectiveness and impact of SECOs support to the Fairtrade system, (ii) to complement findings from previous studies with new data on relevance and sustainability of the program and (iii) to distill key findings, recommendations and lessons learnt in view of the SECOs future engagement in Fairtrade. To this end, CEval examined relevant secondary data and collected additional primary data from stakeholders at FT headquarters, three FT regional networks and FT USA.
Results oriented monitoring of the technical cooperation measure COPLAN II and evaluation of the measure’s contribution towards a better alignment of development cooperation and foreign trade facilitation on a systemic level
Climate Change and the uninterrupted decline of fossil fuels affect different regions of the world in different ways. Many nations have thus formulated ambitious goals: Mexico for instance is determined to cover one third of its energy production by means of renewable energies by 2024. Despite such ambitions however environmental conservation has not been marked by major success to this date. This is partly due to a lack of dispersion of modern environmental technology. Against this background the GIZ Project Cooperation Platform Latin America North II (COPLAN II) scrutinizes central deficits that hinder international investment and cooperation such as a lack of professional expertise regarding modern environmental technology in small and medium enterprises (SMEs).
Within the scope of the present assignment CEval together with iSPO Institute supported COPLAN II in its monitoring and evaluation efforts. Starting point for any results oriented monitoring was typically an impact model with respective indicators and assumptions. Changes within the mechanisms of cooperation are however rarely or marginally observed under the monitoring component. This development of cooperation systems or the suitability of the relevant funding instruments are also examined as part of the evaluation component.
All in all, next to monitoring tasks and the compilation of results of individual measures the assignment at hand intended to also generate insights into COPLAN IIs contribution towards a better alignment of development cooperation and foreign trade facilitation as well as public-private sector cooperation on a systemic level.