Results oriented monitoring of the technical cooperation measure COPLAN II and evaluation of the measure’s contribution towards a better alignment of development cooperation and foreign trade facilitation on a systemic level
Climate Change and the uninterrupted decline of fossil fuels affect different regions of the world in different ways. Many nations have thus formulated ambitious goals: Mexico for instance is determined to cover one third of its energy production by means of renewable energies by 2024. Despite such ambitions however environmental conservation has not been marked by major success to this date. This is partly due to a lack of dispersion of modern environmental technology. Against this background the GIZ Project Cooperation Platform Latin America North II (COPLAN II) scrutinizes central deficits that hinder international investment and cooperation such as a lack of professional expertise regarding modern environmental technology in small and medium enterprises (SMEs).
Within the scope of the present assignment CEval together with iSPO Institute supported COPLAN II in its monitoring and evaluation efforts. Starting point for any results oriented monitoring was typically an impact model with respective indicators and assumptions. Changes within the mechanisms of cooperation are however rarely or marginally observed under the monitoring component. This development of cooperation systems or the suitability of the relevant funding instruments are also examined as part of the evaluation component.
All in all, next to monitoring tasks and the compilation of results of individual measures the assignment at hand intended to also generate insights into COPLAN IIs contribution towards a better alignment of development cooperation and foreign trade facilitation as well as public-private sector cooperation on a systemic level.
Meta-Evaluation of ADA Project and Programme Evaluations 2016-2018
Subject of the meta-evaluation were ADA evaluations of development cooperation projects and programmes implemented between 2016 and 2018. The objective was to assess the quality and usefulness of ADAs (internal and external) evaluations and to deliver recommendations on how to improve the ADA evaluation tool for the design and implementation of future evaluations.
A document analysis of the evaluation reports was at the heart of the meta-evaluation. Based on a detailed analysis protocol the quality of said reports was assessed. The use of such a protocol, which covered all relevant analysis dimensions, indicators and rating scales, guaranteed the transparency and reliability of the meta-evaluation results. In order to evaluate the usefulness for and use of the evaluations under study, their target audience (e.g. programme managers) were interviewed via guided individual and group interviews and in order to achieve the greatest possible representativeness of the results a semi-standardized online survey.
Meta-evaluation and synthesis of Christoffel Blindenmission evaluation reports
On behalf of the Christoffel Blindenmission (CBM), CEval conducted an evaluation and synthesis of 24 evaluation reports from 2016 and 2017. With its mandate to transform the lives of disabled people in the worlds poorest countries the project focus of CBM has been an exhaustive collaboration with as well as the promotion of local partners in developing countries and conflict areas.
The primary purpose of the evaluation and synthesis was to conduct a general assessment of the evaluation reports quality and summarize findings of those reports that pass the quality control. In a first step, to ensure homogeneity of data, the quality of reports was assessed systematically by creating a quality control analysis grid. Reports that fulfil minimum quality standards were then subjected to an in-depth analysis based on the evaluation questions of CBM. Main interest of the synthesis was to drew up general learnings along CBMs mandate areas and examine the actual contribution of joint projects to changes for CBM partners and clients. Lastly, by employing an online survey directed to end users of project evaluations, the evaluation inquired the usage and usefulness of the evaluation reports for CBM staff members in seven world regions.
Development of an operational guidebook for integrating and using impact evaluation-based M&E approaches in the field of climate change adaption
Climate change adaption projects require methodologically elaborated Monitoring and Evaluation (M&E) Systems that are not limited to measuring Project Outputs but also Need to include outcomes and further Impacts as well as their attribution to the measures undertaken.
GIZ engaged the Center for Evaluation (CEval) for technical support to develop an operationally viable impact evaluation and provided examples for their applicability in adaptation projects. Results were presented in a guidebook for integrating and using impact evaluation-based M&E approaches in the field of climate change Adaptation. Target Groups of the Framework are ministries, donors, and other related institutions that are looking for the underlying Drivers of Adaptation and evidence-based findings to improve the design of future climate change adaptation projects.
External Evaluation of the Netherlands Fellowship Programme (NFP II) and Netherlands Initiative for Capacity Development in Higher Education (NICHE II)
On behalf of the Dutch Ministry of Foreign Affairs (MFA) the CEval implemented an external Evaluation of the Dutch Fellowship Programme (NFP II) and Netherlands Initiative for Capacity Development in Higher Education (NICHE II). Financed by the MFA and administered by the Netherlands Organisation for International Cooperation in Higher Education ‘EP-Nuffic’, these programs offer capacity development in higher education for developing partner countries.
The overall aim of NFP and NICHE is to help reduce quantitative and qualitative shortage of trained professional staff in developing countries and to build sustainable capacity within the poverty reduction Framework. The NFP provides Fellowships to mid-career professionals to participate in Trainings and postgraduate courses in the Netherlands. NICHE is a capacity building programme to improve the quality, relevance and gender sensitiveness of post-secondary education in NICHE-countries.
The evaluation covered the period of 2002-2015 while a focus was on the latest project phase since 2012 which put an additional emphasis on economic diplomacy and closer alignment with the four priority themes of the Dutch development cooperation strategy (water, sexual reproductive health and rights, security and rule of law and food security). The purpose of the evaluation was to account for the Netherlands funding provided for the two programs under the current project cycle and to gain lessons for future policy development and implementation. Within the scope of the evaluation, both programs were analysed regarding the OECD/DAC criteria of relevance, efficiency, effectiveness, impact and sustainability. Recommendations for possible optimization of the program were developed on the basis of the findings. The evaluation was conducted in cooperation with ACE Europe, a Belgian consultancy.
Program Evaluation “Support in Strengthening Regional Cooperation and Integration in the Field of Quality Infrastructure (QI) in SAARC”
With this regional Project, PTB Germany aims to reduce the technical barriers to trade by fostering cooperation among the SAARC countries (India, Pakistan, Bangladesh, Nepal, Buthan, Maldives, Afghanistan and Sri Lanka) in the field of metrology, accreditation and standardization. The Project targeted professional and Management staff of national QI-institutions as well as ministries relevant to the QI-System. Amongst others, measures lige Trainings and awareness raising Events have been taken and consultancy has been provided.
The evaluation combined qualitative and quantitative methods (mixed method design). A document Analysis and qualitative interviews with different stakeholders from all SAARC countries during a field visit to Bangladesh, Bhutan and India, and an online Survey of former metrology and accreditation Training participants in awareness raising Events have been carried out.
By statistical data and qualitative Content Analysis, the evaluation aimed to provide empirical evidence about the relevance, effectiveness, efficiency, impact and sustainability of the Project (DAC criteria) and the success factores for the observed results and change processes (Capacity Works criteria: strategy, cooperation, steering structure, processes, learning and innovation).
Thereby, it contributed to the further development of the Project approach and ist revision in view of future activities by PTB (in the region). The evaluation results had also been used as a Basis to decide on a follow-up project and on potentially necessary adaptions of its design.
Evaluation of the Projects “Sustainable Municipal Development through Partnership Projects” and “Support Program for Municipal Climate Protection and Climate Change Adaption Projects”
As part of Engagement Global – Service für Entwicklungsinitiativen (Service for Development Initiatives) the Service Agency Communities in One World in Germany ist the competence Center for local development cooperation. Founded in 2001 the Service Agency offers a platform for dialogue for municipalities in Germany and the South and supports development activities of local actors by providing them with Training, Information and advisory Services.
The two programs “Sustainable Municipal Development through Partnership Projects – Nachhaltige Kommunalentwicklung durch Partnerschaftsprojekte (Nakopa)” and “Support Program for Municipal Climate Protection and Climate Change Adaption Projects – Förderprogramm für kommunale Klimaschutz- und Klimaanpassungsprojekte (FKKP)” are core activities of the Service Agency. While Nakopa supports existing and new municipal partnership Projects in selected relevant fields of development cooperation, FKKP explicitly addresses municipal activities in the field of climate protection and climate change adaptation.
After the first Project cycle of three years has been completed, the current Evaluation aims at evaluating These two programs with respect to processual aspects of project Administration as well as Impacts of both the funded partnership Projects and the Support programs Nakopa and FKKP. Leading questions of this evaluation are guided by the OECD-DAC evaluation criteria and had been answered using the approach of contribution Analysis and documentary and empirical data. Hence, the evaluation was based on Monitoring data and Project documents on the one Hand and empirical data gathered through extensive partly standardized interviews and partly standardized online surweys with participating communities on the other hand.
Improving the Monitoring and reporting System to implement the Action plan "Inclusion of Persons with disabilities" in the German development co-operation
In February 2013, the Federal Ministry for Economic Cooperation and Development (BMZ) published an action plan for the inclusion of persons with disabilities (PWDs). In doing so, it follows ist duties emanating from Article 32 of the UN Convention on the rights of PWDs. The action plan provides Information on strategic Goals and Actions to design development cooperation of the Federal Republic of Germany towards a better inclusion of PWDs.
To assess the progress and to steer the implementation of planned acitivities an appropriate Monitoring and reporting System is necessary. Hence, the Center of Evaluation (CEval) has been contracted to Support the sector Project “Inclusion of PWDs” of the German Agency for International Cooperation (GIZ) and the departement 300 of the BMZ with its methodological expertise to develop an Analysis grid to evaluate the implementation of the action plan.
External Evaluation of the empowering people. Network
On behalf of Siemens Stiftung CEval GmbH implemented an external evaluation of the empowering people. Network. Founded in 2013, the network brings together inventors and social enterprises to develop solutions to address basic supply problems in fields such as health, food or education in developing countries. The empowering people. Award aims to identify simple and appropriate technological innovations with business potential and make them visible and accessible to everyone. As members of the empowering people. Network, currently 80 organizations that are implementing best-practice approaches receive long-term support in terms of organizational development, professionalization, and networking with partners.
In 2017, the empowering people. Network was evaluated for the first time since its foundation. The evaluation covered the entire lifetime of the network and its purpose was to account for progress and goals achieved so far and to gain lessons for strategic development. It therefore examined what impacts could be achieved by the respective modules and how these were assessed with regard to the overarching objectives. Leading questions of the evaluation were guided by the OECD-DAC evaluation criteria. It was characterized by a participative multi-methods approach and drew on contribution analysis that combined document/secondary analyses with qualitative (interviews, focus group discussions, case studies) as well as quantitative methods (partly standardized surveys). Lessons learnt and recommendations for the future development of the network were derived from the findings.
Impact Study: Assessing the Impact of Fairtrade on Poverty Reduction through Rural Development, Follow-up study 2017
TransFair e.V. aims at strengthening the objectives of Fairtrade and enabling producers in the Global South to make a living of the products they produce. Among other things, this should be achieved by establishing long-term and as direct as possible relations between producer organizations in the Global South and markets in the Global North and by complying with standards such as the legal minimum wage, wearing protective clothing, paid vacation and social insurance for workers working on plantations.
In order to assess the impact of Fairtrade on the socioeconomic environment of producers, TransFair e.V. and the Swiss Max Havelaar Foundation had commissioned CEval in 2011 to carry out the impact Evaluation “Assessing the Impact of Fairtrade on Poverty Reduction through Rural Development.” The study was the first of its kind to analyze the potential contribution of Fairtrade to improve the living conditions in rural areas. The study design covered the countries Ghana, India, Kenya and Peru and the product categories banana, cotton, flower, coffee, cocoa and tea.
2017, in its 25th anniversary year, TransFair e.V. wanted to follow up on the previous findings and commissioned CEval for the second time, enabling the assessment of longitudinal (comparison over time) and cross-sectional (FT and non-FT case study comparison) changes. A theory-based contribution analysis was applied, considering FTs Theory of Change and the indicators of the committee on sustainability assessment (COSA). Following a case study approach, the findings were mainly based on primary qualitative data collected via Focus Group Discussions, Key Informant Interviews and observations. A participatory quantitative seed assessment to assess perception changes over the last five years complemented the field research, which took place between October 2017 and January 2018. Findings were, eventually, structured along the economic, social and ecological dimension of the COSA indicators.