Impact Study: Assessing the Impact of Fairtrade on Poverty Reduction through Rural Development, Follow-up study 2017
TransFair e.V. aims at strengthening the objectives of Fairtrade and enabling producers in the Global South to make a living of the products they produce. Among other things, this should be achieved by establishing long-term and as direct as possible relations between producer organizations in the Global South and markets in the Global North and by complying with standards such as the legal minimum wage, wearing protective clothing, paid vacation and social insurance for workers working on plantations.
In order to assess the impact of Fairtrade on the socioeconomic environment of producers, TransFair e.V. and the Swiss Max Havelaar Foundation had commissioned CEval in 2011 to carry out the impact Evaluation “Assessing the Impact of Fairtrade on Poverty Reduction through Rural Development.” The study was the first of its kind to analyze the potential contribution of Fairtrade to improve the living conditions in rural areas. The study design covered the countries Ghana, India, Kenya and Peru and the product categories banana, cotton, flower, coffee, cocoa and tea.
2017, in its 25th anniversary year, TransFair e.V. wanted to follow up on the previous findings and commissioned CEval for the second time, enabling the assessment of longitudinal (comparison over time) and cross-sectional (FT and non-FT case study comparison) changes. A theory-based contribution analysis was applied, considering FTs Theory of Change and the indicators of the committee on sustainability assessment (COSA). Following a case study approach, the findings were mainly based on primary qualitative data collected via Focus Group Discussions, Key Informant Interviews and observations. A participatory quantitative seed assessment to assess perception changes over the last five years complemented the field research, which took place between October 2017 and January 2018. Findings were, eventually, structured along the economic, social and ecological dimension of the COSA indicators.
Formative Evaluation of the Out-of-School-Children Initiative (OOSCI)
The focus of this evaluation was the Out-of-School-Children Initiative (OOSCI) by UNICEF and UNESCO Institute for Statistics (UIS). The initiative was launched in 2010 to achieve a reduction in the number of children out of school in participating countries. Means to achieve this goal were to create comprehensive profiles of out-of-School children in each Country, identify the barriers that push children out of school and to propose changes in the policies and strategies of the partner country to address these barriers.
The evaluation aimed at testing the validity of the theory of change and its assumptions. It is used to strengthen the program logic and to provide a formative assessment of the progress towards the overall goal of achieving a reduction in out-of-school children. The evaluation also provides an in-depth insight into benefits for the different stakeholders and their involvement in the initiative. Furthermore, the findings of the evaluation are used to meet the accountability requirements by the donors of OOSCI.
Main source of information of this evaluation have been documents like OOSCI country studies, OOSCI regional studies, government plans and documents from other donors (e.g. the World Bank, UNESCO, foreign donors etc.). In a document review we assessed progress made through OOSCI for a sample of 42 countries and assessed the contents within the document using a 4-step coding. These codings were used for a descriptive analysis and additionally for a Qualitative Comparative Analysis (QCA). QCA enabled us to analyze thoroughly which set of conditions leads to which outcome. Further instrument of data collection has been an online survey to all country offices. Additional qualitative data has been collected in the US and during eight country visits. The evaluation has been conducted together with Proman and further Consultants. CEval took over the lead with regard to the evaluation methodology.
Meta-Evaluation of 34 evaluation reports of World Vision Deutschland
For the third time, World Vision Germany (WVG) commissioned CEval to conduct a meta-evaluation of evaluation reports of its supported development programmes. In total, 34 evaluation reports of the financial years of 2016 and 2017 were included, to analyze their quality and validity.
The meta-evaluation was based on an analysis matrix, jointly created by CEval and WVG, which is structured along the main criteria of voice and inclusion, transparency, methodology, triangulation, contribution, satisfaction of information needs, conceptualization of findings and the newly added criterion of sustainability. Firstly, criteria were assessed systematically along pre-defined sub-questions and corresponding scales. Secondly, results were then aggregated and visualized. Lastly, the results were compared with previous meta-evaluations and trends were discussed.
Midterm-Review of the FKPW program
In international comparison, German municipalities have comparably broad competencies due to their strong local self-administration. These competencies are also very valuable for development cooperation. Within the framework of municipal partnerships, German municipalities and their partner municipalities jointly carry out projects in municipalities of the Global South. In the process, technical knowledge is transferred from the German municipality to the partner municipality. The Service Agency Communities in One World (SKEW in German) offers various ways of supporting the developmental engagement of municipal partnerships.
One of these funding opportunities is the personnel funding instrument “Experts for Municipal Partnerships Worldwide” (FKPW in German). Within the framework of the FKPW program, municipal partnerships can apply for the placement of an expert in the partner municipality to support the creation of local structures and capacity development. The aim of the program is to enhance the value of the municipal partnership and the commitment of the German municipality through the placement of an expert.
CEval GmbH was commissioned by GIZ to conduct a mid-term review of the FKPW program. The aim of the review was to bring together the views of the different actors involved in the program and to give recommendations for the further development of the program. The main objective therefore was institutional learning on the part of the two implementing organizations, GIZ and SKEW. The methodology of the review was purely qualitative and based on the approach of the open capturing of perspectives. For the review, interviews and focus group discussions were conducted with representatives of the German municipalities, the partner municipalities, with the experts and the steering team of the programme, as well as with representatives of the responsible department of the German Federal Ministry for Economic Cooperation and Development (BMZ in German) and representatives of various departments within the organizations GIZ and SKEW.
Consultancy Services for the Development of a Project Monitoring and Evaluation Framework for ASEAN Cooperation Projects
CEval was commissioned by the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) to provide consultancy on the development of a Project Monitoring and Evaluation Framework (PMEF) for ASEAN Cooperation Projects, which is part of the ASEAN Secretariat Post 2015 Institutional Strengthening and Capacity Development project, an initiative under ASEAN-Germany cooperation implemented by GIZ. Prime task was to look into the ASEAN Secretariats corporate monitoring and evaluation processes and to provide expert support for refining the ASEAN Secretariats Project M&E framework. Special emphasis was put on providing support in harnessing and consolidating the monitoring and evaluation framework for ASEAN cooperation projects.
During the consultancy a number of stakeholder consultations as well as a three-day method training were conducted. The consultations and training served to improve the M&E capacities of key staff and to create a common discussion forum for different departments across the three pillars for exchanging views, and developing ownership and acceptance among the same.
The final report consolidates lessons learned, describes best practices within the sector, and develops concepts, possible mechanisms and recommendations on implementing a functional PMEF.
Meta-evaluation of project and programme evaluations 2015-2017
The Ministry for Foreign Affairs of Finland commissioned CEval together with Particip and Indufor to conduct a meta-evaluation of their evaluations of development cooperation programmes and projects. The Finnish Ministry is regularly conducting meta-evaluations and subject to this meta-evaluation are the evaluations from 2015-2017.
The meta-evaluation has been initiated to improve evaluation reports, evaluation management practices and evaluation capacity development and furthermore to identify possible gaps in the Finnish evaluation practice. Furthermore, also the content of the evaluation reports was the focus e.g. the success in achieving the DAC-Criteria and further Finnish Cross Cutting Topics like gender equality or climate sustainability were evaluated.
A document analysis of the evaluation reports has been at the heart of the meta-evaluation to assess first the quality and afterwards the content of the evaluation reports. Different assessment tools were developed to provide a feasible and reliable assessment process. In the quality check the evaluation reports were reviewed for the existence of information essential to understand the evaluation object, process and results. Additionally, methodological requirements were assessed e.g. data and method triangulation or the logical flow from data over findings and conclusions to recommendations. Another tool was used for the content analysis in which the achievements with respect to the different evaluation criteria as well as conclusions and recommendations were analysed in detail in a semi-structured way.
University cooperation with the University of Costa Rica (UCR) and the Catholic Pontifical University of Ecuador (PUCE) supported through the DAAD funding programme „Subject-Related Partnerships with Institutions of Higher Education in Developing Countries
The Saarland University (UdS) and the University of Costa Rica (UCR) have been maintaining an intensive cooperation since 2008, which was financially supported by the DAAD programme Subject-Related Partnerships with Institutions of Higher Education in Developing Countries between 2012 and 2015. Through the DAAD University Advisory Program, the preceding collaboration had already received financial support. Particularly in the light of an ever-rising need for evaluations and competencies in the field of evaluation, both partners are zealous to continue the collaboration within the framework of the DAAD funding programme and to expand it sustainably by the integration of a second partnering institute of higher education. The Catholic Pontifical University of Ecuador (PUCE) could be won for this objective.
Crucial elements of this third phase of cooperation were supposed to be the implementation of a new blended learning Masters study program in evaluation at the PUCE, and the implementation of a further educational opportunity in the blended learning format, respectively, at the Center for Research and Training in Public Administration (CICAP) at the UCR, and quality assurance of the formation and further education in evaluation in Costa Rica, Ecuador, and Latin America (amelioration of formation and further education as well as research in evaluation). Moreover, the establishment of a Latin American university network in the discipline of evaluation (network formation and internationalization), the strengthening of practical orientation within the academic formation in evaluation and the contribution to evidence-based policy making (application of scientific evaluative research in political practice), and, eventually, the consolidation and institutional expansion of the cooperation between the universities involved and the research institute CICAP (triangular and South-South-cooperation) were on the agenda of the program. The respective cooperation was intended to last for four years (2017-2020).
Development of an operational guidebook for integrating and using impact evaluation-based M&E approaches in the field of climate change adaption
Climate change adaption projects require methodologically elaborated Monitoring and Evaluation (M&E) Systems that are not limited to measuring Project Outputs but also Need to include outcomes and further Impacts as well as their attribution to the measures undertaken.
GIZ engaged the Center for Evaluation (CEval) for technical support to develop an operationally viable impact evaluation and provided examples for their applicability in adaptation projects. Results were presented in a guidebook for integrating and using impact evaluation-based M&E approaches in the field of climate change Adaptation. Target Groups of the Framework are ministries, donors, and other related institutions that are looking for the underlying Drivers of Adaptation and evidence-based findings to improve the design of future climate change adaptation projects.
External Evaluation of the Netherlands Fellowship Programme (NFP II) and Netherlands Initiative for Capacity Development in Higher Education (NICHE II)
On behalf of the Dutch Ministry of Foreign Affairs (MFA) the CEval implemented an external Evaluation of the Dutch Fellowship Programme (NFP II) and Netherlands Initiative for Capacity Development in Higher Education (NICHE II). Financed by the MFA and administered by the Netherlands Organisation for International Cooperation in Higher Education ‘EP-Nuffic’, these programs offer capacity development in higher education for developing partner countries.
The overall aim of NFP and NICHE is to help reduce quantitative and qualitative shortage of trained professional staff in developing countries and to build sustainable capacity within the poverty reduction Framework. The NFP provides Fellowships to mid-career professionals to participate in Trainings and postgraduate courses in the Netherlands. NICHE is a capacity building programme to improve the quality, relevance and gender sensitiveness of post-secondary education in NICHE-countries.
The evaluation covered the period of 2002-2015 while a focus was on the latest project phase since 2012 which put an additional emphasis on economic diplomacy and closer alignment with the four priority themes of the Dutch development cooperation strategy (water, sexual reproductive health and rights, security and rule of law and food security). The purpose of the evaluation was to account for the Netherlands funding provided for the two programs under the current project cycle and to gain lessons for future policy development and implementation. Within the scope of the evaluation, both programs were analysed regarding the OECD/DAC criteria of relevance, efficiency, effectiveness, impact and sustainability. Recommendations for possible optimization of the program were developed on the basis of the findings. The evaluation was conducted in cooperation with ACE Europe, a Belgian consultancy.
Program Evaluation “Support in Strengthening Regional Cooperation and Integration in the Field of Quality Infrastructure (QI) in SAARC”
With this regional Project, PTB Germany aims to reduce the technical barriers to trade by fostering cooperation among the SAARC countries (India, Pakistan, Bangladesh, Nepal, Buthan, Maldives, Afghanistan and Sri Lanka) in the field of metrology, accreditation and standardization. The Project targeted professional and Management staff of national QI-institutions as well as ministries relevant to the QI-System. Amongst others, measures lige Trainings and awareness raising Events have been taken and consultancy has been provided.
The evaluation combined qualitative and quantitative methods (mixed method design). A document Analysis and qualitative interviews with different stakeholders from all SAARC countries during a field visit to Bangladesh, Bhutan and India, and an online Survey of former metrology and accreditation Training participants in awareness raising Events have been carried out.
By statistical data and qualitative Content Analysis, the evaluation aimed to provide empirical evidence about the relevance, effectiveness, efficiency, impact and sustainability of the Project (DAC criteria) and the success factores for the observed results and change processes (Capacity Works criteria: strategy, cooperation, steering structure, processes, learning and innovation).
Thereby, it contributed to the further development of the Project approach and ist revision in view of future activities by PTB (in the region). The evaluation results had also been used as a Basis to decide on a follow-up project and on potentially necessary adaptions of its design.
