Ex-post assessment of the Strategic Fairtrade Funding Programme (SFFP), Phase II (2010-2013)
One of the key objectives of the Swiss Department on Trade Promotion is to support fair trading conditions with developing countries to which private labels and standards substantially contribute. Accordingly, as part of a donor consortium, the State Secretariat for Economic Affairs (SECO) contributed to the Strategic Fairtrade Funding Programme (SFFP) in Phase I from 2008 to 2009 and Phase II from 2010 to 2013. Main purpose of the harmonized funding was to enable disadvantaged producers to establish stronger linkages to consumers, to gain improved market access and to be empowered for combating poverty.
In 2014/2015 an evaluation was already carried out which examined Phase II of the funding programme. Further research on Fairtrade conducted include, among others, a follow-up impact study at the producer-level, covering six product and context settings, implemented by the Center for Evaluation (CEval GmbH) in 2017/2018. Four years after the SFFP ended, it was now of interest for SECO to assess the level of achievement of the strategic indicators set in the Logical Framework of the SFFP as well as SECOs contribution to the same.
The objective of the ex-post assessment was hence threefold: (i) to valorize existing information on the effectiveness and impact of SECOs support to the Fairtrade system, (ii) to complement findings from previous studies with new data on relevance and sustainability of the program and (iii) to distill key findings, recommendations and lessons learnt in view of the SECOs future engagement in Fairtrade. To this end, CEval examined relevant secondary data and collected additional primary data from stakeholders at FT headquarters, three FT regional networks and FT USA.
Results oriented monitoring of the technical cooperation measure COPLAN II and evaluation of the measure’s contribution towards a better alignment of development cooperation and foreign trade facilitation on a systemic level
Climate Change and the uninterrupted decline of fossil fuels affect different regions of the world in different ways. Many nations have thus formulated ambitious goals: Mexico for instance is determined to cover one third of its energy production by means of renewable energies by 2024. Despite such ambitions however environmental conservation has not been marked by major success to this date. This is partly due to a lack of dispersion of modern environmental technology. Against this background the GIZ Project Cooperation Platform Latin America North II (COPLAN II) scrutinizes central deficits that hinder international investment and cooperation such as a lack of professional expertise regarding modern environmental technology in small and medium enterprises (SMEs).
Within the scope of the present assignment CEval together with iSPO Institute supported COPLAN II in its monitoring and evaluation efforts. Starting point for any results oriented monitoring was typically an impact model with respective indicators and assumptions. Changes within the mechanisms of cooperation are however rarely or marginally observed under the monitoring component. This development of cooperation systems or the suitability of the relevant funding instruments are also examined as part of the evaluation component.
All in all, next to monitoring tasks and the compilation of results of individual measures the assignment at hand intended to also generate insights into COPLAN IIs contribution towards a better alignment of development cooperation and foreign trade facilitation as well as public-private sector cooperation on a systemic level.
Meta-Evaluation of ADA Project and Programme Evaluations 2016-2018
Subject of the meta-evaluation were ADA evaluations of development cooperation projects and programmes implemented between 2016 and 2018. The objective was to assess the quality and usefulness of ADAs (internal and external) evaluations and to deliver recommendations on how to improve the ADA evaluation tool for the design and implementation of future evaluations.
A document analysis of the evaluation reports was at the heart of the meta-evaluation. Based on a detailed analysis protocol the quality of said reports was assessed. The use of such a protocol, which covered all relevant analysis dimensions, indicators and rating scales, guaranteed the transparency and reliability of the meta-evaluation results. In order to evaluate the usefulness for and use of the evaluations under study, their target audience (e.g. programme managers) were interviewed via guided individual and group interviews and in order to achieve the greatest possible representativeness of the results a semi-standardized online survey.
Meta-evaluation and synthesis of Christoffel Blindenmission evaluation reports
On behalf of the Christoffel Blindenmission (CBM), CEval conducted an evaluation and synthesis of 24 evaluation reports from 2016 and 2017. With its mandate to transform the lives of disabled people in the worlds poorest countries the project focus of CBM has been an exhaustive collaboration with as well as the promotion of local partners in developing countries and conflict areas.
The primary purpose of the evaluation and synthesis was to conduct a general assessment of the evaluation reports quality and summarize findings of those reports that pass the quality control. In a first step, to ensure homogeneity of data, the quality of reports was assessed systematically by creating a quality control analysis grid. Reports that fulfil minimum quality standards were then subjected to an in-depth analysis based on the evaluation questions of CBM. Main interest of the synthesis was to drew up general learnings along CBMs mandate areas and examine the actual contribution of joint projects to changes for CBM partners and clients. Lastly, by employing an online survey directed to end users of project evaluations, the evaluation inquired the usage and usefulness of the evaluation reports for CBM staff members in seven world regions.
Meta-Evaluation of Plan International evaluation reports
Plan International is working for many years now in Africa, Asia and the Americas to promote child rights and lift children out of poverty. The German National Office of Plan International was founded in 1989, and has been contributing to funding development projects and programs as well as funding and commissioning external evaluations. Plan International Germany decided to implement a meta-evaluation of evaluation reports conducted between 2013 and 2017 covering all geographical working areas.
The primary objectives of the meta-evaluation were to assess the strengths and weaknesses by analyzing the quality of the evaluation reports, to identify the success and risk factors that may affect their quality and to give recommendations. A particular focus was laid upon the influence of the type of evaluation (i.e. by whom it is coordinated and implemented) and the available resources and timeframe.
The meta-evaluation was based on the analysis of the evaluation reports. The document analysis was structured in two phases. While in the first phase (pre-rating) the completeness of the evaluation reports was assessed, in a second phase (in-depth analysis), a more comprehensive set of quality assessment criteria was applied on a sample of reports, to allow for a more detailed assessment of the validity of their findings and usefulness to their addressees. Comprehensive analysis tools were used to analyze the completeness of the reports as well as the validity of findings and the usefulness of evaluation reports.
Evaluation of APPEAR projects (programme periods I and II)
The Austrian Partnership Programme in Higher Education and Research for Development (APPEAR) is conceived, guided and financed by the Austrian Development Agency (ADA) to implement its “Higher Education and Scientific Cooperation” strategy in support of Higher Education and Research for Development on an academic institutional level. The funded projects are implemented by the Austrian Agency for International Mobility and Cooperation in Education, Science and Research (OeAD GmbH). APPEARs overall objective is to strengthen the institutional capacities in higher education, research and management through Academic Partnerships with Austrian higher education institutions and through scholarships. The partnerships aim to improve the general standards in higher education, research and management by means of knowledge sharing and designing innovative projects. The achievement of these objectives should finally contribute to effective and sustainable reduction of poverty in the partner countries.
The overarching goal of the evaluation was to assess the ongoing and completed projects of the APPEAR periods I and II. Thereby, it provided insights about the projects adherence to the APPEAR guidelines and basic principles during their implementation, and about the extent to which they were to achieve their objectives and results. Furthermore, the projects were assessed according to the OECD/DAC criteria Relevance, Effectiveness, Efficiency, Impact and Sustainability. The objective of the evaluation was to identify the strengths, weaknesses and challenges of the APPEAR projects and present conclusions, lessons learnt and recommendations for the remaining or future projects.
The evaluation made use of a variety of methods. Primary data was collected during project visits to Ethiopia and Uganda, interviews and focus group discussions with project partners in Austria as well as virtual interviews with staff and key stakeholders of two projects in Nicaragua. The data collection instruments also comprised semi-standardized surveys, programme and project documentations and previous programme evaluations.
Evaluation of CAPAQUA
CAPAQUA (Development of Educational and Research CAPacity in Eastern Africa for the Sustainable Management of AQUAatic Ecosystems) aims at enhancing the capacity of Eastern African Higher Education, Science and Technology (HEST) institutions. CAPAQUA is jointly implemented by the University of Natural Resources and Life Sciences (BOKU), Vienna, the IHE Delft Institute for Water Education, the Egerton University (EGU), the Addis Ababa University (AAU), the Bahir Dar University and the Ethiopian Institute for Agricultural Research, National Fish and Other Aquatic Lives Research Center (EIAR-NFALRC). CAPAQUAs overall objective is to foster sustainable management of aquatic ecosystems and their resources in order to improve livelihoods in Eastern Africa and to contribute towards the achievement of the Sustainable Development Goals.
The main purpose of the evaluation was to identify successes as well as weaknesses of the project, and to provide recommendations for the project partners to improve the relevance, effectiveness, efficiency and sustainability of possible future CAPAQUA projects. The evaluation was also analysing the extent to which the cross-cutting issue gender mainstreaming was applied and identified the major factors influencing the achievement or non-achievement of the expected project results and objectives. In order to assess the project, the evaluation focussed on the project period from October 2015 to September 2018.
A rich set of mainly qualitative data was collected by means of in-depth interviews and focus group discussions with different stakeholders, programme participants, graduates as well as staff. Programme and project documentations provided secondary data and was complemented by primary data collected during focus group discussions. The geographic focus of the evaluation was the Eastern African region, with emphasis on Uganda, Kenya and Ethiopia, where empirical data was collected during short field visits. Additionally, Austrian institutions were visited and stakeholders interviewed.
Evaluation of the Projects “Sustainable Municipal Development through Partnership Projects” and “Support Program for Municipal Climate Protection and Climate Change Adaption Projects”
As part of Engagement Global – Service für Entwicklungsinitiativen (Service for Development Initiatives) the Service Agency Communities in One World in Germany ist the competence Center for local development cooperation. Founded in 2001 the Service Agency offers a platform for dialogue for municipalities in Germany and the South and supports development activities of local actors by providing them with Training, Information and advisory Services.
The two programs “Sustainable Municipal Development through Partnership Projects – Nachhaltige Kommunalentwicklung durch Partnerschaftsprojekte (Nakopa)” and “Support Program for Municipal Climate Protection and Climate Change Adaption Projects – Förderprogramm für kommunale Klimaschutz- und Klimaanpassungsprojekte (FKKP)” are core activities of the Service Agency. While Nakopa supports existing and new municipal partnership Projects in selected relevant fields of development cooperation, FKKP explicitly addresses municipal activities in the field of climate protection and climate change adaptation.
After the first Project cycle of three years has been completed, the current Evaluation aims at evaluating These two programs with respect to processual aspects of project Administration as well as Impacts of both the funded partnership Projects and the Support programs Nakopa and FKKP. Leading questions of this evaluation are guided by the OECD-DAC evaluation criteria and had been answered using the approach of contribution Analysis and documentary and empirical data. Hence, the evaluation was based on Monitoring data and Project documents on the one Hand and empirical data gathered through extensive partly standardized interviews and partly standardized online surweys with participating communities on the other hand.
Improving the Monitoring and reporting System to implement the Action plan "Inclusion of Persons with disabilities" in the German development co-operation
In February 2013, the Federal Ministry for Economic Cooperation and Development (BMZ) published an action plan for the inclusion of persons with disabilities (PWDs). In doing so, it follows ist duties emanating from Article 32 of the UN Convention on the rights of PWDs. The action plan provides Information on strategic Goals and Actions to design development cooperation of the Federal Republic of Germany towards a better inclusion of PWDs.
To assess the progress and to steer the implementation of planned acitivities an appropriate Monitoring and reporting System is necessary. Hence, the Center of Evaluation (CEval) has been contracted to Support the sector Project “Inclusion of PWDs” of the German Agency for International Cooperation (GIZ) and the departement 300 of the BMZ with its methodological expertise to develop an Analysis grid to evaluate the implementation of the action plan.
External Evaluation of the empowering people. Network
On behalf of Siemens Stiftung CEval GmbH implemented an external evaluation of the empowering people. Network. Founded in 2013, the network brings together inventors and social enterprises to develop solutions to address basic supply problems in fields such as health, food or education in developing countries. The empowering people. Award aims to identify simple and appropriate technological innovations with business potential and make them visible and accessible to everyone. As members of the empowering people. Network, currently 80 organizations that are implementing best-practice approaches receive long-term support in terms of organizational development, professionalization, and networking with partners.
In 2017, the empowering people. Network was evaluated for the first time since its foundation. The evaluation covered the entire lifetime of the network and its purpose was to account for progress and goals achieved so far and to gain lessons for strategic development. It therefore examined what impacts could be achieved by the respective modules and how these were assessed with regard to the overarching objectives. Leading questions of the evaluation were guided by the OECD-DAC evaluation criteria. It was characterized by a participative multi-methods approach and drew on contribution analysis that combined document/secondary analyses with qualitative (interviews, focus group discussions, case studies) as well as quantitative methods (partly standardized surveys). Lessons learnt and recommendations for the future development of the network were derived from the findings.